mirror of
https://github.com/akitaonrails/ai-memory.git
synced 2026-10-02 03:24:46 +08:00
AkitaOnRails
+15
-3
@@ -15,13 +15,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
shipped, and `docker/.env.production.example` already ships an OpenRouter
|
||||
default, but the provider-facing doc mentioned OpenRouter only inside the
|
||||
generic `openai-compat` row. The new subsection covers a full working
|
||||
env-file, the `openai-compat` embedder path for OpenRouter's hidden-but-
|
||||
supported embedding models (`baai/bge-m3`,
|
||||
`intfloat/multilingual-e5-large`), a pointer to
|
||||
env-file, the `openai-compat` embedder path (with a note to verify any
|
||||
provider's `/v1/embeddings` endpoint before relying on it), a pointer to
|
||||
`docs/llm-provider-comparison.md` for model selection, and a security /
|
||||
gotchas block (auth-token requirement for non-loopback binds, env-file
|
||||
hygiene, cheap-model consolidation drift, `:free`-tier shared-pool
|
||||
limits, reasoning-model incompatibility). (#949)
|
||||
- `docs/backup.md` documents the remote-git-mirror backup pattern for a
|
||||
single-user install: what to include, what to exclude (derived SQLite index,
|
||||
models cache, logs, secrets), how to schedule with a `systemd --user` timer,
|
||||
how to restore, and the security posture per `SECURITY.md`, including a
|
||||
"what ends up in your wiki" section that names the exposure (sanitized
|
||||
prompts, tool I/O, page bodies) and lists encrypted-archive alternatives
|
||||
(`age`, `restic`, `borg`, `git-crypt`) for cases where a private mirror
|
||||
repo is not enough. A worked example ships under `docs/examples/backup/`
|
||||
(snapshot script, `.service` and `.timer` unit files, `.gitignore` for
|
||||
the mirror repo). Pointers added from `docs/deploy.md#backups`,
|
||||
`docs/airgapped-install.md`, and the README docs table. The on-box
|
||||
`ai-memory backup --to <tarball>` command
|
||||
(`docs/lifecycle-ops.md#backup`) is unchanged. (#950)
|
||||
|
||||
### Fixed
|
||||
- `memory_query` now returns `global_scope_hits` (standing `_global` user/team
|
||||
|
||||
@@ -448,6 +448,7 @@ diagram, crate breakdown, schema notes, and invariants.
|
||||
| [`docs/users.md`](docs/users.md) | **Multi-user attribution and human login.** Four-rung bearer ladder, password sessions, `ai-memory user` / `api-key` walkthrough, brownfield migration. |
|
||||
| [`docs/https-via-proxy.md`](docs/https-via-proxy.md) | **HTTPS via a reverse proxy.** When you need TLS and when you don't, with copy-paste Caddy / nginx / Cloudflare Tunnel templates and the "secure when you're not" failure modes. |
|
||||
| [`docs/lifecycle-ops.md`](docs/lifecycle-ops.md) | **Read before purge / rename / backup / restore / reset / reindex / restore-page.** Safety matrix, per-project disk layout, checkpoint page recovery, and operator workflows. |
|
||||
| [`docs/backup.md`](docs/backup.md) | Backing up the wiki + data dir to a remote git repository: what to include, what to exclude, scheduled push pattern, restore, and security posture. Companion to `docs/lifecycle-ops.md` (which covers the on-box `ai-memory backup` snapshot). |
|
||||
| [`docs/llm-providers.md`](docs/llm-providers.md) | Provider configuration for consolidation and embeddings. |
|
||||
| [`docs/security.md`](docs/security.md) | The full security model. |
|
||||
| [`docs/support-matrix.md`](docs/support-matrix.md) | The full agent/platform matrix with notes. |
|
||||
|
||||
@@ -59,7 +59,9 @@ runtime or other native library to source separately.
|
||||
|
||||
- **git remote sync**, if you use it to push the wiki repository somewhere,
|
||||
is your own channel to secure (`SECURITY.md`, "Remote sync security" —
|
||||
out of scope for ai-memory itself).
|
||||
out of scope for ai-memory itself). [`docs/backup.md`](backup.md) has a
|
||||
worked example of the rsync + `git push` pattern with the appropriate
|
||||
secret and derived-state exclusions.
|
||||
- **Update/patch delivery** in an air-gapped environment is manual: pull a
|
||||
new release and checksum on a connected machine, then transfer it in,
|
||||
the same as the initial install.
|
||||
|
||||
+233
@@ -0,0 +1,233 @@
|
||||
# Backups and remote git mirror
|
||||
|
||||
> This document covers backup patterns beyond the on-box `ai-memory backup
|
||||
> --to <tarball>` snapshot documented in
|
||||
> [`docs/lifecycle-ops.md#backup`](lifecycle-ops.md#backup) and the "point-in-time
|
||||
> consistency" `scp` idiom in [`docs/deploy.md#backups`](deploy.md#backups). If
|
||||
> you already have a working tarball routine, treat this as an additional
|
||||
> layer.
|
||||
|
||||
Ai-memory's wiki directory is a real git repository. Every consolidation pass
|
||||
writes a checkpoint through libgit2 (see cross-cutting invariant #10 in
|
||||
[`AGENTS.md`](../AGENTS.md) and the wiki design notes in
|
||||
[`docs/ARCHITECTURE.md`](ARCHITECTURE.md)). That means half of a remote-backup
|
||||
setup is already done for you — the wiki has its own commit history and can be
|
||||
pushed to any git remote you own. The rest of this doc is about wiring that
|
||||
up, what else to include, what to exclude, and how to schedule it.
|
||||
|
||||
## What the data dir contains, and what to back up
|
||||
|
||||
`ai-memory status` prints the resolved data dir; per
|
||||
[`docs/deploy.md#backups`](deploy.md#backups) it looks like this:
|
||||
|
||||
```
|
||||
data/
|
||||
├── wiki/ # markdown source of truth (already a git repository)
|
||||
├── raw/ # immutable session-log ledger
|
||||
├── db/ # memory.sqlite (FTS5, entities, embeddings) — DERIVED
|
||||
├── logs/ # daily rolling tracing
|
||||
└── models/ # local embedder weights (redownloadable)
|
||||
```
|
||||
|
||||
Split by durability:
|
||||
|
||||
| Directory | Back up? | Why |
|
||||
|---|---|---|
|
||||
| `wiki/` | **Yes, always.** | Markdown source of truth. Irreplaceable. Already a git repository. |
|
||||
| `raw/` | Yes if you want the full ledger; no if you want a small mirror. | Immutable observation log. Useful for rebuilding session history; not required to keep the wiki working. |
|
||||
| `db/` | **No.** | SQLite index derived from `wiki/`. Rebuildable via `ai-memory reindex`; see [`lifecycle-ops.md#reindex`](lifecycle-ops.md#reindex). Committing a WAL-mode SQLite database to git also wastes space on every write. |
|
||||
| `logs/` | No. | Tracing output; nothing durable. |
|
||||
| `models/` | No. | Local embedder weights, sha256-pinned, redownloaded on first boot per [`local-embeddings.md`](local-embeddings.md). |
|
||||
| `.serve.lock` | No. | Server pidfile. |
|
||||
| `config.toml` and env files under `~/.config/ai-memory/` | Yes, minus secrets. | Provider config is useful for reproducing an install; API-key env files (`*.env`), OAuth tokens (`auth.json`), and private CA bundles must never land in a mirror. |
|
||||
|
||||
If you also need the SQLite index for near-instant recovery (rather than
|
||||
running `reindex` after a restore), pair the git mirror with a periodic
|
||||
`ai-memory backup --to <tarball>` and stash the tarball on a second location
|
||||
(external disk, network share, or on WSL2 the Windows-side filesystem via
|
||||
`/mnt/c/`).
|
||||
|
||||
## Security: what ends up in your wiki
|
||||
|
||||
Read this before configuring a remote — decide what is acceptable to have
|
||||
off your machine, then pick the shape of backup that matches.
|
||||
|
||||
The wiki captures **everything the coding agent sees during a session**
|
||||
that ai-memory's sanitizer classifies as observations to store: user
|
||||
prompts, tool inputs and outputs, page bodies written by consolidation.
|
||||
The sanitizer strips obvious credentials (see `SECURITY.md`), but it
|
||||
cannot prove that every prompt or tool response was safe to store. In
|
||||
practice this means: if any user has ever pasted an API key, a password,
|
||||
a private client name, a customer record, or a file path they consider
|
||||
sensitive into a captured session, it is on disk in the wiki tree, and
|
||||
whatever you push will carry it.
|
||||
|
||||
For any remote target — even a private repository on a service you
|
||||
control — assume an operator or administrator on the receiving side
|
||||
could read the contents. Employer-managed cloud repositories
|
||||
(GitHub Enterprise, GitLab self-hosted, Bitbucket Data Center) may be
|
||||
routinely audited by administrators, and a repository private to you on
|
||||
a shared account is still readable by whoever administers the account.
|
||||
|
||||
**Suggested exclusions**, in addition to the derived-state and
|
||||
secret-file rules already covered above. The mirror repository's
|
||||
`.gitignore` should carry at least:
|
||||
|
||||
```
|
||||
# Provider credentials and OAuth material — never push
|
||||
config/*.env
|
||||
config/*.key
|
||||
config/*.pem
|
||||
config/auth.json
|
||||
config/.secrets
|
||||
|
||||
# Derived index and downloadable state
|
||||
data/models/
|
||||
data/db/
|
||||
data/logs/
|
||||
data/.serve.lock
|
||||
```
|
||||
|
||||
The example `.gitignore` under [`docs/examples/backup/`](examples/backup/README.md)
|
||||
ships this list already; expand it for anything install-specific
|
||||
(host-side certificates, private CA bundles, cloud credential files,
|
||||
site-specific env files).
|
||||
|
||||
**Alternatives when the wiki content is too sensitive to store as text
|
||||
on a remote** — pick whichever fits the threat model:
|
||||
|
||||
- **Encrypted archive to object storage.** `restic` or `borg` snapshots
|
||||
of the data dir to a private object store (S3, Backblaze B2, Wasabi,
|
||||
or a self-hosted MinIO) with a passphrase-derived encryption key. The
|
||||
content leaves the host encrypted; the object-store operator sees
|
||||
ciphertext.
|
||||
- **Selective encryption inside the mirror repo.** `git-crypt` on paths
|
||||
known to carry sensitive material (for example `data/wiki/**/*.md`)
|
||||
keeps the git-remote workflow but encrypts blobs before push. Set up
|
||||
keys before the first commit; retro-encryption of history is manual.
|
||||
- **`age`-encrypted tarball to a second location.** Simpler than
|
||||
`restic`/`borg`, no state file: pipe `tar czf - <data-dir>` through
|
||||
`age -R <recipients>` and drop the ciphertext onto an external disk
|
||||
or a network share.
|
||||
- **Local-only backup, no remote.** Rotate encrypted tarballs onto an
|
||||
external USB drive that lives off-site; give up remote-git
|
||||
convenience for zero third-party exposure.
|
||||
|
||||
If none of the wiki content is more sensitive than what already lives
|
||||
in your project's source repository, a private mirror repo is fine —
|
||||
this section is for the case where it is.
|
||||
|
||||
## Backing up the wiki to a remote git repository
|
||||
|
||||
The wiki is already a git repository. If your only goal is off-site *wiki*
|
||||
storage, adding a remote and pushing is enough:
|
||||
|
||||
```bash
|
||||
data_dir=$(ai-memory status --format=json | jq -r .data_dir)
|
||||
cd "$data_dir/wiki"
|
||||
git remote add origin git@github.com:<you>/ai-memory-wiki.git
|
||||
git push -u origin main
|
||||
```
|
||||
|
||||
Two caveats:
|
||||
|
||||
- The wiki repo is written to by the server through libgit2; do not run
|
||||
interactive git operations (rebases, merges, force-pushes) against it while
|
||||
the server is running. Read-only commands (`git log`, `git show`, `git
|
||||
push`) are safe.
|
||||
- The wiki carries page bodies verbatim, including any content the model
|
||||
synthesized. Treat the remote the same way you treat any project source:
|
||||
private repository, restrictive access controls.
|
||||
|
||||
## Backing up the whole data dir (mirror + push)
|
||||
|
||||
For a full-install backup — wiki, raw ledger, sanitized config — the
|
||||
supported pattern is a scheduled rsync into a mirror git repository, followed
|
||||
by a git push. This avoids running git operations against the live wiki
|
||||
repository (which the server writes to) while still giving you a single
|
||||
remote with the full recoverable state.
|
||||
|
||||
A worked example lives at
|
||||
[`docs/examples/backup/`](examples/backup/README.md) and consists of:
|
||||
|
||||
- `ai-memory-snapshot.sh` — an idempotent rsync + `git commit && git push` +
|
||||
tarball script driven by six environment variables.
|
||||
- `ai-memory-backup.service` — a `systemd --user` oneshot unit that runs it.
|
||||
- `ai-memory-backup.timer` — a daily user timer with a randomised delay and
|
||||
`Persistent=true` so a missed run after a reboot still fires.
|
||||
- `.gitignore` — an exclusion list to drop into the root of the mirror
|
||||
repository so derived state and secrets never land in commits.
|
||||
|
||||
The high-level flow, one invocation per day (adjust the timer for a different
|
||||
cadence):
|
||||
|
||||
1. `rsync -a --delete` the data dir into the mirror checkout, applying an
|
||||
exclude list for `models/`, `db/`, `logs/`, `.serve.lock`, and any glob
|
||||
listed in `SECRET_EXCLUDES` (`*.env`, `auth.json`, `.secrets` by default).
|
||||
2. `rsync -a --delete` the config dir under the same exclude rules so
|
||||
`config.toml` is captured but env files, OAuth tokens, and private CA
|
||||
bundles are not.
|
||||
3. `git add -A && git commit -m "auto: snapshot $(date +%F)"` if anything
|
||||
changed; skip if there is no delta.
|
||||
4. `git push origin main` with a timeout so a hung network does not block the
|
||||
next run.
|
||||
5. `tar czf $TARBALL_DIR/ai-memory-$(date +%F).tar.gz` a copy of the same
|
||||
tree, so the DB and models can be recovered without a `reindex` after a
|
||||
catastrophic disk loss.
|
||||
6. Rotate old tarballs and old script logs on a retention window (30 days by
|
||||
default).
|
||||
|
||||
See the example directory's [README](examples/backup/README.md) for the
|
||||
install-and-enable steps and non-systemd equivalents (macOS launchd, Windows
|
||||
Task Scheduler via WSL, Docker sidecar).
|
||||
|
||||
## Restoring from a mirror repository
|
||||
|
||||
The mirror repository is a copy of the data dir minus derived state.
|
||||
Recovering the full ai-memory install from it is two steps:
|
||||
|
||||
1. Clone the mirror onto the target host, then rsync the `data/` and
|
||||
`config/` subtrees into place under the ai-memory data dir and config dir
|
||||
respectively.
|
||||
2. Rebuild the SQLite index from the recovered markdown:
|
||||
[`lifecycle-ops.md#reindex`](lifecycle-ops.md#reindex) walks through this.
|
||||
Sessions, observations, handoffs, users/tokens, audit rows, and
|
||||
embeddings live only in the DB and are not rebuilt by `reindex` — for
|
||||
those, restore from the paired tarball via `ai-memory restore --from
|
||||
<tarball> --data-dir <path> --force`.
|
||||
|
||||
## Security posture
|
||||
|
||||
Remote sync is your own channel, not ai-memory's — this is called out
|
||||
explicitly in [`SECURITY.md`](../SECURITY.md) ("Remote sync security" under
|
||||
out-of-scope for v1). What that means in practice:
|
||||
|
||||
- **Private repository, always.** ai-memory captures prompts, tool calls, and
|
||||
synthesized page bodies verbatim — the same content you would treat as
|
||||
project source.
|
||||
- **Least-privilege push credential.** Prefer an SSH deploy key or a
|
||||
fine-grained personal-access token scoped to the mirror repository only.
|
||||
Rotate on the same cadence you rotate other long-lived push credentials.
|
||||
- **Explicit secret exclusion.** The example script rejects any file matching
|
||||
`SECRET_EXCLUDES` (`*.env`, `auth.json`, `.secrets` by default), and the
|
||||
suggested mirror-repo `.gitignore` matches the same set. Add anything else
|
||||
install-specific — private CA bundles, host-side certificates, cloud
|
||||
credential files — to both lists.
|
||||
- **Encryption at rest is out of scope for ai-memory.** If your threat model
|
||||
requires it, either put the mirror on an encrypted remote (`git-crypt`,
|
||||
GitLab's group-level encryption, a self-hosted Gitea over TLS with disk
|
||||
encryption) or archive the tarballs with `gpg --encrypt` before dropping
|
||||
them at `TARBALL_DIR`.
|
||||
|
||||
## Related documents
|
||||
|
||||
- [`docs/lifecycle-ops.md`](lifecycle-ops.md) — `ai-memory backup`, `restore`,
|
||||
`restore-page`, `reset`, `reindex` reference.
|
||||
- [`docs/deploy.md#backups`](deploy.md#backups) — the on-box tarball routine
|
||||
and the `scp`-to-laptop idiom for Docker deployments.
|
||||
- [`docs/airgapped-install.md`](airgapped-install.md) — offline installs,
|
||||
including the callout that remote git sync is user-owned plumbing.
|
||||
- [`SECURITY.md`](../SECURITY.md) — the "Remote sync security" out-of-scope
|
||||
note this doc expands into a walkthrough.
|
||||
- [`docs/local-embeddings.md`](local-embeddings.md) — how `models/` is
|
||||
fetched and pinned (why it does not need to be in the mirror).
|
||||
@@ -236,6 +236,11 @@ scp "$SERVER:$DEPLOY_DIR/data/snapshot-$(date +%F).tar.gz" ./backups/
|
||||
The `ai-memory backup` command uses SQLite's online backup API so
|
||||
writes during the snapshot are coherent.
|
||||
|
||||
For a remote-first backup pattern (rsync the data dir into a git mirror on a
|
||||
schedule, push to a private repository, paired with a tarball for the DB
|
||||
and models), see [`docs/backup.md`](backup.md) and the worked example under
|
||||
[`docs/examples/backup/`](examples/backup/README.md).
|
||||
|
||||
## Sharing one server between people or harnesses
|
||||
|
||||
A deployed server is the supported way to share a project — between teammates,
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
# Suggested .gitignore for a mirror repository that receives ai-memory
|
||||
# snapshots. Copy this file into the root of your `$REPO_DIR` before the
|
||||
# first `git add`.
|
||||
#
|
||||
# Rules of thumb:
|
||||
# - Keep the wiki (markdown source of truth) and any per-project _rules /
|
||||
# _prompts pages, since those are irreplaceable.
|
||||
# - Keep raw/ if you want the untouched observation ledger, drop it if you
|
||||
# want a small repository.
|
||||
# - Never commit derived state (db/) or downloaded weights (models/), and
|
||||
# never commit secrets.
|
||||
|
||||
# Secrets — belt and braces alongside the rsync SECRET_EXCLUDES list.
|
||||
config/*.env
|
||||
config/**/*.env
|
||||
config/auth.json
|
||||
config/**/auth.json
|
||||
config/.secrets
|
||||
config/**/.secrets
|
||||
|
||||
# Server pidfile.
|
||||
data/.serve.lock
|
||||
|
||||
# Derived SQLite index (rebuildable via `ai-memory reindex`).
|
||||
data/db/
|
||||
|
||||
# Local embedder weights (redownloadable on next boot; sha256-pinned).
|
||||
data/models/
|
||||
|
||||
# Rotated tracing output.
|
||||
data/logs/
|
||||
|
||||
# Any local scratch produced by the snapshot script itself.
|
||||
*.tmp
|
||||
*.swp
|
||||
@@ -0,0 +1,80 @@
|
||||
# Remote git backup example
|
||||
|
||||
Sample scripts + unit files that push an ai-memory install to a remote git
|
||||
repository on a daily cadence, keeping a tarball on a second location for the
|
||||
SQLite index and models cache that the mirror repo deliberately excludes.
|
||||
|
||||
See [`docs/backup.md`](../../backup.md) for the walkthrough. The pieces in this
|
||||
directory are:
|
||||
|
||||
- `ai-memory-snapshot.sh` — the rsync + git push + tarball loop. Configurable
|
||||
through six environment variables (`REPO_DIR`, `DATA_SRC`, `CFG_SRC`,
|
||||
`TARBALL_DIR`, `LOG_DIR`, `SECRET_EXCLUDES`) with sensible defaults for a
|
||||
single-user Linux/WSL/macOS install.
|
||||
- `ai-memory-backup.service` — a `systemd --user` oneshot that runs the
|
||||
script. `%h` expands to the user's home directory.
|
||||
- `ai-memory-backup.timer` — a daily user timer with a small randomised delay
|
||||
and `Persistent=true` so the missed run after a reboot still fires.
|
||||
- `.gitignore` — the exclusion list to drop into the root of the mirror
|
||||
repository. Belt-and-braces to the rsync/tar exclude lists in the script
|
||||
itself.
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
# 1. Create a private repository on your git host (e.g. github.com), then:
|
||||
mkdir -p ~/ai-memory-backup
|
||||
cd ~/ai-memory-backup
|
||||
git init
|
||||
git remote add origin git@github.com:<you>/ai-memory-backup.git
|
||||
cp .../docs/examples/backup/.gitignore .
|
||||
git add .gitignore
|
||||
git commit -m "initial: ignore secrets, DB, models, logs"
|
||||
git branch -M main
|
||||
# push once so `snapshot.sh` has a remote to push to
|
||||
git push -u origin main
|
||||
|
||||
# 2. Drop the snapshot script somewhere on $PATH.
|
||||
install -Dm755 \
|
||||
.../docs/examples/backup/ai-memory-snapshot.sh \
|
||||
~/.local/bin/ai-memory-snapshot.sh
|
||||
|
||||
# 3. Install the unit files.
|
||||
install -Dm644 \
|
||||
.../docs/examples/backup/ai-memory-backup.service \
|
||||
~/.config/systemd/user/ai-memory-backup.service
|
||||
install -Dm644 \
|
||||
.../docs/examples/backup/ai-memory-backup.timer \
|
||||
~/.config/systemd/user/ai-memory-backup.timer
|
||||
|
||||
# 4. Enable + start.
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now ai-memory-backup.timer
|
||||
|
||||
# 5. Trigger one snapshot to confirm the plumbing works.
|
||||
systemctl --user start ai-memory-backup.service
|
||||
journalctl --user -u ai-memory-backup.service --since '5 minutes ago'
|
||||
```
|
||||
|
||||
## Non-systemd equivalents
|
||||
|
||||
The script is a plain bash program; anything that runs a shell command on a
|
||||
schedule can drive it:
|
||||
|
||||
- **macOS launchd:** wrap `ai-memory-snapshot.sh` in a `.plist` under
|
||||
`~/Library/LaunchAgents/` with a `StartCalendarInterval` block.
|
||||
- **Windows Task Scheduler:** run the script inside WSL via `wsl -e bash -lc
|
||||
'ai-memory-snapshot.sh'` on a daily trigger. On native Windows use the
|
||||
PowerShell equivalent of the rsync + `git add/commit/push` + tar sequence.
|
||||
- **Docker sidecar:** a small cron container mounted at the ai-memory data
|
||||
volume, running the same script.
|
||||
|
||||
## Security note
|
||||
|
||||
The remote is your own channel, not ai-memory's. Follow the existing
|
||||
[`SECURITY.md`](../../../SECURITY.md) guidance ("Remote sync security"): use a
|
||||
private repository, a fine-grained token scoped to that one repo (or an SSH
|
||||
deploy key), and rotate credentials the same way you rotate any other
|
||||
long-lived push credential. The script never commits files matched by
|
||||
`SECRET_EXCLUDES` (default: `*.env`, `auth.json`, `.secrets`); the mirror
|
||||
repo's `.gitignore` catches anything the rsync exclude missed.
|
||||
@@ -0,0 +1,22 @@
|
||||
[Unit]
|
||||
Description=ai-memory snapshot + push to remote git
|
||||
Documentation=file://%h/.local/share/ai-memory/docs/backup.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
# Point ExecStart at wherever you dropped ai-memory-snapshot.sh — this path
|
||||
# is only a sample. Adjust the environment overrides if the defaults in the
|
||||
# script do not match your install.
|
||||
ExecStart=%h/.local/bin/ai-memory-snapshot.sh
|
||||
# Environment=REPO_DIR=%h/ai-memory-backup
|
||||
# Environment=DATA_SRC=%h/.local/share/ai-memory
|
||||
# Environment=CFG_SRC=%h/.config/ai-memory
|
||||
# Environment=TARBALL_DIR=%h/backups/ai-memory
|
||||
Nice=10
|
||||
IOSchedulingClass=best-effort
|
||||
IOSchedulingPriority=7
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=Daily ai-memory snapshot
|
||||
|
||||
[Timer]
|
||||
OnCalendar=daily
|
||||
AccuracySec=15min
|
||||
Persistent=true
|
||||
RandomizedDelaySec=5min
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env bash
|
||||
# ai-memory-snapshot.sh
|
||||
#
|
||||
# Daily snapshot of an ai-memory single-user install:
|
||||
# 1. rsync the mutable data dir into a git-tracked mirror checkout,
|
||||
# excluding derived state and secrets (see .gitignore in this dir).
|
||||
# 2. commit + push the mirror to a remote git repository when anything
|
||||
# changed.
|
||||
# 3. write a compressed tarball of the same tree to a second location
|
||||
# (a mounted external disk, a network share, or on WSL2 the Windows-side
|
||||
# filesystem) so the SQLite index can be recovered from disk even if the
|
||||
# remote repository is unreachable.
|
||||
#
|
||||
# Assumptions:
|
||||
# - You already ran `git init && git remote add origin <url>` inside
|
||||
# $REPO_DIR and staged the .gitignore shipped alongside this file.
|
||||
# - The user running this script has an ssh-agent or credential helper set
|
||||
# up so `git push` is non-interactive.
|
||||
# - $DATA_SRC is the ai-memory data dir (`ai-memory status` prints it) and
|
||||
# $CFG_SRC is where your provider env-file lives.
|
||||
#
|
||||
# Configure these four variables for your host, or export them from the
|
||||
# systemd unit's Environment= lines. Sample values are commented out.
|
||||
: "${REPO_DIR:=$HOME/ai-memory-backup}" # local mirror checkout, `git init`'d and pointed at a remote
|
||||
: "${DATA_SRC:=$HOME/.local/share/ai-memory}" # ai-memory data dir
|
||||
: "${CFG_SRC:=$HOME/.config/ai-memory}" # ai-memory config dir (may hold provider env files)
|
||||
: "${TARBALL_DIR:=$HOME/backups/ai-memory}" # second-location tarball drop (external disk, network share, ...)
|
||||
: "${LOG_DIR:=$HOME/logs/ai-memory-snapshot}" # rotated per run
|
||||
: "${TARBALL_RETENTION_DAYS:=30}" # how long to keep tarballs
|
||||
: "${LOG_RETENTION_DAYS:=30}" # how long to keep run logs
|
||||
: "${SECRET_EXCLUDES:=*.env auth.json .secrets}" # glob patterns for files never to sync
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
stamp=$(date +%Y-%m-%d)
|
||||
log="$LOG_DIR/ai-memory-snapshot_$(date +%Y%m%d-%H%M%S).log"
|
||||
mkdir -p "$LOG_DIR" "$TARBALL_DIR"
|
||||
exec >"$log" 2>&1
|
||||
|
||||
echo "=== $(date -Iseconds) ai-memory snapshot start ==="
|
||||
echo "data_src=$DATA_SRC"
|
||||
echo "cfg_src=$CFG_SRC"
|
||||
echo "repo_dir=$REPO_DIR"
|
||||
echo "tarball_dir=$TARBALL_DIR"
|
||||
|
||||
if [ ! -d "$DATA_SRC" ]; then
|
||||
echo "ERROR: $DATA_SRC does not exist. Aborting." >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ ! -d "$REPO_DIR/.git" ]; then
|
||||
echo "ERROR: $REPO_DIR is not a git repository (run 'git init' and 'git remote add origin <url>' first)." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Build a comma-free rsync exclude list. The .gitignore in the mirror repo
|
||||
# also keeps derived state out of commits, but excluding here keeps the rsync
|
||||
# working set small.
|
||||
rsync_excludes=(
|
||||
--exclude=.git/
|
||||
--exclude=models/ # embedder weights, redownloadable
|
||||
--exclude=db/ # SQLite index, derived from wiki via `ai-memory reindex`
|
||||
--exclude=logs/ # rotated tracing output
|
||||
--exclude=.serve.lock # server pidfile
|
||||
)
|
||||
for glob in $SECRET_EXCLUDES; do
|
||||
rsync_excludes+=(--exclude="$glob")
|
||||
done
|
||||
|
||||
echo "--- rsync data ---"
|
||||
rsync -a --delete "${rsync_excludes[@]}" "$DATA_SRC/" "$REPO_DIR/data/"
|
||||
|
||||
echo "--- rsync config (secrets excluded) ---"
|
||||
config_excludes=()
|
||||
for glob in $SECRET_EXCLUDES; do
|
||||
config_excludes+=(--exclude="$glob")
|
||||
done
|
||||
rsync -a --delete "${config_excludes[@]}" "$CFG_SRC/" "$REPO_DIR/config/"
|
||||
|
||||
cd "$REPO_DIR"
|
||||
git add -A
|
||||
|
||||
if git diff --cached --quiet; then
|
||||
echo "no changes since last snapshot; skipping commit + push"
|
||||
else
|
||||
git commit -m "auto: snapshot $stamp" | tail -3
|
||||
echo "--- push ---"
|
||||
timeout 180 git push origin "$(git rev-parse --abbrev-ref HEAD)" 2>&1 | tail -10
|
||||
fi
|
||||
|
||||
echo "--- tarball to second location ---"
|
||||
tarball="$TARBALL_DIR/ai-memory-$stamp.tar.gz"
|
||||
tar_excludes=(
|
||||
--exclude=.git
|
||||
--exclude=models
|
||||
--exclude=db
|
||||
--exclude=logs
|
||||
--exclude=.serve.lock
|
||||
)
|
||||
for glob in $SECRET_EXCLUDES; do
|
||||
tar_excludes+=(--exclude="$glob")
|
||||
done
|
||||
# Use paths relative to $HOME so the archive is portable across usernames.
|
||||
tar czf "$tarball" "${tar_excludes[@]}" \
|
||||
-C "$HOME" \
|
||||
"${DATA_SRC#$HOME/}" \
|
||||
"${CFG_SRC#$HOME/}"
|
||||
echo "tarball: $(ls -lh "$tarball" | awk '{print $5, $9}')"
|
||||
|
||||
echo "--- rotate tarballs (retention: ${TARBALL_RETENTION_DAYS}d) ---"
|
||||
find "$TARBALL_DIR" -maxdepth 1 -name 'ai-memory-*.tar.gz' -type f \
|
||||
-mtime "+${TARBALL_RETENTION_DAYS}" -delete -print
|
||||
|
||||
echo "--- rotate logs (retention: ${LOG_RETENTION_DAYS}d) ---"
|
||||
find "$LOG_DIR" -maxdepth 1 -name 'ai-memory-snapshot_*.log' -type f \
|
||||
-mtime "+${LOG_RETENTION_DAYS}" -delete -print
|
||||
|
||||
echo "=== $(date -Iseconds) ai-memory snapshot done ==="
|
||||
Reference in New Issue
Block a user