fix(security): create memory stores and other state files owner-only (#3848)

## Why

The memory databases hold facts extracted from conversations, together
with the user id they belong to. These files were created at the process
umask, which is `0644` on a stock host, so any other local account could
read them. The same was true of the FTS, graph and sqlite-vec stores,
the HNSW index file and metadata dump (vectors, raw embeddings and
memory metadata), the licence validation cache and the native memory
directory. The CCR store already did this right with its own private
copy of the logic. This is finding 02-F2.

## What changes

`headroom/fileperms.py` gains shared helpers, and every core call site
uses them:

- `ensure_private_file(path, what=)`: makes a path an owner-only regular
file before a by-path opener such as sqlite touches it. New files are
created `0600` with `O_EXCL`. An existing file is narrowed through an
`O_NOFOLLOW` descriptor. A symlink or non-regular file raises
`PermissionError` (fails closed). This is the CCR store's existing
logic, moved here.
- `connect_private_sqlite(path, what=, **kw)`: `sqlite3.connect` through
`ensure_private_file`. Only genuine in-memory databases pass straight
through (`:memory:`, an empty name, and with `uri=True` `file::memory:`,
an empty URI path or `mode=memory`). A file-backed `file:` URI has its
percent-decoded path (relative or absolute, `localhost` authority; an
undecodable `%XX` kept as that raw byte, as sqlite does) made private
first; a remote authority is refused; a missing database under
`mode=ro`/`rw` is left for sqlite to report. With `uri=False` a
`file:`-prefixed name is a literal path and is hardened like any other,
except on sqlite builds compiled with `SQLITE_USE_URI` (Debian, the
official `python` Docker images), which parse it as a URI anyway; the
helper follows the build. Existing `-wal` / `-shm` / `-journal` sidecars
(left by a crash under an older run's mode) are made owner-only through
`ensure_private_file` before sqlite opens the store: narrowed in place
with contents kept, a symlink refused. Missing sidecars are not
pre-created.
- `private_dir(path)`: creates the directory and makes the leaf `0700`.
A symlinked leaf is left alone.
- `open_owner_only(path, "w")`: write mode added (it was append-only).
Read modes raise `ValueError`. A `newline` argument is passed through to
`open()`.

Call sites:
- `cache/backends/sqlite.py`: the CCR store's own copy is replaced by
`ensure_private_file`. Same behaviour.
- Memory adapters `sqlite.py`, `fts5.py`, `sqlite_graph.py`,
`sqlite_vector.py` and `traffic_learner.py`: `connect_private_sqlite`.
- `memory/adapters/hnsw.py`: the `.hnsw` index path goes through
`ensure_private_file` before hnswlib writes it (hnswlib opens by path;
its truncating write keeps the `0600` mode), and the metadata dump is
written with `open_owner_only(..., "w")`.
- `proxy/memory_handler.py`: native memory directory created with
`private_dir` (`0700`).
- `telemetry/reporter.py`: licence cache written with
`open_owner_only(..., "w", newline="\n")`, so it stays LF on Windows
(#3698).

sqlite creates new `-wal` / `-shm` / `-journal` sidecars with the main
file's mode (checked under umask 022 on macOS and Linux), so fresh ones
are `0600`. Files left wide by an older version, including
crash-leftover sidecars, are narrowed on the next open.

Not changed on purpose: the workspace root `~/.headroom` (an operator
may have set its mode), and files holding only token counts or version
strings (savings ledger, update-check cache, TOIN state).
`subscription_state.json` is already written `0600` via `tempfile`.

## Behaviour change

- Memory store files and the licence cache are created `0600`, and the
native memory directory `0700`. Existing wider files are narrowed on
first open.
- A memory database or HNSW index file reached through a symlink (for
example a symlinked `~/.headroom/headroom_memory.db`) now raises
`PermissionError` instead of opening. The CCR store already behaved this
way.

## Tests

`tests/test_private_state_files.py` (POSIX only, run under a forced
`umask 022`):
- Helpers: private from birth; an existing wide file is narrowed with
content intact; a symlink is refused and its target untouched; a
directory is refused; a failed `fchmod` fails closed; sqlite sidecars
are private; in-memory databases (`:memory:`, empty name,
`file::memory:`, `mode=memory`) create nothing; file-backed URIs
(relative percent-encoded, absolute, `localhost` authority) and a
`uri=False` `file:literal.db` (literal or URI, per the sqlite build) are
created `0600`; a `file:secret%FF.db` URI secures the raw-byte name
(Linux; skipped where the filesystem rejects non-UTF-8 names);
crash-leftover `0644` `-wal`/`-shm` and `-journal` files are `0600`
before sqlite writes, keep their uncheckpointed rows, and hold the new
private row; missing sidecars are not pre-created; a symlinked
`-wal`/`-shm`/`-journal` is refused and its target untouched; a
`mode=ro` URI does not create a missing database; a remote-authority URI
is refused; `private_dir` makes new and existing directories `0700` and
leaves a symlinked one alone; `"w"` truncates, `"a"` appends, read modes
are rejected.
- Call sites: sqlite memory store (fresh and pre-existing `0644`), FTS5,
graph, sqlite-vec (skipped if the extension is absent), HNSW `.hnsw` and
`.meta` files (fresh, and a wide `.hnsw` narrowed on re-save) plus
refusal of a planted `.hnsw` symlink with its target untouched (skipped
without `hnswlib`), native memory directory, licence cache, and the CCR
backend through the shared helper.


`tests/test_usage_reporter_snapshot.py::test_license_cache_is_written_with_lf`
(`windows_newline`, runs on Windows CI too): the licence cache write
pins `newline="\n"`.

On `main`, the memory, FTS, graph databases and `license.json` are
created `0644` under `umask 022`; on this branch they are `0600`.
Existing suites for every touched store pass.

Not tested: Windows (mode bits are not claimed there; see the module
docstring) and remote memory backends.

## Follow-ups, not in this PR

- Extensions that keep their own SQLite state can adopt
`connect_private_sqlite` / `open_owner_only`.
- `memory/easy.py` and `memory/storage_router.py` still create the
database parent at the umask. That parent is the workspace root, which
is left alone on purpose.

Revised 2026-10-04 after review: crash-leftover SQLite
`-wal`/`-shm`/`-journal` sidecars are now narrowed (symlinks refused)
before the store opens, the helper follows sqlite's real file name on
`SQLITE_USE_URI` builds and for undecodable `%XX` escapes, and main is
merged.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3848"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Tejas Chopra
2026-10-05 15:26:54 -05:00
committed by GitHub
co-authored by Claude Fable 5.1 JD Davis
parent 119d1a1939
commit 5119b6eb54
12 changed files with 666 additions and 79 deletions
+8 -59
View File
@@ -22,13 +22,14 @@ import json
import logging
import os
import sqlite3
import stat
import threading
import time
from dataclasses import asdict, fields
from pathlib import Path
from typing import TYPE_CHECKING, Any
from ...fileperms import ensure_private_file
if TYPE_CHECKING:
from ..compression_store import CompressionEntry
@@ -90,65 +91,13 @@ class SQLiteBackend:
The originals stored here can contain sensitive tool output (file
contents, command output). ``mkdir`` created the parent at the umask
default (typically world-traversable ``0o755``), and sqlite would
otherwise create the db file itself at the umask default too — leaving it
world/group readable. This creates the file with an explicit ``0o600``
mode before ``sqlite3.connect`` runs (a *new* db is private from birth;
sqlite treats the empty file as a fresh database), or narrows a
pre-existing db.
A store of raw tool output must not be opened world-readable, so a
failure to create or narrow it privately **raises** rather than silently
proceeding to open a wide file.
The existing-file path is symlink-race resistant. ``O_EXCL`` on the
create refuses to reuse an attacker-planted file or symlink. When the
file already exists it is re-opened with ``O_NOFOLLOW`` and narrowed
through that descriptor (``fstat`` to confirm a regular file, ``fchmod``
to set the mode) rather than by re-resolving the path: a symlink is
refused at the ``open`` syscall, and operating on the verified
descriptor closes the check-then-chmod TOCTOU that a ``chmod(path)``
(which follows symlinks) would leave open. On Windows POSIX permission
bits and ``O_NOFOLLOW`` do not apply, so there is nothing to narrow.
default and sqlite would create the db at the umask default too, so the
file is created (or an existing one narrowed) through
:func:`headroom.fileperms.ensure_private_file`, which is symlink-race
resistant and raises :class:`PermissionError` rather than opening a
wide file. See that function for the exact guarantee per platform.
"""
# Fast path: create the file ourselves. O_EXCL (plus O_NOFOLLOW where
# available) refuses to reuse an existing file or follow a planted
# symlink, so a brand-new db is private from birth at 0o600.
create_flags = os.O_CREAT | os.O_EXCL | os.O_WRONLY
if hasattr(os, "O_NOFOLLOW"):
create_flags |= os.O_NOFOLLOW
try:
os.close(os.open(path, create_flags, 0o600))
return
except FileExistsError:
pass
# The file already exists. Without POSIX no-follow/fd primitives
# (Windows) permission bits do not apply and there is no symlink-race
# hardening to perform; sqlite opens by path as before.
if not (hasattr(os, "O_NOFOLLOW") and hasattr(os, "fchmod")):
return
# Narrow the existing file, but only after proving — through a single
# no-follow descriptor — that it is a real regular file. A symlink is
# refused by O_NOFOLLOW (fail closed), and chmod-ing the descriptor
# rather than the path means an attacker cannot swap the inode we
# verified between the check and the narrow.
try:
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW)
except OSError as exc:
raise PermissionError(
f"refusing to open CCR store at {path}: not a regular file "
f"(symlink or open error: {exc})"
) from exc
try:
if not stat.S_ISREG(os.fstat(fd).st_mode):
raise PermissionError(f"refusing to open CCR store at {path}: not a regular file")
# os.fchmod is POSIX-only (guarded by the hasattr check above); the
# ignore keeps type-checking clean on Windows where it is absent.
os.fchmod(fd, 0o600) # type: ignore[attr-defined]
finally:
os.close(fd)
ensure_private_file(path, what="CCR store")
def _open(self) -> sqlite3.Connection:
self._ensure_private(self._path)
+185 -7
View File
@@ -3,8 +3,22 @@
Headroom's runtime log (``~/.headroom/logs/proxy-<port>.log``) and the optional
JSONL request log can both carry verbatim request and response content: wire
debug dumps, ``--log-messages`` bodies, and — when an operator opts in — CCR
payload previews. None of that should be created at the process umask, which on
a stock developer machine means ``0644``: world-readable.
payload previews. The CCR retrieval store holds verbatim tool output, and the
memory databases hold facts extracted from conversations together with the
user id they belong to. None of that should be created at the process umask,
which on a stock developer machine means ``0644``: world-readable.
This module is the one place that knows how to create such a file privately,
so every store uses the same rules and a reviewer can read the guarantee off
one file. Extensions that keep their own state (Shield, Foresight, Kiro, the
router) should call these helpers rather than ``sqlite3.connect`` /
``Path.write_text`` directly:
* :func:`open_owner_only` — open a log or text file for append or write.
* :func:`ensure_private_file` — make a path a private regular file *before*
a library that opens by path (sqlite) touches it.
* :func:`connect_private_sqlite` — ``sqlite3.connect`` through the above.
* :func:`private_dir` — create a directory that will hold such files.
Scope of the guarantee — read this before citing it in a threat model:
@@ -33,20 +47,27 @@ a kernel-enforced one.
from __future__ import annotations
import functools
import os
import sqlite3
import stat
from pathlib import Path
from typing import IO, Any
from urllib.parse import parse_qs, unquote, urlsplit
#: Mode for every runtime file Headroom creates that may hold request content.
OWNER_ONLY_MODE = 0o600
#: Mode for directories that hold such files.
OWNER_ONLY_DIR_MODE = 0o700
#: ``True`` only where the mode bits above actually decide who can read the
#: file. See the module docstring for why Windows is excluded.
OWNER_ONLY_SUPPORTED = os.name == "posix"
def _open_flags() -> int:
flags = os.O_CREAT | os.O_WRONLY | os.O_APPEND
def _open_flags(*, truncate: bool = False) -> int:
flags = os.O_CREAT | os.O_WRONLY | (os.O_TRUNC if truncate else os.O_APPEND)
# Refuse to open through a symlink where the platform can enforce it, so a
# planted link cannot redirect either the write or the chmod. Absent on
# Windows, where getattr() leaves the flag out.
@@ -97,18 +118,21 @@ def open_owner_only(
*,
encoding: str | None = None,
errors: str | None = None,
newline: str | None = None,
) -> IO[Any]:
"""Open *path* for append, creating it owner-only.
"""Open *path* for append (``"a"``) or write (``"w"``), creating it owner-only.
Raises ``OSError`` — which every caller already treats as "logging is
unavailable, carry on" — if the path cannot be opened, including when it is
a symlink on a platform with ``O_NOFOLLOW``. Failing closed is deliberate:
a redirected sensitive log is worse than no log.
"""
fd = os.open(path, _open_flags(), OWNER_ONLY_MODE)
if mode[:1] not in ("a", "w"):
raise ValueError(f"open_owner_only: mode must start with 'a' or 'w', got {mode!r}")
fd = os.open(path, _open_flags(truncate=mode.startswith("w")), OWNER_ONLY_MODE)
try:
restrict_fd_to_owner(fd)
return open(fd, mode, encoding=encoding, errors=errors, closefd=True)
return open(fd, mode, encoding=encoding, errors=errors, newline=newline, closefd=True)
except BaseException:
try:
os.close(fd)
@@ -116,3 +140,157 @@ def open_owner_only(
# open() can have taken and closed the descriptor on its way out.
pass
raise
def ensure_private_file(path: str | os.PathLike[str], *, what: str = "file") -> None:
"""Make *path* an owner-only regular file before a by-path opener touches it.
For libraries that open files by path and create them at the umask —
sqlite above all. Creating the file here first, with an explicit
``0o600`` mode, means a *new* database is private from birth (sqlite
treats an empty file as a fresh database, and creates its ``-wal`` /
``-shm`` sidecars with the same mode as the main file). A pre-existing
file left wide by an earlier run is narrowed.
Fails **closed**: a store of conversation or tool content must not be
opened world-readable, so a failure to create or narrow the file privately
raises :class:`PermissionError` rather than proceeding to open a wide
file. The existing-file path is symlink-race resistant — ``O_EXCL`` on the
create refuses to reuse a planted file or symlink; an existing file is
re-opened with ``O_NOFOLLOW`` and narrowed through that descriptor
(``fstat`` to confirm a regular file, ``fchmod`` to set the mode) rather
than by re-resolving the path, which closes the check-then-chmod TOCTOU a
``chmod(path)`` would leave open. On Windows POSIX mode bits and
``O_NOFOLLOW`` do not apply, so there is nothing to narrow (see the module
docstring for what is and is not claimed there).
*what* names the store in the error message.
"""
create_flags = os.O_CREAT | os.O_EXCL | os.O_WRONLY
if hasattr(os, "O_NOFOLLOW"):
create_flags |= os.O_NOFOLLOW
try:
os.close(os.open(path, create_flags, OWNER_ONLY_MODE))
return
except FileExistsError:
pass
if not (hasattr(os, "O_NOFOLLOW") and hasattr(os, "fchmod")):
return
try:
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW)
except OSError as exc:
raise PermissionError(
f"refusing to open {what} at {path}: not a regular file (symlink or open error: {exc})"
) from exc
try:
if not stat.S_ISREG(os.fstat(fd).st_mode):
raise PermissionError(f"refusing to open {what} at {path}: not a regular file")
# os.fchmod is POSIX-only (guarded by the hasattr check above); the
# ignore keeps type-checking clean on Windows where it is absent.
os.fchmod(fd, OWNER_ONLY_MODE) # type: ignore[attr-defined]
finally:
os.close(fd)
@functools.cache
def _sqlite_always_uri() -> bool:
"""Whether this sqlite reads ``file:`` names as URIs even without ``uri=True``.
True for builds compiled with ``SQLITE_USE_URI`` — Debian's, and so the
official ``python`` Docker images.
"""
conn = sqlite3.connect(":memory:")
try:
options = {row[0] for row in conn.execute("PRAGMA compile_options")}
finally:
conn.close()
return bool(options & {"USE_URI", "USE_URI=1"})
def _sqlite_file_path(path: str | os.PathLike[str], *, uri: bool) -> str | None:
"""The file sqlite will create or open for *path*; ``None`` if there is none.
Mirrors sqlite's own rules. Without ``uri=True`` the name is a literal
path, even one that starts with ``file:``, unless this sqlite was built to
always parse URIs (:func:`_sqlite_always_uri`). Parsed as a URI, ``file:`` names
its percent-decoded path (relative to the working directory unless
absolute), and is in-memory when that path is empty or ``:memory:`` or the
query says ``mode=memory``. ``mode=ro`` / ``mode=rw`` never create the
file, so a missing one is left for sqlite to report.
"""
text = os.fspath(path)
if text in ("", ":memory:"):
return None
if not (text.startswith("file:") and (uri or _sqlite_always_uri())):
return text
if not OWNER_ONLY_SUPPORTED:
# No mode bits to set (module docstring), and Windows URIs carry a
# drive letter sqlite strips its own way: pass them through.
return None
parts = urlsplit(text)
# surrogateescape keeps an undecodable %XX byte as that raw byte, which is
# the name sqlite opens; the default "replace" would secure a different one.
file_path = unquote(parts.path, errors="surrogateescape")
mode = parse_qs(parts.query).get("mode", [""])[-1]
if file_path in ("", ":memory:") or mode == "memory":
return None
if parts.netloc not in ("", "localhost"):
raise PermissionError(f"refusing SQLite URI with a remote authority: {text}")
if mode in ("ro", "rw") and not os.path.lexists(file_path):
return None
return file_path
#: Files sqlite keeps next to a database and reopens by name.
_SQLITE_SIDECAR_SUFFIXES = ("-wal", "-shm", "-journal")
def connect_private_sqlite(
path: str | os.PathLike[str],
*,
what: str = "database",
**connect_kwargs: Any,
) -> sqlite3.Connection:
"""``sqlite3.connect`` that first makes the database file owner-only.
Drop-in for ``sqlite3.connect(str(path), **kwargs)`` at every store that
holds conversation-derived content. In-memory databases are passed
straight through; a file-backed ``file:`` URI (``uri=True``) has its
underlying file made private like a plain path. Existing ``-wal`` /
``-shm`` / ``-journal`` sidecars are narrowed too. The parent directory must
already exist; create it with :func:`private_dir` if it is dedicated to
this store.
"""
file_path = _sqlite_file_path(path, uri=bool(connect_kwargs.get("uri")))
if file_path is not None:
ensure_private_file(file_path, what=what)
# A crash can leave -wal/-shm/-journal sidecars behind with the mode
# of an older, unhardened run, and sqlite reuses them as they are.
# Narrow any that exist (refusing symlinks) before sqlite writes to
# them. Missing ones are left alone: sqlite creates them with the
# main file's mode, which is now 0600.
for suffix in _SQLITE_SIDECAR_SUFFIXES:
sidecar = file_path + suffix
if os.path.lexists(sidecar):
ensure_private_file(sidecar, what=f"{what} {suffix[1:]} file")
# ``**connect_kwargs: Any`` makes mypy type the call as ``Any``; the
# annotation pins it back to the real return type.
conn: sqlite3.Connection = sqlite3.connect(os.fspath(path), **connect_kwargs)
return conn
def private_dir(path: str | os.PathLike[str]) -> Path:
"""Create *path* (and missing parents) and make the leaf directory ``0o700``.
For a directory that exists only to hold Headroom's sensitive files (the
native memory directory); an existing directory left wider by an earlier
run is narrowed. A symlink at the leaf is left alone: the target is not
ours to re-permission. Returns the path.
"""
target = Path(path)
target.mkdir(parents=True, exist_ok=True)
if OWNER_ONLY_SUPPORTED and not target.is_symlink():
os.chmod(target, OWNER_ONLY_DIR_MODE)
return target
+2 -1
View File
@@ -14,6 +14,7 @@ from dataclasses import dataclass, field
from pathlib import Path
from typing import TYPE_CHECKING, Any
from ...fileperms import connect_private_sqlite
from ..models import Memory
from ..ports import TextFilter, TextSearchResult
@@ -75,7 +76,7 @@ class FTS5TextIndex:
Commits on clean exit, rolls back on exception, and always closes
the connection -- callers use ``with self._get_conn() as conn:``.
"""
conn = sqlite3.connect(str(self.db_path))
conn = connect_private_sqlite(self.db_path, what="memory text index")
conn.row_factory = sqlite3.Row
try:
with conn:
+8 -2
View File
@@ -23,6 +23,7 @@ from typing import TYPE_CHECKING, Any
import numpy as np
from ...fileperms import ensure_private_file, open_owner_only
from ..models import Memory, ScopeLevel, normalize_entity_refs
from ..ports import VectorFilter, VectorSearchResult
@@ -812,8 +813,11 @@ class HNSWVectorIndex:
path = Path(path)
with self._lock:
# Save HNSW index
# Save HNSW index. hnswlib opens the path itself and would create
# it at the umask, so make it a private regular file first (refusing
# a symlink); its truncating write keeps the 0600 mode.
hnsw_path = path.with_suffix(".hnsw")
ensure_private_file(hnsw_path, what="HNSW index")
self._index.save_index(str(hnsw_path))
# Save metadata, mappings, and embeddings
@@ -834,7 +838,9 @@ class HNSWVectorIndex:
"embeddings": {mid: emb.tolist() for mid, emb in self._embeddings.items()},
}
with open(meta_path, "w") as f:
# Memory metadata and raw embeddings: owner-only, like every other
# memory store file.
with open_owner_only(meta_path, "w") as f:
json.dump(meta_data, f)
def load_index(self, path: str | Path) -> None:
+4 -1
View File
@@ -17,6 +17,7 @@ from datetime import datetime, timezone
from pathlib import Path
from typing import TYPE_CHECKING, Any
from ...fileperms import connect_private_sqlite
from ..models import Memory, ScopeLevel, normalize_entity_refs
from ..ports import MemoryFilter
@@ -82,8 +83,10 @@ class SQLiteMemoryStore:
Commits on clean exit, rolls back on exception, and always closes
the connection -- callers use ``with self._get_conn() as conn:``.
The file is created, or narrowed, owner-only before sqlite opens it:
it holds memory content and the user ids it belongs to.
"""
conn = sqlite3.connect(str(self.db_path))
conn = connect_private_sqlite(self.db_path, what="memory store")
conn.row_factory = sqlite3.Row
try:
with conn:
+2 -1
View File
@@ -23,6 +23,7 @@ from pathlib import Path
from threading import RLock
from typing import TYPE_CHECKING, Any
from ...fileperms import connect_private_sqlite
from .graph_models import Entity, Relationship, RelationshipDirection, Subgraph
if TYPE_CHECKING:
@@ -84,7 +85,7 @@ class SQLiteGraphStore:
Commits on clean exit, rolls back on exception, and always closes
the connection -- callers use ``with self._get_conn() as conn:``.
"""
conn = sqlite3.connect(str(self.db_path))
conn = connect_private_sqlite(self.db_path, what="memory graph store")
conn.row_factory = sqlite3.Row
# Configure page cache size (negative = KB, positive = pages)
+3 -2
View File
@@ -29,6 +29,7 @@ from typing import TYPE_CHECKING, Any, cast
import numpy as np
from ...fileperms import connect_private_sqlite
from ..models import Memory, ScopeLevel, normalize_entity_refs
from ..ports import VectorFilter, VectorSearchResult
@@ -235,8 +236,8 @@ class SQLiteVectorIndex:
self._init_db()
def _create_conn(self) -> sqlite3.Connection:
"""Create a SQLite connection with sqlite-vec loaded."""
conn = sqlite3.connect(str(self._db_path))
"""Create a SQLite connection with sqlite-vec loaded (file owner-only)."""
conn = connect_private_sqlite(self._db_path, what="memory vector index")
conn.row_factory = sqlite3.Row
# Load sqlite-vec extension
+3 -1
View File
@@ -1511,7 +1511,9 @@ class TrafficLearner:
now_iso = datetime.now(timezone.utc).isoformat()
def _bump() -> bool:
conn = sqlite3.connect(str(db_path))
from ..fileperms import connect_private_sqlite
conn = connect_private_sqlite(db_path, what="memory store")
try:
cursor = conn.execute(
"UPDATE memories SET metadata = json_set("
+5 -2
View File
@@ -241,8 +241,11 @@ class MemoryHandler:
self._native_memory_dir = _paths.native_memory_dir()
# Create directory if it doesn't exist
self._native_memory_dir.mkdir(parents=True, exist_ok=True)
# Create the directory owner-only (and narrow it if an earlier run left
# it wider): it holds the model's memory files verbatim.
from headroom import fileperms as _fileperms
_fileperms.private_dir(self._native_memory_dir)
logger.info(f"Memory: Native memory directory: {self._native_memory_dir}")
def get_beta_headers(self) -> dict[str, str]:
+7 -3
View File
@@ -29,6 +29,7 @@ from typing import TYPE_CHECKING, Any
import httpx
from headroom import fileperms as _fileperms
from headroom import paths as _paths
if TYPE_CHECKING:
@@ -365,9 +366,12 @@ class UsageReporter:
return
try:
self._cache_path.parent.mkdir(parents=True, exist_ok=True)
self._cache_path.write_text(
json.dumps(self._license_info.to_dict(), indent=2), encoding="utf-8"
)
# Owner-only: the envelope names the org and plan.
# newline pinned: state files are LF on every platform (#3698).
with _fileperms.open_owner_only(
self._cache_path, "w", encoding="utf-8", newline="\n"
) as fh:
fh.write(json.dumps(self._license_info.to_dict(), indent=2))
except OSError:
logger.warning("Could not save license cache to %s", self._cache_path)
+413
View File
@@ -0,0 +1,413 @@
"""Every state file that holds conversation-derived content is created owner-only.
The CCR store and the runtime log already were; the memory databases (facts
extracted from conversations plus the user id they belong to), the vector and
graph stores, the HNSW metadata dump, the licence validation cache and the
native memory directory were created at the process umask — ``0644`` on a
stock host, readable by any local account.
The fix routes them all through ``headroom.fileperms``, so these tests pin
both the primitives and every call site, under a permissive umask so the
result is known to be umask-independent.
"""
from __future__ import annotations
import os
import sqlite3
import stat
import subprocess
import sys
import pytest
from headroom import fileperms
pytestmark = pytest.mark.skipif(
not fileperms.OWNER_ONLY_SUPPORTED, reason="POSIX permission bits only"
)
def _mode(path) -> int:
return stat.S_IMODE(os.stat(path).st_mode)
@pytest.fixture(autouse=True)
def _permissive_umask():
old = os.umask(0o022)
try:
yield
finally:
os.umask(old)
# ---- primitives -------------------------------------------------------------
class TestEnsurePrivateFile:
def test_new_file_is_private_from_birth(self, tmp_path):
p = tmp_path / "store.db"
fileperms.ensure_private_file(p)
assert _mode(p) == 0o600
def test_existing_wide_file_is_narrowed(self, tmp_path):
p = tmp_path / "store.db"
p.write_text("x")
assert _mode(p) == 0o644 # precondition under the 022 umask
fileperms.ensure_private_file(p)
assert _mode(p) == 0o600
assert p.read_text() == "x"
def test_symlink_is_refused_and_target_untouched(self, tmp_path):
target = tmp_path / "victim"
target.write_text("SECRET")
before = _mode(target)
link = tmp_path / "store.db"
link.symlink_to(target)
with pytest.raises(PermissionError):
fileperms.ensure_private_file(link, what="memory store")
assert _mode(target) == before
assert target.read_text() == "SECRET"
def test_directory_is_refused(self, tmp_path):
d = tmp_path / "store.db"
d.mkdir()
with pytest.raises(PermissionError):
fileperms.ensure_private_file(d)
def test_narrowing_failure_fails_closed(self, tmp_path, monkeypatch):
p = tmp_path / "store.db"
p.write_text("")
def boom(fd, mode):
raise PermissionError("cannot fchmod")
monkeypatch.setattr(os, "fchmod", boom)
with pytest.raises(PermissionError):
fileperms.ensure_private_file(p)
class TestConnectPrivateSqlite:
def test_creates_private_db_and_sidecars(self, tmp_path):
p = tmp_path / "m.db"
conn = fileperms.connect_private_sqlite(p)
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("CREATE TABLE t(x)")
conn.execute("INSERT INTO t VALUES (1)")
conn.commit()
assert _mode(p) == 0o600
# sqlite gives the -wal/-shm sidecars the main file's mode.
for sidecar in ("m.db-wal", "m.db-shm"):
if (tmp_path / sidecar).exists():
assert _mode(tmp_path / sidecar) == 0o600
conn.close()
@pytest.mark.parametrize(
("name", "uri"),
[
(":memory:", False),
("", False),
(":memory:", True),
("file::memory:?cache=shared", True),
("file:mem?mode=memory&cache=shared", True),
],
)
def test_in_memory_databases_create_no_files(self, tmp_path, monkeypatch, name, uri):
monkeypatch.chdir(tmp_path)
fileperms.connect_private_sqlite(name, uri=uri).close()
# No file literally named ":memory:" (a bug another store has shipped).
assert sorted(p.name for p in tmp_path.iterdir()) == []
@pytest.mark.parametrize(
("uri_for", "created"),
[
# Relative, percent-encoded, with an explicit create mode.
(lambda d: "file:my%20memory.db?mode=rwc", "my memory.db"),
(lambda d: f"file:{d}/abs.db", "abs.db"),
(lambda d: f"file://localhost{d}/host.db?cache=shared", "host.db"),
],
ids=["relative-percent-encoded", "absolute", "localhost-authority"],
)
def test_file_backed_uri_is_private(self, tmp_path, monkeypatch, uri_for, created):
monkeypatch.chdir(tmp_path)
conn = fileperms.connect_private_sqlite(uri_for(tmp_path), uri=True)
conn.execute("CREATE TABLE t(x)")
conn.commit()
conn.close()
# Under the forced 022 umask sqlite alone would have made this 0644.
assert _mode(tmp_path / created) == 0o600
def test_file_prefixed_literal_path_is_private(self, tmp_path, monkeypatch):
# Without uri=True sqlite treats "file:..." as an ordinary file name,
# unless it was built with SQLITE_USE_URI (Debian, python Docker images).
monkeypatch.chdir(tmp_path)
conn = fileperms.connect_private_sqlite("file:literal.db")
conn.execute("CREATE TABLE t(x)")
conn.commit()
conn.close()
assert [p.name for p in tmp_path.iterdir()] == [
"literal.db" if fileperms._sqlite_always_uri() else "file:literal.db"
]
assert _mode(tmp_path / os.listdir(tmp_path)[0]) == 0o600
def test_file_prefix_is_a_uri_on_builds_that_always_parse_uris(self, monkeypatch):
monkeypatch.setattr(fileperms, "_sqlite_always_uri", lambda: True)
assert fileperms._sqlite_file_path("file:literal.db", uri=False) == "literal.db"
assert fileperms._sqlite_file_path("file::memory:", uri=False) is None
assert fileperms._sqlite_file_path("plain.db", uri=False) == "plain.db"
def test_undecodable_uri_escape_secures_the_raw_byte_name(self, tmp_path, monkeypatch):
# sqlite opens the raw 0xFF byte; securing U+FFFD instead would leave
# the real database at the umask.
monkeypatch.chdir(tmp_path)
try:
conn = fileperms.connect_private_sqlite("file:secret%FF.db", uri=True)
except OSError as exc: # e.g. APFS refuses non-UTF-8 names
pytest.skip(f"filesystem rejects non-UTF-8 names: {exc}")
conn.execute("CREATE TABLE t(x)")
conn.commit()
conn.close()
assert os.listdir(os.fsencode(tmp_path)) == [b"secret\xff.db"]
assert _mode(os.path.join(os.fsencode(tmp_path), b"secret\xff.db")) == 0o600
@pytest.mark.parametrize(
("journal_mode", "sidecars"),
[("wal", ("-wal", "-shm")), ("persist", ("-journal",))],
)
def test_crash_leftover_sidecars_are_narrowed(self, tmp_path, journal_mode, sidecars):
# An older, unhardened run writes under the 022 umask and dies without
# a checkpoint, leaving 0644 sidecars that sqlite would reuse as-is.
db = tmp_path / "old.db"
script = (
"import os, sqlite3, sys\n"
"c = sqlite3.connect(sys.argv[1])\n"
f"c.execute('PRAGMA journal_mode={journal_mode}')\n"
"c.execute('CREATE TABLE t(x)')\n"
"c.execute(\"INSERT INTO t VALUES ('old')\")\n"
"c.commit()\n"
"os._exit(0)\n"
)
subprocess.run([sys.executable, "-c", script, str(db)], check=True)
for suffix in sidecars:
assert _mode(f"{db}{suffix}") == 0o644 # precondition
conn = fileperms.connect_private_sqlite(db)
for suffix in sidecars:
assert _mode(f"{db}{suffix}") == 0o600 # before sqlite writes a byte
conn.execute(f"PRAGMA journal_mode={journal_mode}")
conn.execute("INSERT INTO t VALUES ('PRIVATE-SENTINEL')")
conn.commit()
for suffix in sidecars:
assert _mode(f"{db}{suffix}") == 0o600
if journal_mode == "wal":
# The new row went into the reused WAL, which is now private.
assert b"PRIVATE-SENTINEL" in (tmp_path / "old.db-wal").read_bytes()
# Narrowing kept the leftover contents: the uncheckpointed row survives.
assert conn.execute("SELECT x FROM t ORDER BY rowid").fetchall() == [
("old",),
("PRIVATE-SENTINEL",),
]
conn.close()
def test_missing_sidecars_are_not_precreated(self, tmp_path):
db = tmp_path / "m.db"
fileperms.connect_private_sqlite(db).close()
assert [p.name for p in tmp_path.iterdir()] == ["m.db"]
@pytest.mark.parametrize("suffix", ["-wal", "-shm", "-journal"])
def test_refuses_symlinked_sidecar(self, tmp_path, suffix):
db = tmp_path / "m.db"
sqlite3.connect(db).close()
victim = tmp_path / "victim"
victim.write_text("SECRET")
(tmp_path / f"m.db{suffix}").symlink_to(victim)
with pytest.raises(PermissionError):
fileperms.connect_private_sqlite(db)
assert _mode(victim) == 0o644
assert victim.read_text() == "SECRET"
def test_read_only_uri_does_not_create_missing_db(self, tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
with pytest.raises(sqlite3.OperationalError):
fileperms.connect_private_sqlite("file:missing.db?mode=ro", uri=True)
assert not (tmp_path / "missing.db").exists()
def test_uri_with_remote_authority_is_refused(self):
with pytest.raises(PermissionError):
fileperms.connect_private_sqlite("file://otherhost/srv/m.db", uri=True)
def test_refuses_symlinked_db(self, tmp_path):
target = tmp_path / "victim.db"
sqlite3.connect(target).close()
link = tmp_path / "m.db"
link.symlink_to(target)
with pytest.raises(PermissionError):
fileperms.connect_private_sqlite(link)
class TestPrivateDir:
def test_new_and_existing_dirs_are_0700(self, tmp_path):
new = tmp_path / "a" / "memories"
fileperms.private_dir(new)
assert _mode(new) == 0o700
old = tmp_path / "old"
old.mkdir()
assert _mode(old) == 0o755
fileperms.private_dir(old)
assert _mode(old) == 0o700
def test_symlinked_dir_is_not_re_permissioned(self, tmp_path):
target = tmp_path / "target"
target.mkdir()
link = tmp_path / "memories"
link.symlink_to(target)
fileperms.private_dir(link)
assert _mode(target) == 0o755
class TestOpenOwnerOnlyWriteMode:
def test_write_mode_truncates_and_is_private(self, tmp_path):
p = tmp_path / "cache.json"
p.write_text("old-and-long")
assert _mode(p) == 0o644
with fileperms.open_owner_only(p, "w") as fh:
fh.write("new")
assert p.read_text() == "new"
assert _mode(p) == 0o600
def test_append_mode_still_appends(self, tmp_path):
p = tmp_path / "log"
with fileperms.open_owner_only(p, "a") as fh:
fh.write("a")
with fileperms.open_owner_only(p, "a") as fh:
fh.write("b")
assert p.read_text() == "ab"
def test_read_modes_are_rejected(self, tmp_path):
with pytest.raises(ValueError):
fileperms.open_owner_only(tmp_path / "x", "r")
# ---- call sites -------------------------------------------------------------
class TestMemoryStoresArePrivate:
def test_sqlite_memory_store(self, tmp_path):
from headroom.memory.adapters.sqlite import SQLiteMemoryStore
p = tmp_path / "headroom_memory.db"
SQLiteMemoryStore(p)
assert _mode(p) == 0o600
def test_sqlite_memory_store_narrows_existing_db(self, tmp_path):
from headroom.memory.adapters.sqlite import SQLiteMemoryStore
p = tmp_path / "headroom_memory.db"
sqlite3.connect(p).close() # an older, unhardened run left it 0644
assert _mode(p) == 0o644
SQLiteMemoryStore(p)
assert _mode(p) == 0o600
def test_fts5_index(self, tmp_path):
from headroom.memory.adapters.fts5 import FTS5TextIndex
p = tmp_path / "headroom_memory.db"
try:
FTS5TextIndex(p)
except sqlite3.OperationalError as exc: # sqlite built without FTS5
pytest.skip(f"FTS5 unavailable: {exc}")
assert _mode(p) == 0o600
def test_graph_store(self, tmp_path):
from headroom.memory.adapters.sqlite_graph import SQLiteGraphStore
p = tmp_path / "headroom_graph.db"
SQLiteGraphStore(p)
assert _mode(p) == 0o600
def test_vector_index(self, tmp_path):
pytest.importorskip("sqlite_vec")
from headroom.memory.adapters.sqlite_vector import SQLiteVectorIndex
p = tmp_path / "vectors.db"
try:
SQLiteVectorIndex(dimension=8, db_path=p)
except (ImportError, RuntimeError) as exc:
pytest.skip(f"sqlite-vec unavailable here: {exc}")
assert _mode(p) == 0o600
@staticmethod
def _hnsw_index():
# Never ``importorskip("hnswlib")`` here: on runners without AVX the
# native import dies with SIGILL and takes the whole pytest shard with
# it. The adapter's probe runs the import in a subprocess.
import asyncio
from headroom.memory.adapters.hnsw import _check_hnswlib_available
if not _check_hnswlib_available():
pytest.skip("hnswlib not available (or not usable on this CPU)")
import numpy as np
from headroom.memory.adapters.hnsw import HNSWVectorIndex
from headroom.memory.models import Memory
index = HNSWVectorIndex(dimension=4, max_elements=16)
mem = Memory(content="secret fact", user_id="alice", embedding=np.ones(4, dtype=np.float32))
asyncio.run(index.index(mem))
return index
def test_hnsw_index_and_metadata_files(self, tmp_path):
index = self._hnsw_index()
base = tmp_path / "idx"
index.save_index(base)
assert _mode(base.with_suffix(".hnsw")) == 0o600
assert _mode(base.with_suffix(".meta")) == 0o600
# A re-save over a file left wide by an older version narrows it.
os.chmod(base.with_suffix(".hnsw"), 0o640)
index.save_index(base)
assert _mode(base.with_suffix(".hnsw")) == 0o600
def test_hnsw_index_refuses_planted_symlink(self, tmp_path):
index = self._hnsw_index()
target = tmp_path / "elsewhere.bin"
target.write_bytes(b"untouched")
base = tmp_path / "idx"
base.with_suffix(".hnsw").symlink_to(target)
with pytest.raises(PermissionError):
index.save_index(base)
assert target.read_bytes() == b"untouched"
def test_native_memory_dir_is_0700(tmp_path):
from headroom.proxy.memory_handler import MemoryConfig, MemoryHandler
d = tmp_path / "memories"
d.mkdir()
assert _mode(d) == 0o755
handler = MemoryHandler(MemoryConfig(native_memory_dir=str(d)))
handler._init_native_memory_dir() # what initialize() runs for the native tool
assert _mode(d) == 0o700
def test_license_cache_is_private(tmp_path):
from headroom.telemetry.reporter import LicenseInfo, UsageReporter
cache = tmp_path / "license.json"
cache.write_text("{}")
assert _mode(cache) == 0o644
reporter = UsageReporter("hlk_test", cache_path=cache)
reporter._license_info = LicenseInfo(status="active")
reporter._save_cache()
assert _mode(cache) == 0o600
assert "active" in cache.read_text()
def test_ccr_backend_still_private_via_shared_primitive(tmp_path):
from headroom.cache.backends.sqlite import SQLiteBackend
p = tmp_path / "ccr_store.db"
SQLiteBackend(p)
assert _mode(p) == 0o600
+26
View File
@@ -8,6 +8,7 @@ from __future__ import annotations
from types import SimpleNamespace
import anyio
import pytest
from headroom.telemetry.reporter import UsageReporter
@@ -70,3 +71,28 @@ def test_baseline_not_advanced_on_exception():
anyio.run(r._report_usage)
assert r._last_tokens_saved_by_model == {}
assert r._last_requests_by_model == {}
@pytest.mark.windows_newline
def test_license_cache_is_written_with_lf(tmp_path, monkeypatch):
"""The cache is indented JSON; pin LF so Windows does not write CRLF (#3698).
Asserted at the call: on POSIX an unpinned write is identical on disk.
"""
from headroom import fileperms
from headroom.telemetry.reporter import LicenseInfo
calls = []
original = fileperms.open_owner_only
def spy(path, mode="a", **kwargs):
calls.append(kwargs.get("newline"))
return original(path, mode, **kwargs)
monkeypatch.setattr(fileperms, "open_owner_only", spy)
cache = tmp_path / "license.json"
reporter = UsageReporter("hlk_test", cache_path=cache)
reporter._license_info = LicenseInfo(status="active")
reporter._save_cache()
assert calls == ["\n"]
assert b"\r\n" not in cache.read_bytes()