---
id: submodule-server-history
type: submodule-design
status: active
title: history — chat-history search index
parent: module-server
depends-on: [module-contracts]
tags: [history]
---

## Responsibility
The `history.search` backend: a **lazy in-memory index** over pi's session JSONL files (prompt recall +
full-conversation matches). Enumerates and reads session files **itself** (async `readdir`/`stat`/
`readFile`, replicating pi's pinned discovery layout below) — it must **never call
`SessionManager.listAll()`** on any search/refresh path: `listAll` reads and JSON-parses *every session
file in full* just to build its `SessionInfo`s (`messageCount`/`allMessagesText` we never use), so an
index refresh through it costs the whole corpus even when nothing changed, on the event loop every live
session shares. **Never writes** session files. Because discovery walks every pi session on disk, the
host handler's `all` scope (`host/historyScope.ts`'s `filter = () => true`) deliberately surfaces pi-CLI
sessions outside any registered ThinkRail workspace too — a bit more than `session.getMessages` itself
ever exposes for a single session, but consistent with an owner-scoped host (no multi-tenant isolation
to preserve).

## Design
- `extract.ts` — pure JSONL→`ExtractedSession | null` (`extractSession`): the session's identity
  (`id`/`cwd` from the header, mirroring pi's own rejection rule — the first parseable entry must be a
  `type: "session"` header with a string `id`, else `null`), its `title` (the **latest** `session_info`
  entry's name, including explicit clears — the same latest-wins rule as pi's `buildSessionInfo`), and its
  `HistoryEntry[]` — all from **one parse**, so the index never needs a second source (`listAll`) for
  metadata. Pi session files are **trees** (abandoned branches) that compaction rewrites, so it resolves
  the file the way pi does before the client renders it — `parseSessionEntries` →
  `migrateSessionEntries` → `buildSessionContext` (follow the current leaf, apply the latest compaction,
  drop summarized/abandoned entries) — then indexes the resolved messages, filtered through contracts'
  **`isTranscriptMessageRole`**, the very guard `getSessionMessages` sends by (one policy, not a local copy
  of it). So `messageIndex` matches the client's `turnIdByMessageIndex` exactly
  (no raw-file-order drift), and abandoned/summarized text never becomes a hit — the compaction summary is
  sent (it renders the client's compaction marker), so it consumes an index slot without being searchable.
  The internal
  `TODO_NUDGE_PREFIX` control message (hidden from the transcript on hydrate) is skipped after its index
  slot is consumed, so alignment holds; a **superseded auto-retry attempt** (the shared
  `isRetriedAttempt` reading from contracts — an errored assistant immediately followed by another
  assistant, the adjacent shape pi's `_prepareRetry` produces,
  which hydration renders as no turn / null anchor) is skipped the same way, so a
  failed partial's text never becomes a hit that could only resolve to "couldn't locate the message". Entry text is **full, never truncated** — a hit's `text` is what
  recall inserts and what the overlay's preview presents as the whole prompt, so a cap would silently
  corrupt recall of long pasted-log prompts and make terms past the cutoff unsearchable (the memory
  precedent is pi itself: `SessionInfo.allMessagesText` holds every session's full text in memory). Tool
  results/thinking are not indexed.
- `historyIndex.ts` — `HistoryIndex`: cold build on first search (async per-file IO yields the event loop;
  blocks the search up to a budget, then returns partial with `indexing: true`). Discovery is its own
  async enumeration (see "pi file format" below for the pinned layout): a custom `sessionDir` is a flat,
  non-recursive dir of `.jsonl` files; the default root is `<agentDir>/sessions` — agent dir resolved
  **per refresh** via pi's exported `getAgentDir()`, which reads `PI_CODING_AGENT_DIR` live — holding one
  level of per-cwd directories, including top-level symlinks Pi treats as candidate directories. Broken
  or non-directory symlinks degrade to the same unreadable-directory empty result as Pi. Freshness = `(mtime, size)`
  revalidation throttled to ~2 s (pi appends live messages to the file, so the file IS the live feed — no
  agent-module hook; size is compared alongside mtime so an append landing in the same coarse mtime tick
  still reloads); only a **new or changed** file is read (once, `readFile`) and re-parsed — an unchanged
  corpus costs one `readdir` walk + one `stat` per file, no reads, no parses. A warm revalidation runs in
  the **background** — a bulk change (first refresh after restart, a git checkout) can still mean many
  parses, so a search never blocks on it; results are at most one cycle stale, and the background
  refresh swallows its own errors (an unhandled rejection could crash the in-process host). `indexing` is
  reported whenever any build is in flight (cold OR a background revalidation), so the client's retry loop
  polls until a just-written session lands — read-your-writes without blocking the search. Matching:
  case-insensitive substring AND over whitespace terms (query length + result `limit` clamped to the
  protocol caps, `MAX_HISTORY_QUERY_LENGTH`/`MAX_HISTORY_LIMIT`, at both the handler and `search()`);
  strict recency order; prompts deduped by normalized text keeping newest; caps + true totals.
  The messages section (and `messageTotal`) is filtered to `role === "assistant"` only — a user-role hit
  is always a textual duplicate of its own prompt entry (user text IS a prompt in this extraction), so it
  would add no text, only a location; that location moves onto the prompt hit instead, via the two
  fields below.
- Jump anchors are drift-tolerant: hits carry `anchorText` (message-text prefix) the client validates.
  Every prompt hit now also carries its kept-newest occurrence's `messageIndex`/`anchorText` —
  the same two fields `MessageHit` always had — making the prompt row itself jumpable. Both fields are
  optional on `PromptHit` (absent only when the host predates this feature).
- `testFixtures.ts` — test-only session-file builders (pinned by A5): `writeFixtureSession` writes a
  minimal but real pi-shaped JSONL, one flat file per session, that `historyIndex` tests, the e2e fixture
  seeder, and its own format-pinning test (`testFixtures.test.ts`) drive against; `defaultSessionDirFor`
  replicates pi's private default-layout directory encoding (not importable — see "pi file format" below)
  so fixtures can land where a real no-arg `listAll()` would actually look.

## On-disk JSONL structure (observed from pi session files)
- **`message` entries:** `{ type: "message", ..., message: { role: "system"|"user"|"assistant"|"toolResult", content: string|array, timestamp: ms-number } }`.
  System messages carry prompt `sections` and/or `toolsAdded`/`toolsRemoved`; the first request persists the
  full system message and later prompt/tool changes persist patches. `isTranscriptMessageRole` excludes system,
  so it renders no transcript row and consumes no `messageIndex` slot. A `toolResult` whose tool ran nested
  calls (`ctx.executeTool`) carries a bounded `nestedCalls` record; it is not a transcript row of its own.
- **`usage` entries:** `{ type: "usage", ..., kind, provider, model, usage, note? }`; they are hidden from the
  conversation tree and context, but pi includes their token/cost data in session totals. Compaction entries may
  carry a `systemMessage` snapshot alongside their summary.
- **`custom_message` entries:** `{ type: "custom_message", customType: string, content: string|array, timestamp: ISO-string, display: boolean, ... }`.
  They have no `message` wrapper and become role `custom` context messages.

## pi file format (`@earendil-works/pi-coding-agent`, the catalog-pinned version)
Verified against the installed `dist/core/session-manager.{js,d.ts}`. Re-verify these facts on a pi bump;
`testFixtures.ts` and `historyIndex.ts` both depend on the discovery layout.
- **Header:** pi writes `{ type: "session", version: 3, id, timestamp, cwd, parentSession? }`; `CURRENT_SESSION_VERSION = 3`.
  `cwd` is `path.resolve` of the cwd the `SessionManager` was handed (pinned in `testFixtures.test.ts`); stored
  directories rely on it to compare by `===` (architecture Decision 22).
  Readers require the first *parseable* entry to be a session header with a string `id` (not necessarily physical line 1).
  `cwd` is recovered from that header; a missing/invalid header rejects the file.
- **Naming and custom directories:** discovery only considers direct children whose names end in `.jsonl`; pi writes
  `<ISO timestamp with ':' and '.' replaced by '-'>_<sessionId>.jsonl`, but the filename otherwise has no meaning.
  `list(cwd, sessionDir)` and `listAll(sessionDir)` use a flat, non-recursive custom directory; custom-list filtering
  still compares the header `cwd` with `cwd`. Subdirectory placement is invisible.
- **Default layout:** no-argument `listAll()` scans `<agentDir>/sessions/` one level deep, considering child directories
  and symlinks, then their direct `.jsonl` children. Unreadable or broken symlink targets contribute no files. Each cwd
  directory is `--<resolved cwd with a leading slash removed and slash/backslash/colon replaced by `-`>--`; `getDefaultSessionDirPath`
  is private, while `getDefaultSessionDir` is not exported from the package root. `defaultSessionDirFor` replicates it.
- **When a file appears:** a new session's file is first written once it holds a user or assistant message
  (setup-only sessions leave no file), so a prompt whose first turn never completes is still on disk.
- **Environment:** `getAgentDir()` reads `PI_CODING_AGENT_DIR` when called, so changing that variable before a SessionManager
  discovery call changes the root used by pi.

## Boundary
- **Narrow read for the search palette:** `HistoryIndex.sessions()` answers every indexed session's
  identity, `title` and entries (read-only) with the same freshness, cold-build budget and `indexing`
  flag as `search()`; the `search` module ranks titles with the shared matcher and scans entries with
  `matchesTerms`, so no second parse of the corpus exists.
- **Public surface (`index.ts`):** `HistoryIndex` (+ `HistorySession`), `getHistoryIndex()`, `matchesTerms`, `makeSnippet`,
  `clampLimit`, `extractSession`, types. **No test helpers** — `writeFixtureSession`/`defaultSessionDirFor`
  are disk-writing test-only builders and must not enter the runtime module graph; they're reachable only
  through the server package's dedicated **`@thinkrail/server/history-test-fixtures`** subpath export
  (package.json), the sanctioned test boundary — same pattern as `@thinkrail/server/agent`. In-package
  tests import them relatively (`./testFixtures`); the e2e seeder imports the subpath.
- **Allowed deps:** `@earendil-works/pi-coding-agent` (`getAgentDir` — the default sessions root — plus
  the exported session-tree helpers `parseSessionEntries`/`migrateSessionEntries`/`buildSessionContext`
  used by `extract.ts`; **not `SessionManager`** in production code — see Responsibility. Tests may use
  `SessionManager` to *pin* layout/format facts against the real thing), `@thinkrail/contracts`,
  `node:fs`, `node:fs/promises`, `node:path`.
- **Forbidden:** importing `agent`/`workspaces`/`projects` (scope mapping is injected by the host handler
  via the `filter`/`labels` callbacks passed into `search()`); writing anything to disk (`writeFixtureSession`
  is test-only, never called from production code paths).
