mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(openclaw): record turn advancements as one marker file per key (#4212)
## Description
OpenClaw 2026.9.9 started calling the context engine's `commitTurn`
after every accepted turn. While a commit is pending, it runs every
later turn of that session on its legacy context path, which means no
Headroom compression. The plugin's turn-commit store (from #3442) has
three ways to leave a commit pending, and this PR removes all three.
Each committed key becomes one small marker file, created with exclusive
creation (`O_CREAT | O_EXCL`). There's no lock and no shared file to
rewrite, and only the key is stored.
**What we saw.** We run the plugin on Kubernetes with a read-only root
filesystem. After upgrading OpenClaw to 2026.9.9, every turn logged:
```
[context-engine] durable turn advancement remains queued: <key>: ENOENT: no such file or directory, mkdir '/home/openclaw/.headroom'
```
On the session's next turn, OpenClaw retries the pending commit before
assembling context. When the retry fails, it degrades that turn to the
legacy engine (`degradeBeforeStart("pending durable turn advancement
could not be completed before the next turn")` in OpenClaw's
`context-engine-turn-attempt`). A failed commit only increments
`attempt_count`, so the row stays pending and every later turn of the
session degrades too.
**The three problems in `advancement-key-store.ts`:**
1. **Read-only home.** The default
`~/.headroom/openclaw/commit-log.json` can't be created on a read-only
root. Node reports the failed `mkdir` as `ENOENT`, which reads like a
bug rather than a setting. `commitLogPath` exists on the engine but not
in the manifest's `configSchema` (`additionalProperties: false`), so
`HEADROOM_WORKSPACE_DIR` is the only way to move it.
2. **Unbounded growth.** Every accepted turn's `messages` (OpenClaw caps
a turn at 8 MiB) are kept forever in one JSON file. Each commit
re-reads, parses, copies and rewrites the whole file on the gateway's
event loop, so the file and each commit's latency grow with every turn
ever committed. Nothing reads the messages back (`get` is test-only),
and OpenClaw's transcript already holds them.
3. **A lock nothing reclaims.** By design, the lock is never
force-broken. A gateway killed mid-commit (an OOM kill, a node drain)
leaves `commit-log.json.lock` behind, and every later commit waits 60 s
and then fails until someone deletes it. With the retry above, every
turn then starts 60 s late before degrading.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **One marker per committed key**, at
`commit-log.d/<sha256[0:2]>/<sha256(key)>` next to the old log path,
created with `open(path, "wx")`. The first creator in any process gets
`"committed"`; every later caller gets `EEXIST`, i.e. `"duplicate"`.
Only that exclusive create can mean `"duplicate"`, and only when what
exists at the path is a marker file. An `EEXIST` from creating the shard
directory (a file where the directory belongs), or anything but a file
at the marker's path, is a storage error. The marker holds
`{advancementKey, committedAt}`: 99 bytes for OpenClaw's UUID keys.
- **Only the key is recorded.** For this engine, the advancement *is*
the key: `assemble()` works from the messages OpenClaw passes in, and
the plugin keeps no turn state. The contract asks for "one atomic,
idempotent write keyed by `advancementKey`" (OpenClaw
`docs/concepts/context-engine.md`), and creating the marker is that
write.
- **Keys are still never evicted**, for the reason #3442 gives.
- **Migration.** Before every commit and lookup, a store checks for a
`commit-log.json` from an earlier version. With no log there, that check
is one failed `open()`. If a log is there and it's a version the store
hasn't migrated, the store moves its keys into markers and records which
version that was (size, mtime, inode) in
`commit-log.d/migrated-legacy-log.json`. That record only saves work, so
a store that can't write it carries on and reads the log again later.
The log's version and its contents come from one open handle. The log
itself is never moved or changed, so an older version still running
against the same workspace keeps its own record, and nothing has to
write to the log's directory (which may be read-only, with a writable
marker volume mounted beneath it). An unchanged log is skipped by its
metadata alone, even if it can no longer be read. A changed one is
migrated again, so a key an older version commits at any moment, even
while a store is migrating the log, is never accepted as new. Only keys
the store hasn't seen get markers, so a rewrite costs only the keys it
adds. Both the object and the older array format are read. Several
processes can migrate at once safely, since creating an existing marker
is a no-op. A corrupt log still fails loudly rather than being read as
empty, and the store recovers once it's fixed.
- **An actionable error.** When the marker directory can't be created,
the message says what to do: `Cannot record turn advancements under
<dir>: <error>. Point HEADROOM_WORKSPACE_DIR (or the plugin's
commitLogPath) at a writable directory.` A lookup in a store whose path
a file blocks fails the same way on every platform, although Windows
reports such a path as `ENOENT`, which is also what a missing marker
gives.
- **`commitLogPath` is settable.** It's declared in
`openclaw.plugin.json`, with a UI hint and a README row.
- **Tests.** The store suite is rewritten for the new design. It keeps
the existing guarantees: committed, then duplicate; survives a restart;
never evicts across 600 keys; exactly one winner in-process and across
two OS processes; a failed write is retried, not remembered. New tests
cover:
- a commit stores one small marker holding only the key;
- committing more keys never rewrites an earlier marker (same inode and
mtime);
- a leftover `.lock` from a crashed process doesn't block a commit;
- a file where the marker directory or a shard directory belongs, and a
directory where a marker belongs: each is an error, never `"duplicate"`,
and the same key commits once the path is cleared;
- a file blocking the store's directory or its marker directory,
reported the way Windows reports it (`ENOENT`, not `ENOTDIR`): a lookup
is a storage error, not an absent key;
- both legacy formats, including two stores migrating concurrently;
- a legacy log in a read-only directory: commits go through; a restart
doesn't read the unchanged log again; a migrated log that can no longer
be read doesn't block commits; and once the directory is writable, a key
an older version commits the old way (temp file + rename) is migrated
before a running store's next commit;
- a key an older version commits while a store is migrating the log: the
store's next lookup finds it, and its next commit, like a new store's,
is `"duplicate"`;
- an older version rewriting the log: only the key it added gets a
marker;
- a migration record that can't be written: commits go through, in that
store and in one started later;
- a corrupt legacy log.
In `engine.test.ts`, a commit carrying 1 MB of tool output stores under
1 KB. The lock-holder fixture is removed, since there is no lock.
## Testing
- [ ] Unit tests pass (`pytest`) (n/a, TypeScript plugin only)
- [ ] Linting passes (`ruff check .`) (n/a)
- [ ] Type checking passes (`mypy headroom`) (n/a; `npm run typecheck`
passes)
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ cd plugins/openclaw && npm run typecheck # exit 0
$ npx vitest run # Node v24.21.0, macOS
Test Files 9 passed (9)
Tests 139 passed (139)
$ npx vitest run # Node v22.22.0, GitHub Actions windows-latest
Test Files 9 passed (9)
Tests 137 passed | 2 skipped (139)
```
## Real Behavior Proof
- Environment: macOS, Node v24.21.0, for the benchmark below: "old" is
`advancement-key-store.ts` from `main`, "new" is this PR, and the same
script drives both. In production: OpenClaw 2026.9.9 on Kubernetes with
a read-only root filesystem and `HEADROOM_WORKSPACE_DIR` on a writable
volume, running the plugin at v0.40.0 from a fork tag, with
`advancement-key-store.ts` identical to a9e9ea6e.
- Exact command / steps: one script drives both stores through the first
three cases, and the last compares production logs:
- commit 300 accepted turns, each carrying 100 KB of tool output, and
time commits #1, #100 and #300;
- commit once with a `commit-log.json.lock` left behind, as a killed
process would leave it;
- commit with the workspace's parent directory not writable (mode 0555);
- in production, compare one gateway's OpenClaw logs over a 12-hour
night on v0.40.0's store with its first 3 hours on this store.
- Observed result: the new store stays small and fast and ignores a
leftover lock. In production, no commit failed and no turn fell back to
legacy.
| | Old store | New store |
|---|---|---|
| Store on disk after 300 turns | 30.0 MB | 0.02 MB |
| Commit #1 / #100 / #300 | 3.3 / 14.3 / 40.4 ms | 2.6 / 0.1 / 0.2 ms |
| Leftover lock file from a crashed process | fails after 60.0 s |
committed immediately |
| Workspace not writable | `EACCES: permission denied, mkdir
'…/.headroom'` | same error, plus which setting fixes it |
In production, v0.40.0's store failed all 53 commits overnight with the
`remains queued` line above, and all 8 later turns of a session fell
back to legacy. On this store, the same gateway made 38 commits in its
first 3 hours, none failed, and no turn fell back. The 5 later turns we
could trace to their session all ran on headroom. The 38 markers take
296 KiB on disk, shard directories included.
- Not tested: the two permission-bit tests (a read-only directory, an
unwritable migration record) on Windows or as root, where they are
skipped, since permission bits don't restrict either. Our production
build predates the last three commits, which change only how a legacy
log is migrated and how a blocked path is reported, and that deployment
never had a legacy log.
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: the commit store's on-disk format (an
existing log is migrated on first use). The default location is
unchanged.
- Kill switch / disable path: none needed. Pointing `commitLogPath` or
`HEADROOM_WORKSPACE_DIR` elsewhere starts a fresh store.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert. The old log stays where it was, so an older
version still finds every key it wrote. Keys committed after the upgrade
live only in markers, so an older version could re-accept a turn that
was mid-retry during the downgrade.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (on Node
24; see Additional Notes)
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
- A version older than this one, running at the same moment against the
same workspace, keeps writing the old file, and this store migrates
whatever it adds. The older version doesn't read markers, though, so it
could still accept a key only this version committed. That's the same
mixed-version caveat any format change has.
- Unrelated to this change: the cross-process tests run `node
--experimental-transform-types`, which Node 26 no longer accepts (`bad
option`). They pass on Node 24, which CI uses.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4212"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
96e81ac670
commit
24acbe8f32
@@ -214,6 +214,7 @@ Compression is lossless via CCR (Compress-Cache-Retrieve): originals are stored
|
||||
| `routeCodexViaProxy` | `true` | Rewrite OpenClaw's built-in `openai-codex` provider to use the active Headroom proxy in memory so upstream Codex requests pass through Headroom. |
|
||||
| `gatewayProviderIds` | `[]` | Optional explicit list of OpenClaw provider ids to route through the active Headroom proxy in memory. Friendly aliases `codex`, `claude`, `copilot`, and `gemini` are also accepted. When set, this overrides the default `openai-codex` routing list. |
|
||||
| `announceCompression` | `true` | Adds a static, cache-stable notice to the system prompt the first time a session compresses more than 100 tokens, then keeps it unchanged for the rest of the session. Set to `false` to guarantee Headroom never modifies the system prompt. |
|
||||
| `commitLogPath` | `$HEADROOM_WORKSPACE_DIR/openclaw/commit-log.json`, else `~/.headroom/openclaw/commit-log.json` | Where the plugin records each turn OpenClaw has accepted (one small marker file per turn, in a sibling `commit-log.d/` directory), so a retried commit is answered as a duplicate even after a restart. Must be on a writable filesystem: on a read-only root (common in Kubernetes), point this or `HEADROOM_WORKSPACE_DIR` at a writable volume. While commits fail, OpenClaw 2026.9.9+ runs each later turn of the session on its legacy context path. |
|
||||
|
||||
## Comparison with lossless-claw
|
||||
|
||||
|
||||
@@ -49,6 +49,10 @@
|
||||
"announceCompression": {
|
||||
"label": "Announce Compression In System Prompt",
|
||||
"help": "When enabled, a static, cache-stable notice is added to the system prompt the first time a session compresses more than 100 tokens. Disable to guarantee the plugin never modifies the system prompt."
|
||||
},
|
||||
"commitLogPath": {
|
||||
"label": "Turn Commit Log Path",
|
||||
"help": "Where to record committed turn advancements (the markers go in a sibling directory). Must be writable. Defaults to $HEADROOM_WORKSPACE_DIR/openclaw/commit-log.json, else ~/.headroom/openclaw/commit-log.json."
|
||||
}
|
||||
},
|
||||
"configSchema": {
|
||||
@@ -123,6 +127,10 @@
|
||||
"announceCompression": {
|
||||
"type": "boolean",
|
||||
"default": true
|
||||
},
|
||||
"commitLogPath": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@@ -1,240 +1,294 @@
|
||||
/**
|
||||
* Durable, atomic, restart-safe, cross-process-safe record of committed
|
||||
* turn advancements.
|
||||
* Durable, restart-safe, cross-process-safe record of committed turn
|
||||
* advancements.
|
||||
*
|
||||
* OpenClaw's `atomic-idempotent-v1` transcriptSemantics contract requires
|
||||
* that a retried `commitTurn` call report "duplicate" for a key that was
|
||||
* already committed -- including after this process (or the host) restarts
|
||||
* -- and that "the accepted turn and key advance together" (the committed
|
||||
* messages are persisted atomically with the key, not discarded).
|
||||
* OpenClaw's `atomic-idempotent-v1` transcriptSemantics contract asks for one
|
||||
* atomic, idempotent write keyed by `advancementKey`: "committed" the first
|
||||
* time a key is presented, and "duplicate" for every retry of it -- including
|
||||
* a retry that arrives after this process (or the host) restarted, or one made
|
||||
* by another gateway process sharing the same Headroom workspace.
|
||||
*
|
||||
* The default commit-log path is shared by every gateway process using the
|
||||
* same Headroom workspace, not just every store instance in one process --
|
||||
* an in-process mutex (a JS `Map`/promise queue) does nothing to stop two
|
||||
* separate OS processes from each reading the same file, adding a different
|
||||
* key, and racing their renames, with the loser's accepted turn silently
|
||||
* discarded. The read/check/write transaction is therefore protected by a
|
||||
* real inter-process lock: an exclusive lock FILE (`<path>.lock`), acquired
|
||||
* via atomic exclusive creation (`open(..., "wx")`, which is atomic at the
|
||||
* OS level on POSIX and Windows alike) so only one process -- in this one or
|
||||
* any other -- can be inside the critical section at a time.
|
||||
* Each committed key is one small marker file, created with exclusive creation
|
||||
* (`open(..., "wx")`, i.e. O_CREAT | O_EXCL), which is atomic at the OS level
|
||||
* on POSIX and Windows alike. Whichever caller creates the marker first -- in
|
||||
* this process or any other -- gets "committed"; every later caller gets
|
||||
* EEXIST, i.e. "duplicate". There is no lock to hold and no shared file to
|
||||
* rewrite, so a process that dies mid-commit can't block the commits after it,
|
||||
* and a commit costs the same on the first turn as on the hundred-thousandth.
|
||||
*
|
||||
* The lock is never force-broken by age. An age-based "the holder must have
|
||||
* crashed" heuristic cannot tell a dead holder apart from one that is simply
|
||||
* slow (a large write, a GC pause, disk contention): a second process that
|
||||
* reclaims a lock the first still holds can commit and release before the
|
||||
* first resumes and overwrites that commit with its own now-stale snapshot
|
||||
* -- both processes report "committed", but only the second process's write
|
||||
* survives, and the first process's genuinely-accepted turn silently
|
||||
* vanishes. Node has no built-in binding for a real OS-mediated lock
|
||||
* (POSIX `flock`/Windows `LockFileEx`, where the kernel itself releases the
|
||||
* lock when the holder's process exits, crash included, so no staleness
|
||||
* guess is needed at all), so this fails closed instead: a lock that is
|
||||
* never released is never reclaimed, and a caller that can't acquire it
|
||||
* within the timeout gets a clear error rather than a silent lost write. A
|
||||
* lock file orphaned by a genuinely crashed process requires a human (or an
|
||||
* operator script) to remove `<path>.lock` before commits can resume --
|
||||
* that operational cost is the trade for never silently dropping an
|
||||
* acknowledged commit. Lock ownership is also verified before release (see
|
||||
* `withFileLock`): a process only ever deletes the lock file it created.
|
||||
* Only the key is recorded. This engine keeps no turn state of its own:
|
||||
* `assemble()` works from the messages OpenClaw passes in, and OpenClaw's own
|
||||
* transcript already persists every accepted turn, so for this engine the
|
||||
* advancement *is* the key. Earlier versions also kept a copy of each turn's
|
||||
* messages (OpenClaw allows up to 8 MiB per turn) in one JSON file that was
|
||||
* re-read and rewritten in full on every commit, which made the file, and the
|
||||
* time each commit took, grow without bound.
|
||||
*
|
||||
* Every `tryCommit` re-reads the file fresh from disk inside the lock (no
|
||||
* long-lived in-memory cache), and the in-memory return value is only ever
|
||||
* "committed" once the atomic (temp file + rename) write recording the key
|
||||
* has actually succeeded -- a failed write is never remembered as seen, so
|
||||
* retrying the same key attempts the commit again instead of returning a
|
||||
* phantom "duplicate" for a commit that never reached disk.
|
||||
* Keys are never evicted. OpenClaw may retry at any point after a restart or a
|
||||
* long stall, so pruning by count or age would risk re-accepting an
|
||||
* already-committed turn -- exactly the failure this store exists to avoid. A
|
||||
* marker is a few dozen bytes.
|
||||
*
|
||||
* Keys are never evicted. OpenClaw may retry at any point after a restart or
|
||||
* a long stall, so pruning by count or age would risk re-accepting an
|
||||
* already-committed turn -- exactly the failure this store exists to avoid.
|
||||
* No commit is cached in memory, and only an existing marker file counts as a
|
||||
* commit: a commit that fails before its marker exists (an unwritable or
|
||||
* blocked directory) is retried for real, never answered with a phantom
|
||||
* "duplicate". A marker is created before its contents are written, so
|
||||
* a failure after creation (a full disk) still leaves the key committed: the
|
||||
* caller sees the error, and its retry sees "duplicate" -- never a second
|
||||
* "committed".
|
||||
*/
|
||||
|
||||
import { mkdir, open, readFile, rename, stat, unlink, writeFile } from "node:fs/promises";
|
||||
import { createHash } from "node:crypto";
|
||||
import { lstat, mkdir, open, readFile, stat, writeFile, type FileHandle } from "node:fs/promises";
|
||||
import { dirname, join } from "node:path";
|
||||
import * as os from "node:os";
|
||||
|
||||
/**
|
||||
* True if `p` exists and is something other than a directory (e.g. a file
|
||||
* sitting where a directory needs to be). Some platforms (observed on
|
||||
* Windows) report `ENOENT` -- not a distinct "not a directory" code -- for
|
||||
* `readFile` when a path component like this blocks the rest of the path,
|
||||
* which is indistinguishable from "the commit log just doesn't exist yet"
|
||||
* by error code alone. Checking the parent directly is what tells the two
|
||||
* apart: silently treating a broken storage path as an empty store would
|
||||
* make every future commit re-accept an already-committed key.
|
||||
*/
|
||||
async function isBlockingNonDirectory(p: string): Promise<boolean> {
|
||||
try {
|
||||
const info = await stat(p);
|
||||
return !info.isDirectory();
|
||||
} catch {
|
||||
return false; // Doesn't exist -- not blocking anything.
|
||||
}
|
||||
}
|
||||
|
||||
/** Default commit-log location, honoring the same workspace override the
|
||||
* rest of the Headroom workspace convention uses (`HEADROOM_WORKSPACE_DIR`). */
|
||||
* rest of the Headroom workspace convention uses (`HEADROOM_WORKSPACE_DIR`).
|
||||
* The markers live beside it, in `commit-log.d/`. */
|
||||
export function defaultCommitLogPath(): string {
|
||||
const workspaceDir = process.env.HEADROOM_WORKSPACE_DIR?.trim() || join(os.homedir(), ".headroom");
|
||||
return join(workspaceDir, "openclaw", "commit-log.json");
|
||||
}
|
||||
|
||||
/** One durably-committed turn: the accepted messages plus when they landed. */
|
||||
export interface CommittedTurn {
|
||||
messages: unknown;
|
||||
committedAt: string;
|
||||
/** The marker directory for a commit-log path: `commit-log.json` -> `commit-log.d`. */
|
||||
export function markerDirFor(commitLogPath: string): string {
|
||||
return `${commitLogPath.replace(/\.json$/i, "")}.d`;
|
||||
}
|
||||
|
||||
type CommitLog = Record<string, CommittedTurn>;
|
||||
|
||||
/** Give up (rather than poll forever) if the lock still can't be acquired after this long. */
|
||||
const LOCK_ACQUIRE_TIMEOUT_MS = 60_000;
|
||||
const LOCK_POLL_BASE_MS = 20;
|
||||
|
||||
function sleep(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
function errorCode(error: unknown): string | undefined {
|
||||
return (error as NodeJS.ErrnoException)?.code;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run `fn` while holding an exclusive lock on `path` (a sibling `<path>.lock`
|
||||
* file). Serializes every caller -- this process or another -- against the
|
||||
* same file, not just calls on one object or one process.
|
||||
*
|
||||
* The lock is held until `fn` returns, however long that takes, and is
|
||||
* never reclaimed by another caller on a staleness guess -- see the module
|
||||
* docstring for why an age-based heuristic can silently drop a genuinely
|
||||
* accepted commit. A caller that can't acquire the lock within
|
||||
* `LOCK_ACQUIRE_TIMEOUT_MS` gets a clear timeout error instead of the
|
||||
* transaction being allowed to proceed unsafely.
|
||||
* Whether something other than a directory -- a regular file, say -- sits on
|
||||
* `dir`'s path, so that nothing beneath it can exist. POSIX reports such a
|
||||
* path as ENOTDIR, but Windows reports ENOENT, which reads exactly like an
|
||||
* absent marker; the nearest ancestor that exists tells the two apart.
|
||||
*/
|
||||
async function withFileLock<T>(path: string, fn: () => Promise<T>): Promise<T> {
|
||||
const lockPath = `${path}.lock`;
|
||||
await mkdir(dirname(path), { recursive: true });
|
||||
// Unique per acquisition, so release only ever removes the lock file this
|
||||
// exact call created -- never one a different (later) holder created.
|
||||
const token = `${process.pid}-${Date.now()}-${Math.random().toString(36).slice(2)}`;
|
||||
const deadline = Date.now() + LOCK_ACQUIRE_TIMEOUT_MS;
|
||||
let attempt = 0;
|
||||
for (;;) {
|
||||
async function isBlockedPath(dir: string): Promise<boolean> {
|
||||
for (let current = dir; ; current = dirname(current)) {
|
||||
try {
|
||||
const handle = await open(lockPath, "wx");
|
||||
try {
|
||||
await handle.writeFile(`${token}\n${process.pid}\n${new Date().toISOString()}\n`);
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
break;
|
||||
return !(await stat(current)).isDirectory();
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException)?.code !== "EEXIST") {
|
||||
throw error;
|
||||
const code = errorCode(error);
|
||||
if (code === "ENOTDIR") {
|
||||
return true;
|
||||
}
|
||||
if (Date.now() > deadline) {
|
||||
throw new Error(
|
||||
`Timed out waiting for the advancement-key-store lock on ${path}. If the ` +
|
||||
`process that created ${lockPath} is no longer running, remove it manually.`,
|
||||
);
|
||||
}
|
||||
attempt += 1;
|
||||
await sleep(LOCK_POLL_BASE_MS + Math.random() * LOCK_POLL_BASE_MS * Math.min(attempt, 10));
|
||||
}
|
||||
}
|
||||
try {
|
||||
return await fn();
|
||||
} finally {
|
||||
// Verify the lock file is still the one this call created before
|
||||
// removing it -- if it doesn't start with our token, some other process
|
||||
// (or a human) replaced or is using it, and it must not be touched.
|
||||
try {
|
||||
const current = await readFile(lockPath, "utf8");
|
||||
if (current.startsWith(`${token}\n`)) {
|
||||
await unlink(lockPath);
|
||||
}
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException)?.code !== "ENOENT") {
|
||||
if (code !== "ENOENT" || dirname(current) === current) {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export class DurableAdvancementKeyStore {
|
||||
constructor(private readonly path: string) {}
|
||||
/** Identifies one version of a legacy log file: any rewrite changes at least one of these. */
|
||||
type LegacyLogVersion = { size: number; mtimeMs: number; ino: number };
|
||||
|
||||
private async readAll(): Promise<CommitLog> {
|
||||
let raw: string;
|
||||
try {
|
||||
raw = await readFile(this.path, "utf8");
|
||||
} catch (error) {
|
||||
const code = (error as NodeJS.ErrnoException)?.code;
|
||||
if (code !== "ENOENT" && code !== "ENOTDIR") {
|
||||
throw error;
|
||||
}
|
||||
if (await isBlockingNonDirectory(dirname(this.path))) {
|
||||
throw error;
|
||||
}
|
||||
return {};
|
||||
}
|
||||
const parsed = JSON.parse(raw) as unknown;
|
||||
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
|
||||
return parsed as CommitLog;
|
||||
}
|
||||
if (Array.isArray(parsed)) {
|
||||
// Migrate the earlier key-only array format (no message payload) so
|
||||
// upgrading from it never re-accepts an already-committed key.
|
||||
const migrated: CommitLog = {};
|
||||
for (const entry of parsed) {
|
||||
if (typeof entry === "string") {
|
||||
migrated[entry] = { messages: null, committedAt: "" };
|
||||
}
|
||||
}
|
||||
return migrated;
|
||||
}
|
||||
return {};
|
||||
function sameVersion(a: LegacyLogVersion | undefined, b: LegacyLogVersion): boolean {
|
||||
return a?.size === b.size && a.mtimeMs === b.mtimeMs && a.ino === b.ino;
|
||||
}
|
||||
|
||||
export class DurableAdvancementKeyStore {
|
||||
private readonly markerDir: string;
|
||||
/** The version of the legacy log whose keys all have markers. */
|
||||
private migratedVersion: LegacyLogVersion | undefined;
|
||||
/** Legacy keys known to have markers, so a rewrite of the log costs only the keys it adds. */
|
||||
private readonly migratedKeys = new Set<string>();
|
||||
private migration: Promise<void> = Promise.resolve();
|
||||
|
||||
/**
|
||||
* `path` names the commit log; the markers live in `markerDirFor(path)`.
|
||||
* Versions that kept every key in the log file itself may have left one at
|
||||
* `path`. Before every commit and lookup, a log there that is new or has
|
||||
* changed has its keys migrated to markers. The log itself is never moved or
|
||||
* changed, so an older version still running against the same workspace
|
||||
* keeps its own record, and every key it commits, even while this store is
|
||||
* migrating the log, is migrated before this store's next commit.
|
||||
*/
|
||||
constructor(private readonly path: string) {
|
||||
this.markerDir = markerDirFor(path);
|
||||
}
|
||||
|
||||
private async writeAll(entries: CommitLog): Promise<void> {
|
||||
await mkdir(dirname(this.path), { recursive: true });
|
||||
const tmpPath = `${this.path}.${process.pid}.${Math.random().toString(36).slice(2)}.tmp`;
|
||||
await writeFile(tmpPath, JSON.stringify(entries), "utf8");
|
||||
await rename(tmpPath, this.path);
|
||||
private markerPath(key: string): string {
|
||||
const digest = createHash("sha256").update(key, "utf8").digest("hex");
|
||||
// Two-character fan-out keeps any one directory small.
|
||||
return join(this.markerDir, digest.slice(0, 2), digest);
|
||||
}
|
||||
|
||||
/** Records which version of the legacy log was migrated last. */
|
||||
private migratedLogRecordPath(): string {
|
||||
return join(this.markerDir, "migrated-legacy-log.json");
|
||||
}
|
||||
|
||||
private storageError(error: unknown): Error {
|
||||
// Node reports a directory it can't create on a read-only filesystem as
|
||||
// ENOENT, which reads like a bug rather than a setting -- say which one.
|
||||
return new Error(
|
||||
`Cannot record turn advancements under ${this.markerDir}: ${(error as Error)?.message ?? error}. ` +
|
||||
"Point HEADROOM_WORKSPACE_DIR (or the plugin's commitLogPath) at a writable directory.",
|
||||
{ cause: error },
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Atomically check-and-record `key` together with the `messages` it
|
||||
* advances. Holds the cross-process file lock (see `withFileLock`) for
|
||||
* the full read/check/write transaction, so two processes -- or two
|
||||
* instances in this one -- can't each read the same starting state and
|
||||
* race their writes.
|
||||
*
|
||||
* Throws if the write fails -- the caller sees a rejected commit, not a
|
||||
* false "committed"/"duplicate", and a retry re-reads the (still
|
||||
* unchanged) on-disk state and attempts the write again.
|
||||
* Whether a marker file -- not a directory or anything else -- exists at
|
||||
* `marker`. Storage that can't hold one (a file where a directory belongs)
|
||||
* is an error, never an absent marker.
|
||||
*/
|
||||
async tryCommit(key: string, messages: unknown): Promise<"committed" | "duplicate"> {
|
||||
return withFileLock(this.path, async () => {
|
||||
const entries = await this.readAll();
|
||||
if (key in entries) {
|
||||
return "duplicate";
|
||||
private async isMarker(marker: string): Promise<boolean> {
|
||||
try {
|
||||
return (await lstat(marker)).isFile();
|
||||
} catch (error) {
|
||||
if (errorCode(error) === "ENOENT" && !(await isBlockedPath(dirname(marker)).catch(() => true))) {
|
||||
return false;
|
||||
}
|
||||
const next: CommitLog = {
|
||||
...entries,
|
||||
[key]: { messages, committedAt: new Date().toISOString() },
|
||||
};
|
||||
await this.writeAll(next);
|
||||
return "committed";
|
||||
});
|
||||
throw this.storageError(error);
|
||||
}
|
||||
}
|
||||
|
||||
/** Create `key`'s marker; false if it already existed. */
|
||||
private async createMarker(key: string, committedAt: string): Promise<boolean> {
|
||||
const marker = this.markerPath(key);
|
||||
try {
|
||||
await mkdir(dirname(marker), { recursive: true });
|
||||
} catch (error) {
|
||||
// An EEXIST here means a file sits where the marker's directory belongs,
|
||||
// so no marker can exist beneath it: broken storage, not a duplicate.
|
||||
throw this.storageError(error);
|
||||
}
|
||||
let handle: FileHandle;
|
||||
try {
|
||||
handle = await open(marker, "wx");
|
||||
} catch (error) {
|
||||
// Only an existing marker file makes this a retry of a committed key.
|
||||
if (errorCode(error) === "EEXIST" && (await this.isMarker(marker).catch(() => false))) {
|
||||
return false;
|
||||
}
|
||||
throw this.storageError(error);
|
||||
}
|
||||
try {
|
||||
await handle.writeFile(`${JSON.stringify({ advancementKey: key, committedAt })}\n`, "utf8");
|
||||
} catch (error) {
|
||||
throw this.storageError(error);
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
private ensureMigrated(): Promise<void> {
|
||||
// One check at a time, each starting after its caller arrived: it sees every
|
||||
// change made before then, and a large log is never parsed twice at once. A
|
||||
// failed check isn't remembered, so the next call retries it.
|
||||
const check = this.migration.catch(() => undefined).then(() => this.migrateLegacyLog());
|
||||
this.migration = check;
|
||||
return check;
|
||||
}
|
||||
|
||||
/**
|
||||
* Move the keys of a legacy single-file log into markers. Without a log this
|
||||
* is one failed open(), and with an unchanged one an open() and an fstat(), so
|
||||
* it runs before every commit. Safe to run in several processes at once:
|
||||
* creating a marker that already exists is a no-op.
|
||||
*/
|
||||
private async migrateLegacyLog(): Promise<void> {
|
||||
let log: FileHandle;
|
||||
try {
|
||||
log = await open(this.path, "r");
|
||||
} catch (error) {
|
||||
const code = errorCode(error);
|
||||
if (code === "ENOENT" || code === "ENOTDIR") {
|
||||
return; // No legacy log. A blocked path fails loudly at the first commit.
|
||||
}
|
||||
if ((code === "EACCES" || code === "EPERM") && (await this.isMigratedLog())) {
|
||||
return; // Unreadable now, but every key it held has a marker, and it hasn't changed since.
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
let version: LegacyLogVersion;
|
||||
let raw: string;
|
||||
try {
|
||||
// Version and contents come from the same open file, so they always match.
|
||||
const { size, mtimeMs, ino } = await log.stat();
|
||||
version = { size, mtimeMs, ino };
|
||||
this.migratedVersion ??= await this.readMigratedLogRecord();
|
||||
if (sameVersion(this.migratedVersion, version)) {
|
||||
return; // Migrated already, and unchanged since.
|
||||
}
|
||||
raw = await log.readFile("utf8");
|
||||
} finally {
|
||||
await log.close();
|
||||
}
|
||||
// A corrupt log throws here rather than being treated as empty, which would
|
||||
// silently re-accept every key it held.
|
||||
const parsed = JSON.parse(raw) as unknown;
|
||||
let entries: Array<[string, string]> = [];
|
||||
if (Array.isArray(parsed)) {
|
||||
entries = parsed.filter((key): key is string => typeof key === "string").map((key) => [key, ""]);
|
||||
} else if (parsed && typeof parsed === "object") {
|
||||
entries = Object.entries(parsed as Record<string, { committedAt?: unknown }>).map(([key, entry]) => [
|
||||
key,
|
||||
typeof entry?.committedAt === "string" ? entry.committedAt : "",
|
||||
]);
|
||||
}
|
||||
for (const [key, committedAt] of entries) {
|
||||
if (this.migratedKeys.has(key)) {
|
||||
continue;
|
||||
}
|
||||
await this.createMarker(key, committedAt);
|
||||
this.migratedKeys.add(key);
|
||||
}
|
||||
// Every key in the version that was read has a marker now. Record which
|
||||
// version that was, so checks -- here and after a restart -- skip the log
|
||||
// while it is unchanged, and migrate it again once anything (an older
|
||||
// version still writing to it) changes it, including in the moments since
|
||||
// it was read. Moving the log aside instead would carry off whatever it
|
||||
// held by then, and leave an older version without its record.
|
||||
this.migratedVersion = version;
|
||||
try {
|
||||
await mkdir(this.markerDir, { recursive: true });
|
||||
await writeFile(this.migratedLogRecordPath(), `${JSON.stringify(version)}\n`, "utf8");
|
||||
} catch {
|
||||
// The record only saves work. Without it, a store that starts later
|
||||
// reads the log again, and commits carry on meanwhile.
|
||||
}
|
||||
}
|
||||
|
||||
/** Whether the log's metadata alone identifies it as the version already migrated. */
|
||||
private async isMigratedLog(): Promise<boolean> {
|
||||
try {
|
||||
const { size, mtimeMs, ino } = await stat(this.path);
|
||||
this.migratedVersion ??= await this.readMigratedLogRecord();
|
||||
return sameVersion(this.migratedVersion, { size, mtimeMs, ino });
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private async readMigratedLogRecord(): Promise<LegacyLogVersion | undefined> {
|
||||
try {
|
||||
return JSON.parse(await readFile(this.migratedLogRecordPath(), "utf8")) as LegacyLogVersion;
|
||||
} catch {
|
||||
return undefined; // None yet, or unreadable: migrating again is always safe.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Atomically record `key`: "committed" if this call created its marker,
|
||||
* "duplicate" if any earlier call -- in this process or another, before or
|
||||
* after a restart -- already had. Throws if the marker can't be created, so
|
||||
* the caller sees a rejected commit rather than a false answer, and a retry
|
||||
* attempts the commit again.
|
||||
*/
|
||||
async tryCommit(key: string): Promise<"committed" | "duplicate"> {
|
||||
await this.ensureMigrated();
|
||||
return (await this.createMarker(key, new Date().toISOString())) ? "committed" : "duplicate";
|
||||
}
|
||||
|
||||
/** Test-only: check membership without recording. */
|
||||
async has(key: string): Promise<boolean> {
|
||||
const entries = await this.readAll();
|
||||
return key in entries;
|
||||
}
|
||||
|
||||
/** Test-only: read back a committed entry (messages + commit time) without recording. */
|
||||
async get(key: string): Promise<CommittedTurn | undefined> {
|
||||
const entries = await this.readAll();
|
||||
return entries[key];
|
||||
await this.ensureMigrated();
|
||||
return this.isMarker(this.markerPath(key));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -28,7 +28,9 @@ export interface HeadroomEngineConfig extends ProxyManagerConfig {
|
||||
circuitBreakerThreshold?: number;
|
||||
circuitBreakerCooldownMs?: number;
|
||||
/** Where to durably record committed turn-advancement keys (see
|
||||
* `DurableAdvancementKeyStore`). Defaults to `defaultCommitLogPath()`. */
|
||||
* `DurableAdvancementKeyStore`; the markers live beside it in
|
||||
* `markerDirFor(commitLogPath)`). Must be on a writable filesystem.
|
||||
* Defaults to `defaultCommitLogPath()`. */
|
||||
commitLogPath?: string;
|
||||
announceCompression?: boolean;
|
||||
}
|
||||
@@ -235,17 +237,18 @@ export class HeadroomContextEngine {
|
||||
/**
|
||||
* Durable turn-advancement commit — required by OpenClaw's transcriptSemantics
|
||||
* contract. Called only for the accepted, successful turn; failed or aborted
|
||||
* turns never reach here. Must be an atomic, idempotent write of the
|
||||
* accepted `messages` keyed by `advancementKey` so a host retry with the
|
||||
* same key reports "duplicate" instead of re-applying the advancement —
|
||||
* including a retry that arrives after this process restarted, which is
|
||||
* why the record lives on disk (see `DurableAdvancementKeyStore`) rather
|
||||
* than in memory, and includes the messages rather than just the key.
|
||||
* turns never reach here. Must be one atomic, idempotent write keyed by
|
||||
* `advancementKey`, so a host retry with the same key reports "duplicate"
|
||||
* instead of re-applying the advancement — including a retry that arrives
|
||||
* after this process restarted, which is why the record lives on disk (see
|
||||
* `DurableAdvancementKeyStore`) rather than in memory. This engine keeps no
|
||||
* turn state of its own (OpenClaw's transcript holds the accepted
|
||||
* `messages`), so recording the key is the whole advancement.
|
||||
*/
|
||||
async commitTurn(params: { advancementKey: string; messages: any[] }): Promise<{
|
||||
status: "committed" | "duplicate";
|
||||
}> {
|
||||
const status = await this.advancementKeyStore.tryCommit(params.advancementKey, params.messages);
|
||||
const status = await this.advancementKeyStore.tryCommit(params.advancementKey);
|
||||
return { status };
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
import * as fs from "node:fs/promises";
|
||||
import * as os from "node:os";
|
||||
import * as path from "node:path";
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { DurableAdvancementKeyStore } from "../src/advancement-key-store.js";
|
||||
|
||||
// Where a regular file blocks a path, Windows reports ENOENT and POSIX reports
|
||||
// ENOTDIR. Report it the Windows way on every platform, so the store's handling
|
||||
// of it runs in every test environment, not only on Windows.
|
||||
vi.mock("node:fs/promises", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("node:fs/promises")>();
|
||||
function asOnWindows<F extends (...args: never[]) => Promise<unknown>>(fn: F): F {
|
||||
return (async (...args: Parameters<F>) => {
|
||||
try {
|
||||
return await fn(...args);
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException)?.code === "ENOTDIR") {
|
||||
(error as NodeJS.ErrnoException).code = "ENOENT";
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}) as F;
|
||||
}
|
||||
return {
|
||||
...actual,
|
||||
lstat: asOnWindows(actual.lstat),
|
||||
mkdir: asOnWindows(actual.mkdir),
|
||||
open: asOnWindows(actual.open),
|
||||
readFile: asOnWindows(actual.readFile),
|
||||
stat: asOnWindows(actual.stat),
|
||||
writeFile: asOnWindows(actual.writeFile),
|
||||
};
|
||||
});
|
||||
|
||||
describe("DurableAdvancementKeyStore, with a blocked path reported the Windows way (ENOENT)", () => {
|
||||
let dir: string;
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await fs.mkdtemp(path.join(os.tmpdir(), "headroom-advancement-key-store-win-"));
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await fs.rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it.each([
|
||||
["the store's directory", (storePath: string) => path.dirname(storePath)],
|
||||
["the marker directory", (storePath: string) => `${storePath.replace(/\.json$/, "")}.d`],
|
||||
])("reports a file in place of %s as a storage error, not as an absent key", async (_where, blocked) => {
|
||||
const storePath = path.join(dir, "nested", "commit-log.json");
|
||||
await fs.mkdir(path.dirname(blocked(storePath)), { recursive: true });
|
||||
await fs.writeFile(blocked(storePath), "not a directory", "utf8");
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-1")).rejects.toThrow(/HEADROOM_WORKSPACE_DIR/);
|
||||
await expect(store.has("turn-1")).rejects.toThrow(/HEADROOM_WORKSPACE_DIR/);
|
||||
|
||||
// Clearing the path makes the key absent, and committable.
|
||||
await fs.rm(blocked(storePath));
|
||||
await expect(store.has("turn-1")).resolves.toBe(false);
|
||||
await expect(store.tryCommit("turn-1")).resolves.toBe("committed");
|
||||
await expect(store.has("turn-1")).resolves.toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -1,4 +1,5 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import * as fs from "node:fs/promises";
|
||||
import * as os from "node:os";
|
||||
import * as path from "node:path";
|
||||
@@ -7,7 +8,11 @@ import { promisify } from "node:util";
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
|
||||
import { DurableAdvancementKeyStore, defaultCommitLogPath } from "../src/advancement-key-store.js";
|
||||
import {
|
||||
DurableAdvancementKeyStore,
|
||||
defaultCommitLogPath,
|
||||
markerDirFor,
|
||||
} from "../src/advancement-key-store.js";
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
|
||||
@@ -23,9 +28,6 @@ const NODE_SUPPORTS_TRANSFORM_TYPES = (() => {
|
||||
})();
|
||||
|
||||
const WORKER_PATH = fileURLToPath(new URL("./fixtures/commit-turn-worker.ts", import.meta.url));
|
||||
const HOLD_LOCK_WORKER_PATH = fileURLToPath(
|
||||
new URL("./fixtures/hold-lock-worker.ts", import.meta.url),
|
||||
);
|
||||
|
||||
/** Run the commit-turn worker as a real child process; returns its parsed JSON result. */
|
||||
async function runCommitWorker(
|
||||
@@ -45,15 +47,16 @@ async function runCommitWorker(
|
||||
return JSON.parse(await fs.readFile(resultPath, "utf8"));
|
||||
}
|
||||
|
||||
async function waitForFile(p: string): Promise<void> {
|
||||
for (;;) {
|
||||
try {
|
||||
await fs.access(p);
|
||||
return;
|
||||
} catch {
|
||||
await new Promise((r) => setTimeout(r, 5));
|
||||
}
|
||||
}
|
||||
/** Every file under `dir`, recursively. */
|
||||
async function filesUnder(dir: string): Promise<string[]> {
|
||||
const entries = await fs.readdir(dir, { recursive: true, withFileTypes: true });
|
||||
return entries.filter((e) => e.isFile()).map((e) => path.join(e.parentPath, e.name));
|
||||
}
|
||||
|
||||
/** Where `key`'s marker lives: `<marker dir>/<first two hex digits of its sha256>/<sha256>`. */
|
||||
function markerPathFor(storePath: string, key: string): string {
|
||||
const digest = createHash("sha256").update(key, "utf8").digest("hex");
|
||||
return path.join(markerDirFor(storePath), digest.slice(0, 2), digest);
|
||||
}
|
||||
|
||||
describe("DurableAdvancementKeyStore", () => {
|
||||
@@ -69,54 +72,44 @@ describe("DurableAdvancementKeyStore", () => {
|
||||
await fs.rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("commits a new key and creates the parent directory", async () => {
|
||||
it("commits a new key and creates the marker directory", async () => {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-1", [{ role: "user", content: "hi" }])).resolves.toBe(
|
||||
"committed",
|
||||
);
|
||||
await expect(fs.stat(storePath)).resolves.toBeTruthy();
|
||||
await expect(store.tryCommit("turn-1")).resolves.toBe("committed");
|
||||
await expect(fs.stat(markerDirFor(storePath))).resolves.toBeTruthy();
|
||||
});
|
||||
|
||||
it("reports duplicate for an already-committed key", async () => {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-1", [])).resolves.toBe("committed");
|
||||
await expect(store.tryCommit("turn-1", [])).resolves.toBe("duplicate");
|
||||
await expect(store.tryCommit("turn-1")).resolves.toBe("committed");
|
||||
await expect(store.tryCommit("turn-1")).resolves.toBe("duplicate");
|
||||
});
|
||||
|
||||
it("persists across a simulated restart (a fresh store instance, same path)", async () => {
|
||||
const before = new DurableAdvancementKeyStore(storePath);
|
||||
await expect(before.tryCommit("turn-1", [])).resolves.toBe("committed");
|
||||
await expect(before.tryCommit("turn-1")).resolves.toBe("committed");
|
||||
|
||||
const after = new DurableAdvancementKeyStore(storePath);
|
||||
await expect(after.tryCommit("turn-1", [])).resolves.toBe("duplicate");
|
||||
await expect(after.tryCommit("turn-1")).resolves.toBe("duplicate");
|
||||
// A key that was never committed is still accepted normally.
|
||||
await expect(after.tryCommit("turn-2", [])).resolves.toBe("committed");
|
||||
await expect(after.tryCommit("turn-2")).resolves.toBe("committed");
|
||||
});
|
||||
|
||||
it("treats a missing commit-log file as an empty store rather than an error", async () => {
|
||||
it("treats a store that doesn't exist yet as empty rather than an error", async () => {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.has("turn-1")).resolves.toBe(false);
|
||||
await expect(store.tryCommit("turn-1", [])).resolves.toBe("committed");
|
||||
});
|
||||
|
||||
it("fails loudly on a corrupt commit-log file instead of silently forgetting its contents", async () => {
|
||||
await fs.mkdir(path.dirname(storePath), { recursive: true });
|
||||
await fs.writeFile(storePath, "{not valid json", "utf8");
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-1", [])).rejects.toThrow();
|
||||
await expect(store.tryCommit("turn-1")).resolves.toBe("committed");
|
||||
});
|
||||
|
||||
it("serializes concurrent commits of the same key so exactly one wins", async () => {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
const results = await Promise.all([
|
||||
store.tryCommit("turn-race", []),
|
||||
store.tryCommit("turn-race", []),
|
||||
store.tryCommit("turn-race", []),
|
||||
store.tryCommit("turn-race"),
|
||||
store.tryCommit("turn-race"),
|
||||
store.tryCommit("turn-race"),
|
||||
]);
|
||||
|
||||
expect(results.filter((r) => r === "committed")).toHaveLength(1);
|
||||
@@ -129,98 +122,304 @@ describe("DurableAdvancementKeyStore", () => {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
for (let i = 0; i < 600; i++) {
|
||||
await store.tryCommit(`turn-${i}`, []);
|
||||
await store.tryCommit(`turn-${i}`);
|
||||
}
|
||||
|
||||
await expect(store.tryCommit("turn-0", [])).resolves.toBe("duplicate");
|
||||
await expect(store.tryCommit("turn-0")).resolves.toBe("duplicate");
|
||||
},
|
||||
20_000,
|
||||
);
|
||||
|
||||
// ---------------------------------------------------------------------
|
||||
// PR #3442 review, round 2: three additional atomic-commit failures.
|
||||
// ---------------------------------------------------------------------
|
||||
|
||||
it("persists the accepted messages together with the key, not just the key", async () => {
|
||||
it("records each commit as one small marker holding only the key", async () => {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
const messages = [
|
||||
{ role: "user", content: "what's the weather?" },
|
||||
{ role: "assistant", content: "Let me check." },
|
||||
];
|
||||
|
||||
await store.tryCommit("turn-1", messages);
|
||||
await store.tryCommit("turn-1");
|
||||
|
||||
const entry = await store.get("turn-1");
|
||||
expect(entry).toBeDefined();
|
||||
expect(entry?.messages).toEqual(messages);
|
||||
expect(typeof entry?.committedAt).toBe("string");
|
||||
expect(entry?.committedAt.length).toBeGreaterThan(0);
|
||||
const files = await filesUnder(markerDirFor(storePath));
|
||||
expect(files).toHaveLength(1);
|
||||
const record = JSON.parse(await fs.readFile(files[0], "utf8"));
|
||||
expect(record.advancementKey).toBe("turn-1");
|
||||
expect(typeof record.committedAt).toBe("string");
|
||||
expect((await fs.stat(files[0])).size).toBeLessThan(200);
|
||||
});
|
||||
|
||||
// Survives a restart too -- the messages aren't an in-memory-only echo.
|
||||
const after = new DurableAdvancementKeyStore(storePath);
|
||||
const reloaded = await after.get("turn-1");
|
||||
expect(reloaded?.messages).toEqual(messages);
|
||||
it("never rewrites an earlier commit's record when later keys are committed", async () => {
|
||||
// Regression: every commit used to re-read and rewrite one file holding every
|
||||
// key so far, so a commit's cost grew with the number of turns ever committed.
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
await store.tryCommit("turn-0");
|
||||
const [first] = await filesUnder(markerDirFor(storePath));
|
||||
const before = await fs.stat(first);
|
||||
|
||||
for (let i = 1; i <= 50; i++) {
|
||||
await store.tryCommit(`turn-${i}`);
|
||||
}
|
||||
|
||||
const after = await fs.stat(first);
|
||||
expect(after.ino).toBe(before.ino);
|
||||
expect(after.mtimeMs).toBe(before.mtimeMs);
|
||||
});
|
||||
|
||||
it("does not remember a key when storage is inaccessible, so a retry can still succeed", async () => {
|
||||
// Block the parent directory with a file. Both the commit's initial
|
||||
// read and a membership lookup must propagate the filesystem error.
|
||||
// Block the parent directory with a file. Both a commit and a membership
|
||||
// lookup must propagate the filesystem error.
|
||||
const blockedParent = path.join(dir, "blocked");
|
||||
await fs.writeFile(blockedParent, "not a directory", "utf8");
|
||||
const blockedPath = path.join(blockedParent, "commit-log.json");
|
||||
const store = new DurableAdvancementKeyStore(blockedPath);
|
||||
|
||||
await expect(store.tryCommit("turn-1", [])).rejects.toThrow();
|
||||
// The failed write must not be remembered in memory: retrying the same
|
||||
// key must attempt the commit again (and fail again, for the same
|
||||
// reason) rather than silently reporting "duplicate" for a commit that
|
||||
// never reached disk.
|
||||
await expect(store.tryCommit("turn-1", [])).rejects.toThrow();
|
||||
// The error names the setting that fixes it.
|
||||
await expect(store.tryCommit("turn-1")).rejects.toThrow(/HEADROOM_WORKSPACE_DIR/);
|
||||
// The failed write must not be remembered: retrying the same key must
|
||||
// attempt the commit again (and fail again, for the same reason) rather
|
||||
// than report "duplicate" for a commit that never reached disk.
|
||||
await expect(store.tryCommit("turn-1")).rejects.toThrow();
|
||||
await expect(store.has("turn-1")).rejects.toThrow();
|
||||
|
||||
// Once the underlying problem is fixed, the key is still absent and
|
||||
// the same key commits normally.
|
||||
// Once the underlying problem is fixed, the key is still absent and the
|
||||
// same key commits normally.
|
||||
await fs.rm(blockedParent, { force: true });
|
||||
await expect(store.has("turn-1")).resolves.toBe(false);
|
||||
await expect(store.tryCommit("turn-1", [])).resolves.toBe("committed");
|
||||
await expect(store.tryCommit("turn-1")).resolves.toBe("committed");
|
||||
await expect(store.has("turn-1")).resolves.toBe(true);
|
||||
});
|
||||
|
||||
it("serializes across store instances sharing the same file, not only within one object", async () => {
|
||||
// Regression: creating a marker's directory and creating the marker itself
|
||||
// both fail with EEXIST, and both were read as "duplicate". A file where the
|
||||
// shard directory belongs acknowledged a key that was never recorded.
|
||||
it.each([
|
||||
["the marker directory", (storePath: string) => markerDirFor(storePath)],
|
||||
["the key's shard directory", (storePath: string) => path.dirname(markerPathFor(storePath, "turn-1"))],
|
||||
])("reports a file in place of %s as an error, never as a duplicate", async (_where, blockedPath) => {
|
||||
const blocker = blockedPath(storePath);
|
||||
await fs.mkdir(path.dirname(blocker), { recursive: true });
|
||||
await fs.writeFile(blocker, "not a directory", "utf8");
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-1")).rejects.toThrow(/HEADROOM_WORKSPACE_DIR/);
|
||||
await expect(store.tryCommit("turn-1")).rejects.toThrow(/HEADROOM_WORKSPACE_DIR/);
|
||||
|
||||
await fs.rm(blocker);
|
||||
await expect(store.tryCommit("turn-1")).resolves.toBe("committed");
|
||||
await expect(store.tryCommit("turn-1")).resolves.toBe("duplicate");
|
||||
});
|
||||
|
||||
it("reports anything but a marker file at a marker's path as an error, never as a duplicate", async () => {
|
||||
const marker = markerPathFor(storePath, "turn-1");
|
||||
await fs.mkdir(marker, { recursive: true });
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-1")).rejects.toThrow(/HEADROOM_WORKSPACE_DIR/);
|
||||
await expect(store.has("turn-1")).resolves.toBe(false);
|
||||
|
||||
await fs.rm(marker, { recursive: true });
|
||||
await expect(store.tryCommit("turn-1")).resolves.toBe("committed");
|
||||
await expect(store.has("turn-1")).resolves.toBe(true);
|
||||
});
|
||||
|
||||
it("is never blocked by a lock file a crashed process left behind", async () => {
|
||||
// Regression: the previous store serialized commits with `<path>.lock` and,
|
||||
// by design, never reclaimed it, so a gateway killed mid-commit left every
|
||||
// later commit waiting 60 seconds and then failing until someone deleted it.
|
||||
await fs.mkdir(path.dirname(storePath), { recursive: true });
|
||||
await fs.writeFile(`${storePath}.lock`, "4242-1-abc\n4242\n2026-10-14T00:00:00.000Z\n", "utf8");
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-1")).resolves.toBe("committed");
|
||||
}, 5_000);
|
||||
|
||||
it("commits through store instances sharing the same directory, not only within one object", async () => {
|
||||
const storeA = new DurableAdvancementKeyStore(storePath);
|
||||
const storeB = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
// Interleaved commits through two independent instances against the
|
||||
// same file must not lose either one to a read-modify-write race.
|
||||
const [resultA, resultB] = await Promise.all([
|
||||
storeA.tryCommit("turn-a", ["from a"]),
|
||||
storeB.tryCommit("turn-b", ["from b"]),
|
||||
]);
|
||||
const [resultA, resultB] = await Promise.all([storeA.tryCommit("turn-a"), storeB.tryCommit("turn-b")]);
|
||||
|
||||
expect(resultA).toBe("committed");
|
||||
expect(resultB).toBe("committed");
|
||||
|
||||
// A third, fresh instance (simulating a restart) must see both.
|
||||
const after = new DurableAdvancementKeyStore(storePath);
|
||||
await expect(after.has("turn-a")).resolves.toBe(true);
|
||||
await expect(after.has("turn-b")).resolves.toBe(true);
|
||||
await expect(after.tryCommit("turn-a", [])).resolves.toBe("duplicate");
|
||||
await expect(after.tryCommit("turn-b", [])).resolves.toBe("duplicate");
|
||||
await expect(after.tryCommit("turn-a")).resolves.toBe("duplicate");
|
||||
await expect(after.tryCommit("turn-b")).resolves.toBe("duplicate");
|
||||
});
|
||||
|
||||
describe("a commit log written by an earlier version", () => {
|
||||
async function writeLegacyLog(contents: unknown): Promise<void> {
|
||||
await fs.mkdir(path.dirname(storePath), { recursive: true });
|
||||
await fs.writeFile(storePath, JSON.stringify(contents), "utf8");
|
||||
}
|
||||
|
||||
/** Rewrite the log the way an older version commits a key: a temp file renamed over it. */
|
||||
async function olderVersionWrites(contents: unknown): Promise<void> {
|
||||
const temp = `${storePath}.tmp`;
|
||||
await fs.writeFile(temp, JSON.stringify(contents), "utf8");
|
||||
await fs.rename(temp, storePath);
|
||||
}
|
||||
|
||||
it("keeps its keys committed and leaves the file as it was, for any older version still using it", async () => {
|
||||
await writeLegacyLog({
|
||||
"turn-old": { messages: [{ role: "user", content: "hi" }], committedAt: "2026-10-01T00:00:00.000Z" },
|
||||
});
|
||||
const log = await fs.readFile(storePath, "utf8");
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-old")).resolves.toBe("duplicate");
|
||||
await expect(store.tryCommit("turn-new")).resolves.toBe("committed");
|
||||
await expect(fs.readFile(storePath, "utf8")).resolves.toBe(log);
|
||||
});
|
||||
|
||||
it("reads the key-only array format too", async () => {
|
||||
await writeLegacyLog(["turn-a", "turn-b"]);
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-a")).resolves.toBe("duplicate");
|
||||
await expect(store.tryCommit("turn-b")).resolves.toBe("duplicate");
|
||||
});
|
||||
|
||||
it("is migrated before the first commit of every store, so no store re-accepts a legacy key", async () => {
|
||||
await writeLegacyLog({ "turn-old": { messages: [], committedAt: "" } });
|
||||
|
||||
const results = await Promise.all([
|
||||
new DurableAdvancementKeyStore(storePath).tryCommit("turn-old"),
|
||||
new DurableAdvancementKeyStore(storePath).tryCommit("turn-old"),
|
||||
]);
|
||||
|
||||
expect(results).toEqual(["duplicate", "duplicate"]);
|
||||
});
|
||||
|
||||
it("never re-accepts a key an older version commits while the log is being migrated", async () => {
|
||||
await writeLegacyLog({ "turn-a": { messages: [], committedAt: "" }, "turn-b": { messages: [], committedAt: "" } });
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
// The older version commits turn-late after this store has read the log,
|
||||
// while it is creating the markers for what it read.
|
||||
const internals = store as unknown as { createMarker(key: string, committedAt: string): Promise<boolean> };
|
||||
const createMarker = internals.createMarker.bind(store);
|
||||
let olderVersionCommitted = false;
|
||||
internals.createMarker = async (key, committedAt) => {
|
||||
if (!olderVersionCommitted) {
|
||||
olderVersionCommitted = true;
|
||||
await olderVersionWrites({
|
||||
"turn-a": { committedAt: "" },
|
||||
"turn-b": { committedAt: "" },
|
||||
"turn-late": { committedAt: "" },
|
||||
});
|
||||
}
|
||||
return createMarker(key, committedAt);
|
||||
};
|
||||
|
||||
await expect(store.tryCommit("turn-new")).resolves.toBe("committed");
|
||||
expect(olderVersionCommitted).toBe(true);
|
||||
await expect(store.has("turn-late")).resolves.toBe(true);
|
||||
await expect(store.tryCommit("turn-late")).resolves.toBe("duplicate");
|
||||
await expect(new DurableAdvancementKeyStore(storePath).tryCommit("turn-late")).resolves.toBe("duplicate");
|
||||
});
|
||||
|
||||
it("creates markers only for the keys an older version adds when it rewrites the log", async () => {
|
||||
await writeLegacyLog({ "turn-a": { messages: [], committedAt: "" }, "turn-b": { messages: [], committedAt: "" } });
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
await expect(store.tryCommit("turn-new")).resolves.toBe("committed");
|
||||
|
||||
const internals = store as unknown as { createMarker(key: string, committedAt: string): Promise<boolean> };
|
||||
const createMarker = internals.createMarker.bind(store);
|
||||
const markerAttempts: string[] = [];
|
||||
internals.createMarker = async (key, committedAt) => {
|
||||
markerAttempts.push(key);
|
||||
return createMarker(key, committedAt);
|
||||
};
|
||||
await olderVersionWrites({
|
||||
"turn-a": { committedAt: "" },
|
||||
"turn-b": { committedAt: "" },
|
||||
"turn-c": { committedAt: "" },
|
||||
});
|
||||
|
||||
await expect(store.tryCommit("turn-next")).resolves.toBe("committed");
|
||||
expect(markerAttempts).toEqual(["turn-c", "turn-next"]);
|
||||
await expect(store.tryCommit("turn-c")).resolves.toBe("duplicate");
|
||||
});
|
||||
|
||||
// Permission bits don't stop root, and Windows doesn't apply them the same way.
|
||||
it.skipIf(process.platform === "win32" || process.getuid?.() === 0)(
|
||||
"keeps committing when the migration record can't be written",
|
||||
async () => {
|
||||
await writeLegacyLog({ "turn-old": { messages: [], committedAt: "" } });
|
||||
// A record this process may not overwrite, say one another user wrote.
|
||||
const record = path.join(markerDirFor(storePath), "migrated-legacy-log.json");
|
||||
await fs.mkdir(markerDirFor(storePath), { recursive: true });
|
||||
await fs.writeFile(record, "{}\n", "utf8");
|
||||
await fs.chmod(record, 0o444);
|
||||
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
await expect(store.tryCommit("turn-old")).resolves.toBe("duplicate");
|
||||
await expect(store.tryCommit("turn-new")).resolves.toBe("committed");
|
||||
// A store that starts later reads the log again, and still knows its keys.
|
||||
await expect(new DurableAdvancementKeyStore(storePath).tryCommit("turn-old")).resolves.toBe("duplicate");
|
||||
},
|
||||
);
|
||||
|
||||
// Permission bits don't stop root, and Windows doesn't apply them to directories.
|
||||
it.skipIf(process.platform === "win32" || process.getuid?.() === 0)(
|
||||
"keeps commits working while the log's directory is read-only, and migrates what an older version adds later",
|
||||
async () => {
|
||||
// A writable marker volume mounted beneath a read-only directory that
|
||||
// holds the legacy log: its keys migrate without writing to that directory.
|
||||
await writeLegacyLog({ "turn-a": { messages: [], committedAt: "" } });
|
||||
const written = new Date("2026-10-01T00:00:00.000Z");
|
||||
await fs.utimes(storePath, written, written);
|
||||
const logDir = path.dirname(storePath);
|
||||
await fs.mkdir(markerDirFor(storePath));
|
||||
await fs.chmod(logDir, 0o555);
|
||||
try {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
await expect(store.tryCommit("turn-a")).resolves.toBe("duplicate");
|
||||
await expect(store.tryCommit("turn-new")).resolves.toBe("committed");
|
||||
const log = await fs.readFile(storePath, "utf8");
|
||||
expect(log).toContain("turn-a");
|
||||
|
||||
// A restart doesn't read the unchanged log again. Probe: contents that
|
||||
// can't be parsed, at the same size and mtime.
|
||||
await fs.writeFile(storePath, "x".repeat(log.length), "utf8");
|
||||
await fs.utimes(storePath, written, written);
|
||||
const restarted = new DurableAdvancementKeyStore(storePath);
|
||||
await expect(restarted.tryCommit("turn-next")).resolves.toBe("committed");
|
||||
|
||||
// Nor does a migrated log that can no longer be read block commits.
|
||||
await fs.chmod(storePath, 0o000);
|
||||
await expect(new DurableAdvancementKeyStore(storePath).tryCommit("turn-c")).resolves.toBe("committed");
|
||||
|
||||
// The directory becomes writable, and an older version that is still
|
||||
// running commits turn-b the way it always did (temp file + rename).
|
||||
// The running store migrates it before its next commit.
|
||||
await fs.chmod(logDir, 0o755);
|
||||
await olderVersionWrites({ "turn-a": { committedAt: "" }, "turn-b": { committedAt: "" } });
|
||||
await expect(restarted.tryCommit("turn-b")).resolves.toBe("duplicate");
|
||||
} finally {
|
||||
await fs.chmod(logDir, 0o755);
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("fails loudly when corrupt instead of silently forgetting its keys", async () => {
|
||||
await fs.mkdir(path.dirname(storePath), { recursive: true });
|
||||
await fs.writeFile(storePath, "{not valid json", "utf8");
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-1")).rejects.toThrow();
|
||||
// Fixing the file lets the same store recover without a restart.
|
||||
await fs.writeFile(storePath, JSON.stringify(["turn-1"]), "utf8");
|
||||
await expect(store.tryCommit("turn-1")).resolves.toBe("duplicate");
|
||||
});
|
||||
});
|
||||
|
||||
// -------------------------------------------------------------------
|
||||
// PR #3442 review, round 3: an in-process lock (a JS Map/promise queue)
|
||||
// does nothing to stop two separate OS processes -- e.g. two gateway
|
||||
// processes sharing one Headroom workspace -- from racing the same
|
||||
// commit-log file. These spawn the real DurableAdvancementKeyStore in
|
||||
// real child processes (not Worker threads, which share the parent's
|
||||
// memory and wouldn't exercise inter-process file locking) and release
|
||||
// them via a shared "go" file so their read/check/write transactions
|
||||
// contend for the file lock as close to simultaneously as possible.
|
||||
// Two separate OS processes -- e.g. two gateway processes sharing one
|
||||
// Headroom workspace -- committing at the same moment. These spawn the
|
||||
// real DurableAdvancementKeyStore in real child processes (not Worker
|
||||
// threads, which share the parent's memory) and release them via a shared
|
||||
// "go" file so their commits race as closely as possible.
|
||||
// -------------------------------------------------------------------
|
||||
|
||||
it.skipIf(!NODE_SUPPORTS_TRANSFORM_TYPES)(
|
||||
"serializes commits from two separate OS processes sharing the same file",
|
||||
"keeps commits from two separate OS processes sharing the same directory",
|
||||
async () => {
|
||||
const goPath = path.join(dir, "go");
|
||||
const resultAPath = path.join(dir, "result-a.json");
|
||||
@@ -238,7 +437,6 @@ describe("DurableAdvancementKeyStore", () => {
|
||||
expect(resultA).toEqual({ status: "committed" });
|
||||
expect(resultB).toEqual({ status: "committed" });
|
||||
|
||||
// Neither commit was lost to a cross-process race.
|
||||
const after = new DurableAdvancementKeyStore(storePath);
|
||||
await expect(after.has("turn-a")).resolves.toBe(true);
|
||||
await expect(after.has("turn-b")).resolves.toBe(true);
|
||||
@@ -260,65 +458,13 @@ describe("DurableAdvancementKeyStore", () => {
|
||||
|
||||
const [resultA, resultB] = await Promise.all([workerA, workerB]);
|
||||
|
||||
// Exactly one process's transaction should win the race and see
|
||||
// "committed"; the other must see "duplicate", never both
|
||||
// "committed" (which would mean the file lock let them interleave).
|
||||
// Exactly one process creates the marker and sees "committed"; the
|
||||
// other must see "duplicate", never both "committed".
|
||||
const statuses = [resultA.status, resultB.status].sort();
|
||||
expect(statuses).toEqual(["committed", "duplicate"]);
|
||||
},
|
||||
30_000,
|
||||
);
|
||||
|
||||
it.skipIf(!NODE_SUPPORTS_TRANSFORM_TYPES)(
|
||||
"never lets a second writer commit while a live holder still holds the lock",
|
||||
async () => {
|
||||
// Regression for PR #3442 review, round 4: a lock reclaimed on an
|
||||
// age-based staleness guess let a second process commit and release
|
||||
// while the first (merely slow, not dead) holder still believed it
|
||||
// owned the lock -- the first then resumed and overwrote the
|
||||
// second's accepted commit with its own stale snapshot. This proves
|
||||
// the invariant that replaces that heuristic: a second writer must
|
||||
// never proceed while ANY holder -- however long it takes -- still
|
||||
// holds the lock. Deterministic and fast: the "holder" is a worker
|
||||
// that occupies the real lock file and only releases it when told
|
||||
// to, rather than relying on a real multi-second sleep.
|
||||
const readyPath = path.join(dir, "holder-ready");
|
||||
const releasePath = path.join(dir, "holder-release");
|
||||
|
||||
const holder = execFileAsync(process.execPath, [
|
||||
"--experimental-transform-types",
|
||||
HOLD_LOCK_WORKER_PATH,
|
||||
storePath,
|
||||
readyPath,
|
||||
releasePath,
|
||||
]);
|
||||
|
||||
// Wait until the holder has actually acquired the lock before racing
|
||||
// a commit against it.
|
||||
await waitForFile(readyPath);
|
||||
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
let secondCommitSettled = false;
|
||||
const secondCommit = store.tryCommit("turn-b", ["b"]).finally(() => {
|
||||
secondCommitSettled = true;
|
||||
});
|
||||
|
||||
// Give the second commit ample opportunity to (wrongly) proceed
|
||||
// while the holder is still alive and has not released.
|
||||
try {
|
||||
await new Promise((r) => setTimeout(r, 500));
|
||||
expect(secondCommitSettled).toBe(false);
|
||||
} finally {
|
||||
// Release even if the assertion fails, so the child cannot leak.
|
||||
await fs.writeFile(releasePath, "go", "utf8");
|
||||
await holder;
|
||||
}
|
||||
// The second commit must now complete successfully, with nothing lost.
|
||||
await expect(secondCommit).resolves.toBe("committed");
|
||||
await expect(store.has("turn-b")).resolves.toBe(true);
|
||||
},
|
||||
15_000,
|
||||
);
|
||||
});
|
||||
|
||||
describe("defaultCommitLogPath", () => {
|
||||
@@ -347,4 +493,10 @@ describe("defaultCommitLogPath", () => {
|
||||
path.join(os.homedir(), ".headroom", "openclaw", "commit-log.json"),
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps the markers beside the commit log", () => {
|
||||
expect(markerDirFor(path.join("/w", "openclaw", "commit-log.json"))).toBe(
|
||||
path.join("/w", "openclaw", "commit-log.d"),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -452,17 +452,28 @@ describe("HeadroomContextEngine transcriptSemantics contract", () => {
|
||||
20_000,
|
||||
);
|
||||
|
||||
it("persists the accepted messages together with the advancement key", async () => {
|
||||
it("records only the advancement key, so the store doesn't grow with the turn's size", async () => {
|
||||
// Regression: every accepted turn's messages used to be kept forever in one
|
||||
// file that was rewritten on each commit. OpenClaw's transcript already holds
|
||||
// them, and this engine never reads them back.
|
||||
const engine = new HeadroomContextEngine({ commitLogPath });
|
||||
const messages = [{ role: "user", content: "hello" }];
|
||||
const toolOutput = "x".repeat(1_000_000);
|
||||
|
||||
await expect(
|
||||
engine.commitTurn({ advancementKey: "turn-1", messages }),
|
||||
engine.commitTurn({
|
||||
advancementKey: "turn-1",
|
||||
messages: [{ role: "toolResult", content: [{ type: "text", text: toolOutput }] }],
|
||||
}),
|
||||
).resolves.toEqual({ status: "committed" });
|
||||
|
||||
const raw = await fs.readFile(commitLogPath, "utf8");
|
||||
const entries = JSON.parse(raw) as Record<string, { messages: unknown }>;
|
||||
expect(entries["turn-1"].messages).toEqual(messages);
|
||||
let storedBytes = 0;
|
||||
for (const entry of await fs.readdir(commitLogDir, { recursive: true, withFileTypes: true })) {
|
||||
if (entry.isFile()) {
|
||||
storedBytes += (await fs.stat(path.join(entry.parentPath, entry.name))).size;
|
||||
}
|
||||
}
|
||||
expect(storedBytes).toBeGreaterThan(0);
|
||||
expect(storedBytes).toBeLessThan(1_000);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
+6
-7
@@ -1,18 +1,17 @@
|
||||
/**
|
||||
* Child-process worker for the cross-process DurableAdvancementKeyStore
|
||||
* regression tests (PR #3442 review, round 3).
|
||||
* tests.
|
||||
*
|
||||
* Run as a real, separate OS process (not a Worker thread, which shares the
|
||||
* parent's memory and wouldn't exercise inter-process file locking) via
|
||||
* `node --experimental-transform-types commit-turn-worker.ts <storePath>
|
||||
* <key> <goPath> <resultPath>`.
|
||||
* parent's memory) via `node --experimental-transform-types
|
||||
* commit-turn-worker.ts <storePath> <key> <goPath> <resultPath>`.
|
||||
*
|
||||
* Blocks until `goPath` exists (the parent test creates it only after every
|
||||
* worker has been spawned and had time to reach this poll loop), then
|
||||
* commits `key` through the real `DurableAdvancementKeyStore` and writes
|
||||
* the outcome to `resultPath` as JSON -- this is what lets two independently
|
||||
* launched processes race their read/check/write transactions against the
|
||||
* same commit-log file as closely to simultaneously as possible.
|
||||
* launched processes race their commits against the same store as closely
|
||||
* to simultaneously as possible.
|
||||
*/
|
||||
|
||||
import { access, writeFile } from "node:fs/promises";
|
||||
@@ -45,7 +44,7 @@ async function main(): Promise<void> {
|
||||
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
try {
|
||||
const status = await store.tryCommit(key, [key]);
|
||||
const status = await store.tryCommit(key);
|
||||
await writeFile(resultPath, JSON.stringify({ status }), "utf8");
|
||||
} catch (error) {
|
||||
await writeFile(
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
/**
|
||||
* Simulates a live-but-slow lock holder for the DurableAdvancementKeyStore
|
||||
* cross-process regression tests (PR #3442 review, round 4): acquires the
|
||||
* exact same lock file the real store uses, signals readiness, then holds
|
||||
* it until told to release -- proving a real, still-alive holder is never
|
||||
* force-preempted no matter how long its critical section runs, unlike the
|
||||
* round-3 age-based staleness heuristic this replaces.
|
||||
*
|
||||
* Deliberately does not import `DurableAdvancementKeyStore`: it only needs
|
||||
* to occupy the lock file at the path the real class would use, using the
|
||||
* same `open(path, "wx")` protocol, so the *other* worker's real
|
||||
* `tryCommit` call is what gets exercised and observed.
|
||||
*
|
||||
* Usage: node --experimental-transform-types hold-lock-worker.ts
|
||||
* <storePath> <readyPath> <releasePath>
|
||||
*/
|
||||
|
||||
import { access, mkdir, open, unlink, utimes, writeFile } from "node:fs/promises";
|
||||
import { dirname } from "node:path";
|
||||
|
||||
async function waitFor(p: string): Promise<void> {
|
||||
for (;;) {
|
||||
try {
|
||||
await access(p);
|
||||
return;
|
||||
} catch {
|
||||
await new Promise((resolveDelay) => setTimeout(resolveDelay, 5));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const [, , storePath, readyPath, releasePath] = process.argv;
|
||||
if (!storePath || !readyPath || !releasePath) {
|
||||
throw new Error("usage: hold-lock-worker.ts <storePath> <readyPath> <releasePath>");
|
||||
}
|
||||
|
||||
const lockPath = `${storePath}.lock`;
|
||||
await mkdir(dirname(storePath), { recursive: true });
|
||||
const handle = await open(lockPath, "wx");
|
||||
await handle.close();
|
||||
// Exercise the former 30-second reclaim threshold without a slow test.
|
||||
const oldTime = new Date(Date.now() - 60_000);
|
||||
await utimes(lockPath, oldTime, oldTime);
|
||||
|
||||
await writeFile(readyPath, "ready", "utf8");
|
||||
|
||||
await waitFor(releasePath);
|
||||
await unlink(lockPath).catch(() => undefined);
|
||||
}
|
||||
|
||||
main();
|
||||
Reference in New Issue
Block a user