fix(openclaw): declare transcriptSemantics and implement commitTurn (#3442)

## Description

Adds OpenClaw transcript-semantics declarations and an accepted-turn log
for admitted-turn advancement. Fixes #3441.

Messages and advancement keys are written together in
`HEADROOM_WORKSPACE_DIR/openclaw/commit-log.json` (default workspace:
`~/.headroom`). Each commit re-reads the log inside an exclusive
sibling-file lock and replaces the log atomically. Keys are retained
without eviction; prior key-only logs are recognized for migration.
Failed writes do not create an in-memory phantom duplicate.

## Locking and recovery

The lock serializes processes sharing the workspace and is never
reclaimed merely because it is old. A waiting caller times out after 60
seconds instead of committing without ownership. Release checks the
acquisition token.

A crashed holder can leave an orphan `.lock` file. Recovery is
operational: stop writers using that workspace, establish that the
holder is no longer running, remove the orphan lock, then resume. Do not
delete a live writer's lock. Automatic crash recovery and power-loss
durability are not claimed.

## Testing

- 99 plugin tests pass, including distinct-key and same-key
child-process commits and an aged live-lock regression.
- TypeScript checking passes.
- The aged-lock regression fails against the prior age-reclaim
implementation and passes against the current source. Test cleanup
releases the child even on assertion failure.
- Current-head CI is pending.

## Real Behavior Proof

- Environment: macOS, two real Node child processes importing
`DurableAdvancementKeyStore`.
- Exact command / steps: A acquires the lock and reads the empty log;
pause A while alive for 31 seconds. Start B committing another key and
verify it remains pending, then resume A.
- Observed result: B does not finish while A holds the lock. After
release, both return `committed` and both message payloads remain in the
log. The prior implementation acknowledged both but lost one turn.
- Not tested: automatic recovery from crashed holders (requires the
operator procedure above); power-loss durability.

## Runtime Rollout Safety

- Rollout-managed feature(s): OpenClaw transcript-semantics declaration.
- Minimum rollout channel: Host capability negotiation.
- Stable/default behavior changed: Accepted messages and keys persist in
a workspace-shared log. The log grows without eviction, and an orphan
lock requires operator recovery.
- Kill switch / disable path: Revert the plugin change or use the host's
legacy context-engine path.
- Unsafe override required: No.
- Qualification impact: Admission uses atomic/idempotent commit
behavior; contention fails closed.
- Rollback path: Revert the plugin change. Preserve the log if retry
history must remain available.

## Review Readiness

- [x] Prior message persistence, retry, cross-process, and stale-lock
findings resolved.
- [x] Local tests and type checking pass.
- [ ] Current-head CI complete.
- Final human review is required.

---------

Signed-off-by: Radhakrishnan P <gingeekrishna@gmail.com>
Co-authored-by: Jerrett Davis <mxjerrett@gmail.com>
This commit is contained in:
Radhakrishnan Pachyappan
2026-09-24 17:17:46 -05:00
committed by GitHub
co-authored by Jerrett Davis
parent a102e05b51
commit a28dd4eab7
6 changed files with 838 additions and 1 deletions
@@ -0,0 +1,240 @@
/**
* Durable, atomic, restart-safe, cross-process-safe record of committed
* turn advancements.
*
* OpenClaw's `atomic-idempotent-v1` transcriptSemantics contract requires
* that a retried `commitTurn` call report "duplicate" for a key that was
* already committed -- including after this process (or the host) restarts
* -- and that "the accepted turn and key advance together" (the committed
* messages are persisted atomically with the key, not discarded).
*
* The default commit-log path is shared by every gateway process using the
* same Headroom workspace, not just every store instance in one process --
* an in-process mutex (a JS `Map`/promise queue) does nothing to stop two
* separate OS processes from each reading the same file, adding a different
* key, and racing their renames, with the loser's accepted turn silently
* discarded. The read/check/write transaction is therefore protected by a
* real inter-process lock: an exclusive lock FILE (`<path>.lock`), acquired
* via atomic exclusive creation (`open(..., "wx")`, which is atomic at the
* OS level on POSIX and Windows alike) so only one process -- in this one or
* any other -- can be inside the critical section at a time.
*
* The lock is never force-broken by age. An age-based "the holder must have
* crashed" heuristic cannot tell a dead holder apart from one that is simply
* slow (a large write, a GC pause, disk contention): a second process that
* reclaims a lock the first still holds can commit and release before the
* first resumes and overwrites that commit with its own now-stale snapshot
* -- both processes report "committed", but only the second process's write
* survives, and the first process's genuinely-accepted turn silently
* vanishes. Node has no built-in binding for a real OS-mediated lock
* (POSIX `flock`/Windows `LockFileEx`, where the kernel itself releases the
* lock when the holder's process exits, crash included, so no staleness
* guess is needed at all), so this fails closed instead: a lock that is
* never released is never reclaimed, and a caller that can't acquire it
* within the timeout gets a clear error rather than a silent lost write. A
* lock file orphaned by a genuinely crashed process requires a human (or an
* operator script) to remove `<path>.lock` before commits can resume --
* that operational cost is the trade for never silently dropping an
* acknowledged commit. Lock ownership is also verified before release (see
* `withFileLock`): a process only ever deletes the lock file it created.
*
* Every `tryCommit` re-reads the file fresh from disk inside the lock (no
* long-lived in-memory cache), and the in-memory return value is only ever
* "committed" once the atomic (temp file + rename) write recording the key
* has actually succeeded -- a failed write is never remembered as seen, so
* retrying the same key attempts the commit again instead of returning a
* phantom "duplicate" for a commit that never reached disk.
*
* Keys are never evicted. OpenClaw may retry at any point after a restart or
* a long stall, so pruning by count or age would risk re-accepting an
* already-committed turn -- exactly the failure this store exists to avoid.
*/
import { mkdir, open, readFile, rename, stat, unlink, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import * as os from "node:os";
/**
* True if `p` exists and is something other than a directory (e.g. a file
* sitting where a directory needs to be). Some platforms (observed on
* Windows) report `ENOENT` -- not a distinct "not a directory" code -- for
* `readFile` when a path component like this blocks the rest of the path,
* which is indistinguishable from "the commit log just doesn't exist yet"
* by error code alone. Checking the parent directly is what tells the two
* apart: silently treating a broken storage path as an empty store would
* make every future commit re-accept an already-committed key.
*/
async function isBlockingNonDirectory(p: string): Promise<boolean> {
try {
const info = await stat(p);
return !info.isDirectory();
} catch {
return false; // Doesn't exist -- not blocking anything.
}
}
/** Default commit-log location, honoring the same workspace override the
* rest of the Headroom workspace convention uses (`HEADROOM_WORKSPACE_DIR`). */
export function defaultCommitLogPath(): string {
const workspaceDir = process.env.HEADROOM_WORKSPACE_DIR?.trim() || join(os.homedir(), ".headroom");
return join(workspaceDir, "openclaw", "commit-log.json");
}
/** One durably-committed turn: the accepted messages plus when they landed. */
export interface CommittedTurn {
messages: unknown;
committedAt: string;
}
type CommitLog = Record<string, CommittedTurn>;
/** Give up (rather than poll forever) if the lock still can't be acquired after this long. */
const LOCK_ACQUIRE_TIMEOUT_MS = 60_000;
const LOCK_POLL_BASE_MS = 20;
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
/**
* Run `fn` while holding an exclusive lock on `path` (a sibling `<path>.lock`
* file). Serializes every caller -- this process or another -- against the
* same file, not just calls on one object or one process.
*
* The lock is held until `fn` returns, however long that takes, and is
* never reclaimed by another caller on a staleness guess -- see the module
* docstring for why an age-based heuristic can silently drop a genuinely
* accepted commit. A caller that can't acquire the lock within
* `LOCK_ACQUIRE_TIMEOUT_MS` gets a clear timeout error instead of the
* transaction being allowed to proceed unsafely.
*/
async function withFileLock<T>(path: string, fn: () => Promise<T>): Promise<T> {
const lockPath = `${path}.lock`;
await mkdir(dirname(path), { recursive: true });
// Unique per acquisition, so release only ever removes the lock file this
// exact call created -- never one a different (later) holder created.
const token = `${process.pid}-${Date.now()}-${Math.random().toString(36).slice(2)}`;
const deadline = Date.now() + LOCK_ACQUIRE_TIMEOUT_MS;
let attempt = 0;
for (;;) {
try {
const handle = await open(lockPath, "wx");
try {
await handle.writeFile(`${token}\n${process.pid}\n${new Date().toISOString()}\n`);
} finally {
await handle.close();
}
break;
} catch (error) {
if ((error as NodeJS.ErrnoException)?.code !== "EEXIST") {
throw error;
}
if (Date.now() > deadline) {
throw new Error(
`Timed out waiting for the advancement-key-store lock on ${path}. If the ` +
`process that created ${lockPath} is no longer running, remove it manually.`,
);
}
attempt += 1;
await sleep(LOCK_POLL_BASE_MS + Math.random() * LOCK_POLL_BASE_MS * Math.min(attempt, 10));
}
}
try {
return await fn();
} finally {
// Verify the lock file is still the one this call created before
// removing it -- if it doesn't start with our token, some other process
// (or a human) replaced or is using it, and it must not be touched.
try {
const current = await readFile(lockPath, "utf8");
if (current.startsWith(`${token}\n`)) {
await unlink(lockPath);
}
} catch (error) {
if ((error as NodeJS.ErrnoException)?.code !== "ENOENT") {
throw error;
}
}
}
}
export class DurableAdvancementKeyStore {
constructor(private readonly path: string) {}
private async readAll(): Promise<CommitLog> {
let raw: string;
try {
raw = await readFile(this.path, "utf8");
} catch (error) {
const code = (error as NodeJS.ErrnoException)?.code;
if (code !== "ENOENT" && code !== "ENOTDIR") {
throw error;
}
if (await isBlockingNonDirectory(dirname(this.path))) {
throw error;
}
return {};
}
const parsed = JSON.parse(raw) as unknown;
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
return parsed as CommitLog;
}
if (Array.isArray(parsed)) {
// Migrate the earlier key-only array format (no message payload) so
// upgrading from it never re-accepts an already-committed key.
const migrated: CommitLog = {};
for (const entry of parsed) {
if (typeof entry === "string") {
migrated[entry] = { messages: null, committedAt: "" };
}
}
return migrated;
}
return {};
}
private async writeAll(entries: CommitLog): Promise<void> {
await mkdir(dirname(this.path), { recursive: true });
const tmpPath = `${this.path}.${process.pid}.${Math.random().toString(36).slice(2)}.tmp`;
await writeFile(tmpPath, JSON.stringify(entries), "utf8");
await rename(tmpPath, this.path);
}
/**
* Atomically check-and-record `key` together with the `messages` it
* advances. Holds the cross-process file lock (see `withFileLock`) for
* the full read/check/write transaction, so two processes -- or two
* instances in this one -- can't each read the same starting state and
* race their writes.
*
* Throws if the write fails -- the caller sees a rejected commit, not a
* false "committed"/"duplicate", and a retry re-reads the (still
* unchanged) on-disk state and attempts the write again.
*/
async tryCommit(key: string, messages: unknown): Promise<"committed" | "duplicate"> {
return withFileLock(this.path, async () => {
const entries = await this.readAll();
if (key in entries) {
return "duplicate";
}
const next: CommitLog = {
...entries,
[key]: { messages, committedAt: new Date().toISOString() },
};
await this.writeAll(next);
return "committed";
});
}
/** Test-only: check membership without recording. */
async has(key: string): Promise<boolean> {
const entries = await this.readAll();
return key in entries;
}
/** Test-only: read back a committed entry (messages + commit time) without recording. */
async get(key: string): Promise<CommittedTurn | undefined> {
const entries = await this.readAll();
return entries[key];
}
}
+33
View File
@@ -10,6 +10,7 @@
import { compress } from "headroom-ai";
import { ProxyManager, defaultLogger, type ProxyManagerConfig, type ProxyManagerLogger } from "./proxy-manager.js";
import { agentToOpenAI, normalizeAgentMessages, openAIToAgent } from "./convert.js";
import { DurableAdvancementKeyStore, defaultCommitLogPath } from "./advancement-key-store.js";
import {
delegateCompactionToRuntime,
type OpenClawCompactParams,
@@ -32,6 +33,9 @@ export interface HeadroomEngineConfig extends ProxyManagerConfig {
requestTimeoutMs?: number;
circuitBreakerThreshold?: number;
circuitBreakerCooldownMs?: number;
/** Where to durably record committed turn-advancement keys (see
* `DurableAdvancementKeyStore`). Defaults to `defaultCommitLogPath()`. */
commitLogPath?: string;
}
export class HeadroomContextEngine {
@@ -40,8 +44,17 @@ export class HeadroomContextEngine {
name: "Headroom Context Compression",
version: "0.1.0",
ownsCompaction: false,
transcriptSemantics: {
currentTurnFence: "before-current-turn-entry-v1",
turnAdvancementIdempotency: "atomic-idempotent-v1",
},
};
// Durable, restart-safe record of committed advancement keys, for
// commitTurn's idempotent-retry check. See advancement-key-store.ts for
// why this must survive a process restart and must not evict entries.
private advancementKeyStore: DurableAdvancementKeyStore;
private proxyManager: ProxyManager;
private proxyUrl: string | null = null;
private config: HeadroomEngineConfig;
@@ -61,6 +74,9 @@ export class HeadroomContextEngine {
this.config = config;
this.logger = logger ?? defaultLogger;
this.proxyManager = new ProxyManager(config, this.logger);
this.advancementKeyStore = new DurableAdvancementKeyStore(
config.commitLogPath ?? defaultCommitLogPath(),
);
}
// === ContextEngine Lifecycle ===
@@ -189,6 +205,23 @@ export class HeadroomContextEngine {
return result;
}
/**
* Durable turn-advancement commit — required by OpenClaw's transcriptSemantics
* contract. Called only for the accepted, successful turn; failed or aborted
* turns never reach here. Must be an atomic, idempotent write of the
* accepted `messages` keyed by `advancementKey` so a host retry with the
* same key reports "duplicate" instead of re-applying the advancement —
* including a retry that arrives after this process restarted, which is
* why the record lives on disk (see `DurableAdvancementKeyStore`) rather
* than in memory, and includes the messages rather than just the key.
*/
async commitTurn(params: { advancementKey: string; messages: any[] }): Promise<{
status: "committed" | "duplicate";
}> {
const status = await this.advancementKeyStore.tryCommit(params.advancementKey, params.messages);
return { status };
}
async afterTurn?(params: {
sessionId: string;
messages: any[];
@@ -0,0 +1,350 @@
import { execFile } from "node:child_process";
import * as fs from "node:fs/promises";
import * as os from "node:os";
import * as path from "node:path";
import { fileURLToPath } from "node:url";
import { promisify } from "node:util";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { DurableAdvancementKeyStore, defaultCommitLogPath } from "../src/advancement-key-store.js";
const execFileAsync = promisify(execFile);
// Node's native TypeScript support (needed to run the worker fixture, which
// imports the real DurableAdvancementKeyStore, as a plain `node` child
// process with no build step) requires `--experimental-transform-types`,
// added in Node 22.6. Skip the cross-process tests rather than fail on an
// older Node -- the failure mode being tested (a lost write across two OS
// processes) is orthogonal to which Node version happens to run the test.
const NODE_SUPPORTS_TRANSFORM_TYPES = (() => {
const [major, minor] = process.versions.node.split(".").map(Number);
return major > 22 || (major === 22 && minor >= 6);
})();
const WORKER_PATH = fileURLToPath(new URL("./fixtures/commit-turn-worker.ts", import.meta.url));
const HOLD_LOCK_WORKER_PATH = fileURLToPath(
new URL("./fixtures/hold-lock-worker.ts", import.meta.url),
);
/** Run the commit-turn worker as a real child process; returns its parsed JSON result. */
async function runCommitWorker(
storePath: string,
key: string,
goPath: string,
resultPath: string,
): Promise<{ status?: string; error?: string }> {
await execFileAsync(process.execPath, [
"--experimental-transform-types",
WORKER_PATH,
storePath,
key,
goPath,
resultPath,
]);
return JSON.parse(await fs.readFile(resultPath, "utf8"));
}
async function waitForFile(p: string): Promise<void> {
for (;;) {
try {
await fs.access(p);
return;
} catch {
await new Promise((r) => setTimeout(r, 5));
}
}
}
describe("DurableAdvancementKeyStore", () => {
let dir: string;
let storePath: string;
beforeEach(async () => {
dir = await fs.mkdtemp(path.join(os.tmpdir(), "headroom-advancement-key-store-"));
storePath = path.join(dir, "nested", "commit-log.json");
});
afterEach(async () => {
await fs.rm(dir, { recursive: true, force: true });
});
it("commits a new key and creates the parent directory", async () => {
const store = new DurableAdvancementKeyStore(storePath);
await expect(store.tryCommit("turn-1", [{ role: "user", content: "hi" }])).resolves.toBe(
"committed",
);
await expect(fs.stat(storePath)).resolves.toBeTruthy();
});
it("reports duplicate for an already-committed key", async () => {
const store = new DurableAdvancementKeyStore(storePath);
await expect(store.tryCommit("turn-1", [])).resolves.toBe("committed");
await expect(store.tryCommit("turn-1", [])).resolves.toBe("duplicate");
});
it("persists across a simulated restart (a fresh store instance, same path)", async () => {
const before = new DurableAdvancementKeyStore(storePath);
await expect(before.tryCommit("turn-1", [])).resolves.toBe("committed");
const after = new DurableAdvancementKeyStore(storePath);
await expect(after.tryCommit("turn-1", [])).resolves.toBe("duplicate");
// A key that was never committed is still accepted normally.
await expect(after.tryCommit("turn-2", [])).resolves.toBe("committed");
});
it("treats a missing commit-log file as an empty store rather than an error", async () => {
const store = new DurableAdvancementKeyStore(storePath);
await expect(store.has("turn-1")).resolves.toBe(false);
await expect(store.tryCommit("turn-1", [])).resolves.toBe("committed");
});
it("fails loudly on a corrupt commit-log file instead of silently forgetting its contents", async () => {
await fs.mkdir(path.dirname(storePath), { recursive: true });
await fs.writeFile(storePath, "{not valid json", "utf8");
const store = new DurableAdvancementKeyStore(storePath);
await expect(store.tryCommit("turn-1", [])).rejects.toThrow();
});
it("serializes concurrent commits of the same key so exactly one wins", async () => {
const store = new DurableAdvancementKeyStore(storePath);
const results = await Promise.all([
store.tryCommit("turn-race", []),
store.tryCommit("turn-race", []),
store.tryCommit("turn-race", []),
]);
expect(results.filter((r) => r === "committed")).toHaveLength(1);
expect(results.filter((r) => r === "duplicate")).toHaveLength(2);
});
it(
"never evicts a key regardless of how many others were committed since",
async () => {
const store = new DurableAdvancementKeyStore(storePath);
for (let i = 0; i < 600; i++) {
await store.tryCommit(`turn-${i}`, []);
}
await expect(store.tryCommit("turn-0", [])).resolves.toBe("duplicate");
},
20_000,
);
// ---------------------------------------------------------------------
// PR #3442 review, round 2: three additional atomic-commit failures.
// ---------------------------------------------------------------------
it("persists the accepted messages together with the key, not just the key", async () => {
const store = new DurableAdvancementKeyStore(storePath);
const messages = [
{ role: "user", content: "what's the weather?" },
{ role: "assistant", content: "Let me check." },
];
await store.tryCommit("turn-1", messages);
const entry = await store.get("turn-1");
expect(entry).toBeDefined();
expect(entry?.messages).toEqual(messages);
expect(typeof entry?.committedAt).toBe("string");
expect(entry?.committedAt.length).toBeGreaterThan(0);
// Survives a restart too -- the messages aren't an in-memory-only echo.
const after = new DurableAdvancementKeyStore(storePath);
const reloaded = await after.get("turn-1");
expect(reloaded?.messages).toEqual(messages);
});
it("does not remember a key when storage is inaccessible, so a retry can still succeed", async () => {
// Block the parent directory with a file. Both the commit's initial
// read and a membership lookup must propagate the filesystem error.
const blockedParent = path.join(dir, "blocked");
await fs.writeFile(blockedParent, "not a directory", "utf8");
const blockedPath = path.join(blockedParent, "commit-log.json");
const store = new DurableAdvancementKeyStore(blockedPath);
await expect(store.tryCommit("turn-1", [])).rejects.toThrow();
// The failed write must not be remembered in memory: retrying the same
// key must attempt the commit again (and fail again, for the same
// reason) rather than silently reporting "duplicate" for a commit that
// never reached disk.
await expect(store.tryCommit("turn-1", [])).rejects.toThrow();
await expect(store.has("turn-1")).rejects.toThrow();
// Once the underlying problem is fixed, the key is still absent and
// the same key commits normally.
await fs.rm(blockedParent, { force: true });
await expect(store.has("turn-1")).resolves.toBe(false);
await expect(store.tryCommit("turn-1", [])).resolves.toBe("committed");
await expect(store.has("turn-1")).resolves.toBe(true);
});
it("serializes across store instances sharing the same file, not only within one object", async () => {
const storeA = new DurableAdvancementKeyStore(storePath);
const storeB = new DurableAdvancementKeyStore(storePath);
// Interleaved commits through two independent instances against the
// same file must not lose either one to a read-modify-write race.
const [resultA, resultB] = await Promise.all([
storeA.tryCommit("turn-a", ["from a"]),
storeB.tryCommit("turn-b", ["from b"]),
]);
expect(resultA).toBe("committed");
expect(resultB).toBe("committed");
// A third, fresh instance (simulating a restart) must see both.
const after = new DurableAdvancementKeyStore(storePath);
await expect(after.has("turn-a")).resolves.toBe(true);
await expect(after.has("turn-b")).resolves.toBe(true);
await expect(after.tryCommit("turn-a", [])).resolves.toBe("duplicate");
await expect(after.tryCommit("turn-b", [])).resolves.toBe("duplicate");
});
// -------------------------------------------------------------------
// PR #3442 review, round 3: an in-process lock (a JS Map/promise queue)
// does nothing to stop two separate OS processes -- e.g. two gateway
// processes sharing one Headroom workspace -- from racing the same
// commit-log file. These spawn the real DurableAdvancementKeyStore in
// real child processes (not Worker threads, which share the parent's
// memory and wouldn't exercise inter-process file locking) and release
// them via a shared "go" file so their read/check/write transactions
// contend for the file lock as close to simultaneously as possible.
// -------------------------------------------------------------------
it.skipIf(!NODE_SUPPORTS_TRANSFORM_TYPES)(
"serializes commits from two separate OS processes sharing the same file",
async () => {
const goPath = path.join(dir, "go");
const resultAPath = path.join(dir, "result-a.json");
const resultBPath = path.join(dir, "result-b.json");
const workerA = runCommitWorker(storePath, "turn-a", goPath, resultAPath);
const workerB = runCommitWorker(storePath, "turn-b", goPath, resultBPath);
// Give both worker processes time to spawn and reach their poll loop
// before releasing them together.
await new Promise((r) => setTimeout(r, 300));
await fs.writeFile(goPath, "go", "utf8");
const [resultA, resultB] = await Promise.all([workerA, workerB]);
expect(resultA).toEqual({ status: "committed" });
expect(resultB).toEqual({ status: "committed" });
// Neither commit was lost to a cross-process race.
const after = new DurableAdvancementKeyStore(storePath);
await expect(after.has("turn-a")).resolves.toBe(true);
await expect(after.has("turn-b")).resolves.toBe(true);
},
30_000,
);
it.skipIf(!NODE_SUPPORTS_TRANSFORM_TYPES)(
"reports duplicate for the same key committed from two separate OS processes",
async () => {
const goPath = path.join(dir, "go");
const resultAPath = path.join(dir, "result-a.json");
const resultBPath = path.join(dir, "result-b.json");
const workerA = runCommitWorker(storePath, "turn-same", goPath, resultAPath);
const workerB = runCommitWorker(storePath, "turn-same", goPath, resultBPath);
await new Promise((r) => setTimeout(r, 300));
await fs.writeFile(goPath, "go", "utf8");
const [resultA, resultB] = await Promise.all([workerA, workerB]);
// Exactly one process's transaction should win the race and see
// "committed"; the other must see "duplicate", never both
// "committed" (which would mean the file lock let them interleave).
const statuses = [resultA.status, resultB.status].sort();
expect(statuses).toEqual(["committed", "duplicate"]);
},
30_000,
);
it.skipIf(!NODE_SUPPORTS_TRANSFORM_TYPES)(
"never lets a second writer commit while a live holder still holds the lock",
async () => {
// Regression for PR #3442 review, round 4: a lock reclaimed on an
// age-based staleness guess let a second process commit and release
// while the first (merely slow, not dead) holder still believed it
// owned the lock -- the first then resumed and overwrote the
// second's accepted commit with its own stale snapshot. This proves
// the invariant that replaces that heuristic: a second writer must
// never proceed while ANY holder -- however long it takes -- still
// holds the lock. Deterministic and fast: the "holder" is a worker
// that occupies the real lock file and only releases it when told
// to, rather than relying on a real multi-second sleep.
const readyPath = path.join(dir, "holder-ready");
const releasePath = path.join(dir, "holder-release");
const holder = execFileAsync(process.execPath, [
"--experimental-transform-types",
HOLD_LOCK_WORKER_PATH,
storePath,
readyPath,
releasePath,
]);
// Wait until the holder has actually acquired the lock before racing
// a commit against it.
await waitForFile(readyPath);
const store = new DurableAdvancementKeyStore(storePath);
let secondCommitSettled = false;
const secondCommit = store.tryCommit("turn-b", ["b"]).finally(() => {
secondCommitSettled = true;
});
// Give the second commit ample opportunity to (wrongly) proceed
// while the holder is still alive and has not released.
try {
await new Promise((r) => setTimeout(r, 500));
expect(secondCommitSettled).toBe(false);
} finally {
// Release even if the assertion fails, so the child cannot leak.
await fs.writeFile(releasePath, "go", "utf8");
await holder;
}
// The second commit must now complete successfully, with nothing lost.
await expect(secondCommit).resolves.toBe("committed");
await expect(store.has("turn-b")).resolves.toBe(true);
},
15_000,
);
});
describe("defaultCommitLogPath", () => {
const originalWorkspaceDir = process.env.HEADROOM_WORKSPACE_DIR;
afterEach(() => {
if (originalWorkspaceDir === undefined) {
delete process.env.HEADROOM_WORKSPACE_DIR;
} else {
process.env.HEADROOM_WORKSPACE_DIR = originalWorkspaceDir;
}
});
it("honors HEADROOM_WORKSPACE_DIR when set", () => {
process.env.HEADROOM_WORKSPACE_DIR = "/custom/workspace";
expect(defaultCommitLogPath()).toBe(
path.join("/custom/workspace", "openclaw", "commit-log.json"),
);
});
it("falls back to ~/.headroom when unset", () => {
delete process.env.HEADROOM_WORKSPACE_DIR;
expect(defaultCommitLogPath()).toBe(
path.join(os.homedir(), ".headroom", "openclaw", "commit-log.json"),
);
});
});
+104 -1
View File
@@ -1,4 +1,8 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import * as fs from "node:fs/promises";
import * as os from "node:os";
import * as path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
const mocked = vi.hoisted(() => ({
compress: vi.fn(),
@@ -325,3 +329,102 @@ describe("HeadroomContextEngine proxy startup helpers", () => {
);
});
});
describe("HeadroomContextEngine transcriptSemantics contract", () => {
let commitLogDir: string;
let commitLogPath: string;
beforeEach(async () => {
commitLogDir = await fs.mkdtemp(path.join(os.tmpdir(), "headroom-commit-log-"));
commitLogPath = path.join(commitLogDir, "commit-log.json");
});
afterEach(async () => {
await fs.rm(commitLogDir, { recursive: true, force: true });
});
it("declares the durable-commit transcript semantics OpenClaw requires", () => {
const engine = new HeadroomContextEngine({ commitLogPath });
expect(engine.info.transcriptSemantics).toEqual({
currentTurnFence: "before-current-turn-entry-v1",
turnAdvancementIdempotency: "atomic-idempotent-v1",
});
});
it("commits a new advancement key", async () => {
const engine = new HeadroomContextEngine({ commitLogPath });
await expect(
engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
).resolves.toEqual({ status: "committed" });
});
it("reports duplicate on a retried advancement key", async () => {
const engine = new HeadroomContextEngine({ commitLogPath });
await expect(
engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
).resolves.toEqual({ status: "committed" });
await expect(
engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
).resolves.toEqual({ status: "duplicate" });
});
it("treats distinct advancement keys independently", async () => {
const engine = new HeadroomContextEngine({ commitLogPath });
await expect(
engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
).resolves.toEqual({ status: "committed" });
await expect(
engine.commitTurn({ advancementKey: "turn-2", messages: [] }),
).resolves.toEqual({ status: "committed" });
});
it("reports duplicate for a key committed before a process restart", async () => {
// Simulate a restart: a brand new engine instance (no shared in-memory
// state) pointed at the same durable commit-log path.
const before = new HeadroomContextEngine({ commitLogPath });
await expect(
before.commitTurn({ advancementKey: "turn-restart", messages: [] }),
).resolves.toEqual({ status: "committed" });
const after = new HeadroomContextEngine({ commitLogPath });
await expect(
after.commitTurn({ advancementKey: "turn-restart", messages: [] }),
).resolves.toEqual({ status: "duplicate" });
});
it(
"never forgets a key regardless of how many other keys were committed since",
async () => {
// Regression: the old implementation evicted the oldest key past a
// 512-entry cap, so a retry of an early key was wrongly re-accepted as
// new instead of reported as a duplicate. There is no such cap now.
const engine = new HeadroomContextEngine({ commitLogPath });
for (let i = 0; i < 600; i++) {
await engine.commitTurn({ advancementKey: `turn-${i}`, messages: [] });
}
await expect(
engine.commitTurn({ advancementKey: "turn-0", messages: [] }),
).resolves.toEqual({ status: "duplicate" });
},
20_000,
);
it("persists the accepted messages together with the advancement key", async () => {
const engine = new HeadroomContextEngine({ commitLogPath });
const messages = [{ role: "user", content: "hello" }];
await expect(
engine.commitTurn({ advancementKey: "turn-1", messages }),
).resolves.toEqual({ status: "committed" });
const raw = await fs.readFile(commitLogPath, "utf8");
const entries = JSON.parse(raw) as Record<string, { messages: unknown }>;
expect(entries["turn-1"].messages).toEqual(messages);
});
});
+59
View File
@@ -0,0 +1,59 @@
/**
* Child-process worker for the cross-process DurableAdvancementKeyStore
* regression tests (PR #3442 review, round 3).
*
* Run as a real, separate OS process (not a Worker thread, which shares the
* parent's memory and wouldn't exercise inter-process file locking) via
* `node --experimental-transform-types commit-turn-worker.ts <storePath>
* <key> <goPath> <resultPath>`.
*
* Blocks until `goPath` exists (the parent test creates it only after every
* worker has been spawned and had time to reach this poll loop), then
* commits `key` through the real `DurableAdvancementKeyStore` and writes
* the outcome to `resultPath` as JSON -- this is what lets two independently
* launched processes race their read/check/write transactions against the
* same commit-log file as closely to simultaneously as possible.
*/
import { access, writeFile } from "node:fs/promises";
// Explicit `.ts` extension: this file runs directly under Node's native
// TypeScript support (`--experimental-transform-types`), which -- unlike
// the TypeScript compiler / bundler used to build the plugin itself --
// resolves imports literally and does not map a `.js` specifier onto a
// sibling `.ts` file.
import { DurableAdvancementKeyStore } from "../../src/advancement-key-store.ts";
async function waitForGo(goPath: string): Promise<void> {
for (;;) {
try {
await access(goPath);
return;
} catch {
await new Promise((resolveDelay) => setTimeout(resolveDelay, 5));
}
}
}
async function main(): Promise<void> {
const [, , storePath, key, goPath, resultPath] = process.argv;
if (!storePath || !key || !goPath || !resultPath) {
throw new Error("usage: commit-turn-worker.ts <storePath> <key> <goPath> <resultPath>");
}
await waitForGo(goPath);
const store = new DurableAdvancementKeyStore(storePath);
try {
const status = await store.tryCommit(key, [key]);
await writeFile(resultPath, JSON.stringify({ status }), "utf8");
} catch (error) {
await writeFile(
resultPath,
JSON.stringify({ error: error instanceof Error ? error.message : String(error) }),
"utf8",
);
}
}
main();
+52
View File
@@ -0,0 +1,52 @@
/**
* Simulates a live-but-slow lock holder for the DurableAdvancementKeyStore
* cross-process regression tests (PR #3442 review, round 4): acquires the
* exact same lock file the real store uses, signals readiness, then holds
* it until told to release -- proving a real, still-alive holder is never
* force-preempted no matter how long its critical section runs, unlike the
* round-3 age-based staleness heuristic this replaces.
*
* Deliberately does not import `DurableAdvancementKeyStore`: it only needs
* to occupy the lock file at the path the real class would use, using the
* same `open(path, "wx")` protocol, so the *other* worker's real
* `tryCommit` call is what gets exercised and observed.
*
* Usage: node --experimental-transform-types hold-lock-worker.ts
* <storePath> <readyPath> <releasePath>
*/
import { access, mkdir, open, unlink, utimes, writeFile } from "node:fs/promises";
import { dirname } from "node:path";
async function waitFor(p: string): Promise<void> {
for (;;) {
try {
await access(p);
return;
} catch {
await new Promise((resolveDelay) => setTimeout(resolveDelay, 5));
}
}
}
async function main(): Promise<void> {
const [, , storePath, readyPath, releasePath] = process.argv;
if (!storePath || !readyPath || !releasePath) {
throw new Error("usage: hold-lock-worker.ts <storePath> <readyPath> <releasePath>");
}
const lockPath = `${storePath}.lock`;
await mkdir(dirname(storePath), { recursive: true });
const handle = await open(lockPath, "wx");
await handle.close();
// Exercise the former 30-second reclaim threshold without a slow test.
const oldTime = new Date(Date.now() - 60_000);
await utimes(lockPath, oldTime, oldTime);
await writeFile(readyPath, "ready", "utf8");
await waitFor(releasePath);
await unlink(lockPath).catch(() => undefined);
}
main();