mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(openclaw): declare transcriptSemantics and implement commitTurn (#3442)
## Description Adds OpenClaw transcript-semantics declarations and an accepted-turn log for admitted-turn advancement. Fixes #3441. Messages and advancement keys are written together in `HEADROOM_WORKSPACE_DIR/openclaw/commit-log.json` (default workspace: `~/.headroom`). Each commit re-reads the log inside an exclusive sibling-file lock and replaces the log atomically. Keys are retained without eviction; prior key-only logs are recognized for migration. Failed writes do not create an in-memory phantom duplicate. ## Locking and recovery The lock serializes processes sharing the workspace and is never reclaimed merely because it is old. A waiting caller times out after 60 seconds instead of committing without ownership. Release checks the acquisition token. A crashed holder can leave an orphan `.lock` file. Recovery is operational: stop writers using that workspace, establish that the holder is no longer running, remove the orphan lock, then resume. Do not delete a live writer's lock. Automatic crash recovery and power-loss durability are not claimed. ## Testing - 99 plugin tests pass, including distinct-key and same-key child-process commits and an aged live-lock regression. - TypeScript checking passes. - The aged-lock regression fails against the prior age-reclaim implementation and passes against the current source. Test cleanup releases the child even on assertion failure. - Current-head CI is pending. ## Real Behavior Proof - Environment: macOS, two real Node child processes importing `DurableAdvancementKeyStore`. - Exact command / steps: A acquires the lock and reads the empty log; pause A while alive for 31 seconds. Start B committing another key and verify it remains pending, then resume A. - Observed result: B does not finish while A holds the lock. After release, both return `committed` and both message payloads remain in the log. The prior implementation acknowledged both but lost one turn. - Not tested: automatic recovery from crashed holders (requires the operator procedure above); power-loss durability. ## Runtime Rollout Safety - Rollout-managed feature(s): OpenClaw transcript-semantics declaration. - Minimum rollout channel: Host capability negotiation. - Stable/default behavior changed: Accepted messages and keys persist in a workspace-shared log. The log grows without eviction, and an orphan lock requires operator recovery. - Kill switch / disable path: Revert the plugin change or use the host's legacy context-engine path. - Unsafe override required: No. - Qualification impact: Admission uses atomic/idempotent commit behavior; contention fails closed. - Rollback path: Revert the plugin change. Preserve the log if retry history must remain available. ## Review Readiness - [x] Prior message persistence, retry, cross-process, and stale-lock findings resolved. - [x] Local tests and type checking pass. - [ ] Current-head CI complete. - Final human review is required. --------- Signed-off-by: Radhakrishnan P <gingeekrishna@gmail.com> Co-authored-by: Jerrett Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Jerrett Davis
parent
a102e05b51
commit
a28dd4eab7
@@ -0,0 +1,240 @@
|
||||
/**
|
||||
* Durable, atomic, restart-safe, cross-process-safe record of committed
|
||||
* turn advancements.
|
||||
*
|
||||
* OpenClaw's `atomic-idempotent-v1` transcriptSemantics contract requires
|
||||
* that a retried `commitTurn` call report "duplicate" for a key that was
|
||||
* already committed -- including after this process (or the host) restarts
|
||||
* -- and that "the accepted turn and key advance together" (the committed
|
||||
* messages are persisted atomically with the key, not discarded).
|
||||
*
|
||||
* The default commit-log path is shared by every gateway process using the
|
||||
* same Headroom workspace, not just every store instance in one process --
|
||||
* an in-process mutex (a JS `Map`/promise queue) does nothing to stop two
|
||||
* separate OS processes from each reading the same file, adding a different
|
||||
* key, and racing their renames, with the loser's accepted turn silently
|
||||
* discarded. The read/check/write transaction is therefore protected by a
|
||||
* real inter-process lock: an exclusive lock FILE (`<path>.lock`), acquired
|
||||
* via atomic exclusive creation (`open(..., "wx")`, which is atomic at the
|
||||
* OS level on POSIX and Windows alike) so only one process -- in this one or
|
||||
* any other -- can be inside the critical section at a time.
|
||||
*
|
||||
* The lock is never force-broken by age. An age-based "the holder must have
|
||||
* crashed" heuristic cannot tell a dead holder apart from one that is simply
|
||||
* slow (a large write, a GC pause, disk contention): a second process that
|
||||
* reclaims a lock the first still holds can commit and release before the
|
||||
* first resumes and overwrites that commit with its own now-stale snapshot
|
||||
* -- both processes report "committed", but only the second process's write
|
||||
* survives, and the first process's genuinely-accepted turn silently
|
||||
* vanishes. Node has no built-in binding for a real OS-mediated lock
|
||||
* (POSIX `flock`/Windows `LockFileEx`, where the kernel itself releases the
|
||||
* lock when the holder's process exits, crash included, so no staleness
|
||||
* guess is needed at all), so this fails closed instead: a lock that is
|
||||
* never released is never reclaimed, and a caller that can't acquire it
|
||||
* within the timeout gets a clear error rather than a silent lost write. A
|
||||
* lock file orphaned by a genuinely crashed process requires a human (or an
|
||||
* operator script) to remove `<path>.lock` before commits can resume --
|
||||
* that operational cost is the trade for never silently dropping an
|
||||
* acknowledged commit. Lock ownership is also verified before release (see
|
||||
* `withFileLock`): a process only ever deletes the lock file it created.
|
||||
*
|
||||
* Every `tryCommit` re-reads the file fresh from disk inside the lock (no
|
||||
* long-lived in-memory cache), and the in-memory return value is only ever
|
||||
* "committed" once the atomic (temp file + rename) write recording the key
|
||||
* has actually succeeded -- a failed write is never remembered as seen, so
|
||||
* retrying the same key attempts the commit again instead of returning a
|
||||
* phantom "duplicate" for a commit that never reached disk.
|
||||
*
|
||||
* Keys are never evicted. OpenClaw may retry at any point after a restart or
|
||||
* a long stall, so pruning by count or age would risk re-accepting an
|
||||
* already-committed turn -- exactly the failure this store exists to avoid.
|
||||
*/
|
||||
|
||||
import { mkdir, open, readFile, rename, stat, unlink, writeFile } from "node:fs/promises";
|
||||
import { dirname, join } from "node:path";
|
||||
import * as os from "node:os";
|
||||
|
||||
/**
|
||||
* True if `p` exists and is something other than a directory (e.g. a file
|
||||
* sitting where a directory needs to be). Some platforms (observed on
|
||||
* Windows) report `ENOENT` -- not a distinct "not a directory" code -- for
|
||||
* `readFile` when a path component like this blocks the rest of the path,
|
||||
* which is indistinguishable from "the commit log just doesn't exist yet"
|
||||
* by error code alone. Checking the parent directly is what tells the two
|
||||
* apart: silently treating a broken storage path as an empty store would
|
||||
* make every future commit re-accept an already-committed key.
|
||||
*/
|
||||
async function isBlockingNonDirectory(p: string): Promise<boolean> {
|
||||
try {
|
||||
const info = await stat(p);
|
||||
return !info.isDirectory();
|
||||
} catch {
|
||||
return false; // Doesn't exist -- not blocking anything.
|
||||
}
|
||||
}
|
||||
|
||||
/** Default commit-log location, honoring the same workspace override the
|
||||
* rest of the Headroom workspace convention uses (`HEADROOM_WORKSPACE_DIR`). */
|
||||
export function defaultCommitLogPath(): string {
|
||||
const workspaceDir = process.env.HEADROOM_WORKSPACE_DIR?.trim() || join(os.homedir(), ".headroom");
|
||||
return join(workspaceDir, "openclaw", "commit-log.json");
|
||||
}
|
||||
|
||||
/** One durably-committed turn: the accepted messages plus when they landed. */
|
||||
export interface CommittedTurn {
|
||||
messages: unknown;
|
||||
committedAt: string;
|
||||
}
|
||||
|
||||
type CommitLog = Record<string, CommittedTurn>;
|
||||
|
||||
/** Give up (rather than poll forever) if the lock still can't be acquired after this long. */
|
||||
const LOCK_ACQUIRE_TIMEOUT_MS = 60_000;
|
||||
const LOCK_POLL_BASE_MS = 20;
|
||||
|
||||
function sleep(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
/**
|
||||
* Run `fn` while holding an exclusive lock on `path` (a sibling `<path>.lock`
|
||||
* file). Serializes every caller -- this process or another -- against the
|
||||
* same file, not just calls on one object or one process.
|
||||
*
|
||||
* The lock is held until `fn` returns, however long that takes, and is
|
||||
* never reclaimed by another caller on a staleness guess -- see the module
|
||||
* docstring for why an age-based heuristic can silently drop a genuinely
|
||||
* accepted commit. A caller that can't acquire the lock within
|
||||
* `LOCK_ACQUIRE_TIMEOUT_MS` gets a clear timeout error instead of the
|
||||
* transaction being allowed to proceed unsafely.
|
||||
*/
|
||||
async function withFileLock<T>(path: string, fn: () => Promise<T>): Promise<T> {
|
||||
const lockPath = `${path}.lock`;
|
||||
await mkdir(dirname(path), { recursive: true });
|
||||
// Unique per acquisition, so release only ever removes the lock file this
|
||||
// exact call created -- never one a different (later) holder created.
|
||||
const token = `${process.pid}-${Date.now()}-${Math.random().toString(36).slice(2)}`;
|
||||
const deadline = Date.now() + LOCK_ACQUIRE_TIMEOUT_MS;
|
||||
let attempt = 0;
|
||||
for (;;) {
|
||||
try {
|
||||
const handle = await open(lockPath, "wx");
|
||||
try {
|
||||
await handle.writeFile(`${token}\n${process.pid}\n${new Date().toISOString()}\n`);
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
break;
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException)?.code !== "EEXIST") {
|
||||
throw error;
|
||||
}
|
||||
if (Date.now() > deadline) {
|
||||
throw new Error(
|
||||
`Timed out waiting for the advancement-key-store lock on ${path}. If the ` +
|
||||
`process that created ${lockPath} is no longer running, remove it manually.`,
|
||||
);
|
||||
}
|
||||
attempt += 1;
|
||||
await sleep(LOCK_POLL_BASE_MS + Math.random() * LOCK_POLL_BASE_MS * Math.min(attempt, 10));
|
||||
}
|
||||
}
|
||||
try {
|
||||
return await fn();
|
||||
} finally {
|
||||
// Verify the lock file is still the one this call created before
|
||||
// removing it -- if it doesn't start with our token, some other process
|
||||
// (or a human) replaced or is using it, and it must not be touched.
|
||||
try {
|
||||
const current = await readFile(lockPath, "utf8");
|
||||
if (current.startsWith(`${token}\n`)) {
|
||||
await unlink(lockPath);
|
||||
}
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException)?.code !== "ENOENT") {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export class DurableAdvancementKeyStore {
|
||||
constructor(private readonly path: string) {}
|
||||
|
||||
private async readAll(): Promise<CommitLog> {
|
||||
let raw: string;
|
||||
try {
|
||||
raw = await readFile(this.path, "utf8");
|
||||
} catch (error) {
|
||||
const code = (error as NodeJS.ErrnoException)?.code;
|
||||
if (code !== "ENOENT" && code !== "ENOTDIR") {
|
||||
throw error;
|
||||
}
|
||||
if (await isBlockingNonDirectory(dirname(this.path))) {
|
||||
throw error;
|
||||
}
|
||||
return {};
|
||||
}
|
||||
const parsed = JSON.parse(raw) as unknown;
|
||||
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
|
||||
return parsed as CommitLog;
|
||||
}
|
||||
if (Array.isArray(parsed)) {
|
||||
// Migrate the earlier key-only array format (no message payload) so
|
||||
// upgrading from it never re-accepts an already-committed key.
|
||||
const migrated: CommitLog = {};
|
||||
for (const entry of parsed) {
|
||||
if (typeof entry === "string") {
|
||||
migrated[entry] = { messages: null, committedAt: "" };
|
||||
}
|
||||
}
|
||||
return migrated;
|
||||
}
|
||||
return {};
|
||||
}
|
||||
|
||||
private async writeAll(entries: CommitLog): Promise<void> {
|
||||
await mkdir(dirname(this.path), { recursive: true });
|
||||
const tmpPath = `${this.path}.${process.pid}.${Math.random().toString(36).slice(2)}.tmp`;
|
||||
await writeFile(tmpPath, JSON.stringify(entries), "utf8");
|
||||
await rename(tmpPath, this.path);
|
||||
}
|
||||
|
||||
/**
|
||||
* Atomically check-and-record `key` together with the `messages` it
|
||||
* advances. Holds the cross-process file lock (see `withFileLock`) for
|
||||
* the full read/check/write transaction, so two processes -- or two
|
||||
* instances in this one -- can't each read the same starting state and
|
||||
* race their writes.
|
||||
*
|
||||
* Throws if the write fails -- the caller sees a rejected commit, not a
|
||||
* false "committed"/"duplicate", and a retry re-reads the (still
|
||||
* unchanged) on-disk state and attempts the write again.
|
||||
*/
|
||||
async tryCommit(key: string, messages: unknown): Promise<"committed" | "duplicate"> {
|
||||
return withFileLock(this.path, async () => {
|
||||
const entries = await this.readAll();
|
||||
if (key in entries) {
|
||||
return "duplicate";
|
||||
}
|
||||
const next: CommitLog = {
|
||||
...entries,
|
||||
[key]: { messages, committedAt: new Date().toISOString() },
|
||||
};
|
||||
await this.writeAll(next);
|
||||
return "committed";
|
||||
});
|
||||
}
|
||||
|
||||
/** Test-only: check membership without recording. */
|
||||
async has(key: string): Promise<boolean> {
|
||||
const entries = await this.readAll();
|
||||
return key in entries;
|
||||
}
|
||||
|
||||
/** Test-only: read back a committed entry (messages + commit time) without recording. */
|
||||
async get(key: string): Promise<CommittedTurn | undefined> {
|
||||
const entries = await this.readAll();
|
||||
return entries[key];
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,7 @@
|
||||
import { compress } from "headroom-ai";
|
||||
import { ProxyManager, defaultLogger, type ProxyManagerConfig, type ProxyManagerLogger } from "./proxy-manager.js";
|
||||
import { agentToOpenAI, normalizeAgentMessages, openAIToAgent } from "./convert.js";
|
||||
import { DurableAdvancementKeyStore, defaultCommitLogPath } from "./advancement-key-store.js";
|
||||
import {
|
||||
delegateCompactionToRuntime,
|
||||
type OpenClawCompactParams,
|
||||
@@ -32,6 +33,9 @@ export interface HeadroomEngineConfig extends ProxyManagerConfig {
|
||||
requestTimeoutMs?: number;
|
||||
circuitBreakerThreshold?: number;
|
||||
circuitBreakerCooldownMs?: number;
|
||||
/** Where to durably record committed turn-advancement keys (see
|
||||
* `DurableAdvancementKeyStore`). Defaults to `defaultCommitLogPath()`. */
|
||||
commitLogPath?: string;
|
||||
}
|
||||
|
||||
export class HeadroomContextEngine {
|
||||
@@ -40,8 +44,17 @@ export class HeadroomContextEngine {
|
||||
name: "Headroom Context Compression",
|
||||
version: "0.1.0",
|
||||
ownsCompaction: false,
|
||||
transcriptSemantics: {
|
||||
currentTurnFence: "before-current-turn-entry-v1",
|
||||
turnAdvancementIdempotency: "atomic-idempotent-v1",
|
||||
},
|
||||
};
|
||||
|
||||
// Durable, restart-safe record of committed advancement keys, for
|
||||
// commitTurn's idempotent-retry check. See advancement-key-store.ts for
|
||||
// why this must survive a process restart and must not evict entries.
|
||||
private advancementKeyStore: DurableAdvancementKeyStore;
|
||||
|
||||
private proxyManager: ProxyManager;
|
||||
private proxyUrl: string | null = null;
|
||||
private config: HeadroomEngineConfig;
|
||||
@@ -61,6 +74,9 @@ export class HeadroomContextEngine {
|
||||
this.config = config;
|
||||
this.logger = logger ?? defaultLogger;
|
||||
this.proxyManager = new ProxyManager(config, this.logger);
|
||||
this.advancementKeyStore = new DurableAdvancementKeyStore(
|
||||
config.commitLogPath ?? defaultCommitLogPath(),
|
||||
);
|
||||
}
|
||||
|
||||
// === ContextEngine Lifecycle ===
|
||||
@@ -189,6 +205,23 @@ export class HeadroomContextEngine {
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Durable turn-advancement commit — required by OpenClaw's transcriptSemantics
|
||||
* contract. Called only for the accepted, successful turn; failed or aborted
|
||||
* turns never reach here. Must be an atomic, idempotent write of the
|
||||
* accepted `messages` keyed by `advancementKey` so a host retry with the
|
||||
* same key reports "duplicate" instead of re-applying the advancement —
|
||||
* including a retry that arrives after this process restarted, which is
|
||||
* why the record lives on disk (see `DurableAdvancementKeyStore`) rather
|
||||
* than in memory, and includes the messages rather than just the key.
|
||||
*/
|
||||
async commitTurn(params: { advancementKey: string; messages: any[] }): Promise<{
|
||||
status: "committed" | "duplicate";
|
||||
}> {
|
||||
const status = await this.advancementKeyStore.tryCommit(params.advancementKey, params.messages);
|
||||
return { status };
|
||||
}
|
||||
|
||||
async afterTurn?(params: {
|
||||
sessionId: string;
|
||||
messages: any[];
|
||||
|
||||
@@ -0,0 +1,350 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import * as fs from "node:fs/promises";
|
||||
import * as os from "node:os";
|
||||
import * as path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { promisify } from "node:util";
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
|
||||
import { DurableAdvancementKeyStore, defaultCommitLogPath } from "../src/advancement-key-store.js";
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
|
||||
// Node's native TypeScript support (needed to run the worker fixture, which
|
||||
// imports the real DurableAdvancementKeyStore, as a plain `node` child
|
||||
// process with no build step) requires `--experimental-transform-types`,
|
||||
// added in Node 22.6. Skip the cross-process tests rather than fail on an
|
||||
// older Node -- the failure mode being tested (a lost write across two OS
|
||||
// processes) is orthogonal to which Node version happens to run the test.
|
||||
const NODE_SUPPORTS_TRANSFORM_TYPES = (() => {
|
||||
const [major, minor] = process.versions.node.split(".").map(Number);
|
||||
return major > 22 || (major === 22 && minor >= 6);
|
||||
})();
|
||||
|
||||
const WORKER_PATH = fileURLToPath(new URL("./fixtures/commit-turn-worker.ts", import.meta.url));
|
||||
const HOLD_LOCK_WORKER_PATH = fileURLToPath(
|
||||
new URL("./fixtures/hold-lock-worker.ts", import.meta.url),
|
||||
);
|
||||
|
||||
/** Run the commit-turn worker as a real child process; returns its parsed JSON result. */
|
||||
async function runCommitWorker(
|
||||
storePath: string,
|
||||
key: string,
|
||||
goPath: string,
|
||||
resultPath: string,
|
||||
): Promise<{ status?: string; error?: string }> {
|
||||
await execFileAsync(process.execPath, [
|
||||
"--experimental-transform-types",
|
||||
WORKER_PATH,
|
||||
storePath,
|
||||
key,
|
||||
goPath,
|
||||
resultPath,
|
||||
]);
|
||||
return JSON.parse(await fs.readFile(resultPath, "utf8"));
|
||||
}
|
||||
|
||||
async function waitForFile(p: string): Promise<void> {
|
||||
for (;;) {
|
||||
try {
|
||||
await fs.access(p);
|
||||
return;
|
||||
} catch {
|
||||
await new Promise((r) => setTimeout(r, 5));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
describe("DurableAdvancementKeyStore", () => {
|
||||
let dir: string;
|
||||
let storePath: string;
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = await fs.mkdtemp(path.join(os.tmpdir(), "headroom-advancement-key-store-"));
|
||||
storePath = path.join(dir, "nested", "commit-log.json");
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await fs.rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("commits a new key and creates the parent directory", async () => {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-1", [{ role: "user", content: "hi" }])).resolves.toBe(
|
||||
"committed",
|
||||
);
|
||||
await expect(fs.stat(storePath)).resolves.toBeTruthy();
|
||||
});
|
||||
|
||||
it("reports duplicate for an already-committed key", async () => {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-1", [])).resolves.toBe("committed");
|
||||
await expect(store.tryCommit("turn-1", [])).resolves.toBe("duplicate");
|
||||
});
|
||||
|
||||
it("persists across a simulated restart (a fresh store instance, same path)", async () => {
|
||||
const before = new DurableAdvancementKeyStore(storePath);
|
||||
await expect(before.tryCommit("turn-1", [])).resolves.toBe("committed");
|
||||
|
||||
const after = new DurableAdvancementKeyStore(storePath);
|
||||
await expect(after.tryCommit("turn-1", [])).resolves.toBe("duplicate");
|
||||
// A key that was never committed is still accepted normally.
|
||||
await expect(after.tryCommit("turn-2", [])).resolves.toBe("committed");
|
||||
});
|
||||
|
||||
it("treats a missing commit-log file as an empty store rather than an error", async () => {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.has("turn-1")).resolves.toBe(false);
|
||||
await expect(store.tryCommit("turn-1", [])).resolves.toBe("committed");
|
||||
});
|
||||
|
||||
it("fails loudly on a corrupt commit-log file instead of silently forgetting its contents", async () => {
|
||||
await fs.mkdir(path.dirname(storePath), { recursive: true });
|
||||
await fs.writeFile(storePath, "{not valid json", "utf8");
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
await expect(store.tryCommit("turn-1", [])).rejects.toThrow();
|
||||
});
|
||||
|
||||
it("serializes concurrent commits of the same key so exactly one wins", async () => {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
const results = await Promise.all([
|
||||
store.tryCommit("turn-race", []),
|
||||
store.tryCommit("turn-race", []),
|
||||
store.tryCommit("turn-race", []),
|
||||
]);
|
||||
|
||||
expect(results.filter((r) => r === "committed")).toHaveLength(1);
|
||||
expect(results.filter((r) => r === "duplicate")).toHaveLength(2);
|
||||
});
|
||||
|
||||
it(
|
||||
"never evicts a key regardless of how many others were committed since",
|
||||
async () => {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
for (let i = 0; i < 600; i++) {
|
||||
await store.tryCommit(`turn-${i}`, []);
|
||||
}
|
||||
|
||||
await expect(store.tryCommit("turn-0", [])).resolves.toBe("duplicate");
|
||||
},
|
||||
20_000,
|
||||
);
|
||||
|
||||
// ---------------------------------------------------------------------
|
||||
// PR #3442 review, round 2: three additional atomic-commit failures.
|
||||
// ---------------------------------------------------------------------
|
||||
|
||||
it("persists the accepted messages together with the key, not just the key", async () => {
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
const messages = [
|
||||
{ role: "user", content: "what's the weather?" },
|
||||
{ role: "assistant", content: "Let me check." },
|
||||
];
|
||||
|
||||
await store.tryCommit("turn-1", messages);
|
||||
|
||||
const entry = await store.get("turn-1");
|
||||
expect(entry).toBeDefined();
|
||||
expect(entry?.messages).toEqual(messages);
|
||||
expect(typeof entry?.committedAt).toBe("string");
|
||||
expect(entry?.committedAt.length).toBeGreaterThan(0);
|
||||
|
||||
// Survives a restart too -- the messages aren't an in-memory-only echo.
|
||||
const after = new DurableAdvancementKeyStore(storePath);
|
||||
const reloaded = await after.get("turn-1");
|
||||
expect(reloaded?.messages).toEqual(messages);
|
||||
});
|
||||
|
||||
it("does not remember a key when storage is inaccessible, so a retry can still succeed", async () => {
|
||||
// Block the parent directory with a file. Both the commit's initial
|
||||
// read and a membership lookup must propagate the filesystem error.
|
||||
const blockedParent = path.join(dir, "blocked");
|
||||
await fs.writeFile(blockedParent, "not a directory", "utf8");
|
||||
const blockedPath = path.join(blockedParent, "commit-log.json");
|
||||
const store = new DurableAdvancementKeyStore(blockedPath);
|
||||
|
||||
await expect(store.tryCommit("turn-1", [])).rejects.toThrow();
|
||||
// The failed write must not be remembered in memory: retrying the same
|
||||
// key must attempt the commit again (and fail again, for the same
|
||||
// reason) rather than silently reporting "duplicate" for a commit that
|
||||
// never reached disk.
|
||||
await expect(store.tryCommit("turn-1", [])).rejects.toThrow();
|
||||
await expect(store.has("turn-1")).rejects.toThrow();
|
||||
|
||||
// Once the underlying problem is fixed, the key is still absent and
|
||||
// the same key commits normally.
|
||||
await fs.rm(blockedParent, { force: true });
|
||||
await expect(store.has("turn-1")).resolves.toBe(false);
|
||||
await expect(store.tryCommit("turn-1", [])).resolves.toBe("committed");
|
||||
await expect(store.has("turn-1")).resolves.toBe(true);
|
||||
});
|
||||
|
||||
it("serializes across store instances sharing the same file, not only within one object", async () => {
|
||||
const storeA = new DurableAdvancementKeyStore(storePath);
|
||||
const storeB = new DurableAdvancementKeyStore(storePath);
|
||||
|
||||
// Interleaved commits through two independent instances against the
|
||||
// same file must not lose either one to a read-modify-write race.
|
||||
const [resultA, resultB] = await Promise.all([
|
||||
storeA.tryCommit("turn-a", ["from a"]),
|
||||
storeB.tryCommit("turn-b", ["from b"]),
|
||||
]);
|
||||
|
||||
expect(resultA).toBe("committed");
|
||||
expect(resultB).toBe("committed");
|
||||
|
||||
// A third, fresh instance (simulating a restart) must see both.
|
||||
const after = new DurableAdvancementKeyStore(storePath);
|
||||
await expect(after.has("turn-a")).resolves.toBe(true);
|
||||
await expect(after.has("turn-b")).resolves.toBe(true);
|
||||
await expect(after.tryCommit("turn-a", [])).resolves.toBe("duplicate");
|
||||
await expect(after.tryCommit("turn-b", [])).resolves.toBe("duplicate");
|
||||
});
|
||||
|
||||
// -------------------------------------------------------------------
|
||||
// PR #3442 review, round 3: an in-process lock (a JS Map/promise queue)
|
||||
// does nothing to stop two separate OS processes -- e.g. two gateway
|
||||
// processes sharing one Headroom workspace -- from racing the same
|
||||
// commit-log file. These spawn the real DurableAdvancementKeyStore in
|
||||
// real child processes (not Worker threads, which share the parent's
|
||||
// memory and wouldn't exercise inter-process file locking) and release
|
||||
// them via a shared "go" file so their read/check/write transactions
|
||||
// contend for the file lock as close to simultaneously as possible.
|
||||
// -------------------------------------------------------------------
|
||||
|
||||
it.skipIf(!NODE_SUPPORTS_TRANSFORM_TYPES)(
|
||||
"serializes commits from two separate OS processes sharing the same file",
|
||||
async () => {
|
||||
const goPath = path.join(dir, "go");
|
||||
const resultAPath = path.join(dir, "result-a.json");
|
||||
const resultBPath = path.join(dir, "result-b.json");
|
||||
|
||||
const workerA = runCommitWorker(storePath, "turn-a", goPath, resultAPath);
|
||||
const workerB = runCommitWorker(storePath, "turn-b", goPath, resultBPath);
|
||||
// Give both worker processes time to spawn and reach their poll loop
|
||||
// before releasing them together.
|
||||
await new Promise((r) => setTimeout(r, 300));
|
||||
await fs.writeFile(goPath, "go", "utf8");
|
||||
|
||||
const [resultA, resultB] = await Promise.all([workerA, workerB]);
|
||||
|
||||
expect(resultA).toEqual({ status: "committed" });
|
||||
expect(resultB).toEqual({ status: "committed" });
|
||||
|
||||
// Neither commit was lost to a cross-process race.
|
||||
const after = new DurableAdvancementKeyStore(storePath);
|
||||
await expect(after.has("turn-a")).resolves.toBe(true);
|
||||
await expect(after.has("turn-b")).resolves.toBe(true);
|
||||
},
|
||||
30_000,
|
||||
);
|
||||
|
||||
it.skipIf(!NODE_SUPPORTS_TRANSFORM_TYPES)(
|
||||
"reports duplicate for the same key committed from two separate OS processes",
|
||||
async () => {
|
||||
const goPath = path.join(dir, "go");
|
||||
const resultAPath = path.join(dir, "result-a.json");
|
||||
const resultBPath = path.join(dir, "result-b.json");
|
||||
|
||||
const workerA = runCommitWorker(storePath, "turn-same", goPath, resultAPath);
|
||||
const workerB = runCommitWorker(storePath, "turn-same", goPath, resultBPath);
|
||||
await new Promise((r) => setTimeout(r, 300));
|
||||
await fs.writeFile(goPath, "go", "utf8");
|
||||
|
||||
const [resultA, resultB] = await Promise.all([workerA, workerB]);
|
||||
|
||||
// Exactly one process's transaction should win the race and see
|
||||
// "committed"; the other must see "duplicate", never both
|
||||
// "committed" (which would mean the file lock let them interleave).
|
||||
const statuses = [resultA.status, resultB.status].sort();
|
||||
expect(statuses).toEqual(["committed", "duplicate"]);
|
||||
},
|
||||
30_000,
|
||||
);
|
||||
|
||||
it.skipIf(!NODE_SUPPORTS_TRANSFORM_TYPES)(
|
||||
"never lets a second writer commit while a live holder still holds the lock",
|
||||
async () => {
|
||||
// Regression for PR #3442 review, round 4: a lock reclaimed on an
|
||||
// age-based staleness guess let a second process commit and release
|
||||
// while the first (merely slow, not dead) holder still believed it
|
||||
// owned the lock -- the first then resumed and overwrote the
|
||||
// second's accepted commit with its own stale snapshot. This proves
|
||||
// the invariant that replaces that heuristic: a second writer must
|
||||
// never proceed while ANY holder -- however long it takes -- still
|
||||
// holds the lock. Deterministic and fast: the "holder" is a worker
|
||||
// that occupies the real lock file and only releases it when told
|
||||
// to, rather than relying on a real multi-second sleep.
|
||||
const readyPath = path.join(dir, "holder-ready");
|
||||
const releasePath = path.join(dir, "holder-release");
|
||||
|
||||
const holder = execFileAsync(process.execPath, [
|
||||
"--experimental-transform-types",
|
||||
HOLD_LOCK_WORKER_PATH,
|
||||
storePath,
|
||||
readyPath,
|
||||
releasePath,
|
||||
]);
|
||||
|
||||
// Wait until the holder has actually acquired the lock before racing
|
||||
// a commit against it.
|
||||
await waitForFile(readyPath);
|
||||
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
let secondCommitSettled = false;
|
||||
const secondCommit = store.tryCommit("turn-b", ["b"]).finally(() => {
|
||||
secondCommitSettled = true;
|
||||
});
|
||||
|
||||
// Give the second commit ample opportunity to (wrongly) proceed
|
||||
// while the holder is still alive and has not released.
|
||||
try {
|
||||
await new Promise((r) => setTimeout(r, 500));
|
||||
expect(secondCommitSettled).toBe(false);
|
||||
} finally {
|
||||
// Release even if the assertion fails, so the child cannot leak.
|
||||
await fs.writeFile(releasePath, "go", "utf8");
|
||||
await holder;
|
||||
}
|
||||
// The second commit must now complete successfully, with nothing lost.
|
||||
await expect(secondCommit).resolves.toBe("committed");
|
||||
await expect(store.has("turn-b")).resolves.toBe(true);
|
||||
},
|
||||
15_000,
|
||||
);
|
||||
});
|
||||
|
||||
describe("defaultCommitLogPath", () => {
|
||||
const originalWorkspaceDir = process.env.HEADROOM_WORKSPACE_DIR;
|
||||
|
||||
afterEach(() => {
|
||||
if (originalWorkspaceDir === undefined) {
|
||||
delete process.env.HEADROOM_WORKSPACE_DIR;
|
||||
} else {
|
||||
process.env.HEADROOM_WORKSPACE_DIR = originalWorkspaceDir;
|
||||
}
|
||||
});
|
||||
|
||||
it("honors HEADROOM_WORKSPACE_DIR when set", () => {
|
||||
process.env.HEADROOM_WORKSPACE_DIR = "/custom/workspace";
|
||||
|
||||
expect(defaultCommitLogPath()).toBe(
|
||||
path.join("/custom/workspace", "openclaw", "commit-log.json"),
|
||||
);
|
||||
});
|
||||
|
||||
it("falls back to ~/.headroom when unset", () => {
|
||||
delete process.env.HEADROOM_WORKSPACE_DIR;
|
||||
|
||||
expect(defaultCommitLogPath()).toBe(
|
||||
path.join(os.homedir(), ".headroom", "openclaw", "commit-log.json"),
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -1,4 +1,8 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import * as fs from "node:fs/promises";
|
||||
import * as os from "node:os";
|
||||
import * as path from "node:path";
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const mocked = vi.hoisted(() => ({
|
||||
compress: vi.fn(),
|
||||
@@ -325,3 +329,102 @@ describe("HeadroomContextEngine proxy startup helpers", () => {
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("HeadroomContextEngine transcriptSemantics contract", () => {
|
||||
let commitLogDir: string;
|
||||
let commitLogPath: string;
|
||||
|
||||
beforeEach(async () => {
|
||||
commitLogDir = await fs.mkdtemp(path.join(os.tmpdir(), "headroom-commit-log-"));
|
||||
commitLogPath = path.join(commitLogDir, "commit-log.json");
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await fs.rm(commitLogDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("declares the durable-commit transcript semantics OpenClaw requires", () => {
|
||||
const engine = new HeadroomContextEngine({ commitLogPath });
|
||||
|
||||
expect(engine.info.transcriptSemantics).toEqual({
|
||||
currentTurnFence: "before-current-turn-entry-v1",
|
||||
turnAdvancementIdempotency: "atomic-idempotent-v1",
|
||||
});
|
||||
});
|
||||
|
||||
it("commits a new advancement key", async () => {
|
||||
const engine = new HeadroomContextEngine({ commitLogPath });
|
||||
|
||||
await expect(
|
||||
engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
|
||||
).resolves.toEqual({ status: "committed" });
|
||||
});
|
||||
|
||||
it("reports duplicate on a retried advancement key", async () => {
|
||||
const engine = new HeadroomContextEngine({ commitLogPath });
|
||||
|
||||
await expect(
|
||||
engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
|
||||
).resolves.toEqual({ status: "committed" });
|
||||
await expect(
|
||||
engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
|
||||
).resolves.toEqual({ status: "duplicate" });
|
||||
});
|
||||
|
||||
it("treats distinct advancement keys independently", async () => {
|
||||
const engine = new HeadroomContextEngine({ commitLogPath });
|
||||
|
||||
await expect(
|
||||
engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
|
||||
).resolves.toEqual({ status: "committed" });
|
||||
await expect(
|
||||
engine.commitTurn({ advancementKey: "turn-2", messages: [] }),
|
||||
).resolves.toEqual({ status: "committed" });
|
||||
});
|
||||
|
||||
it("reports duplicate for a key committed before a process restart", async () => {
|
||||
// Simulate a restart: a brand new engine instance (no shared in-memory
|
||||
// state) pointed at the same durable commit-log path.
|
||||
const before = new HeadroomContextEngine({ commitLogPath });
|
||||
await expect(
|
||||
before.commitTurn({ advancementKey: "turn-restart", messages: [] }),
|
||||
).resolves.toEqual({ status: "committed" });
|
||||
|
||||
const after = new HeadroomContextEngine({ commitLogPath });
|
||||
await expect(
|
||||
after.commitTurn({ advancementKey: "turn-restart", messages: [] }),
|
||||
).resolves.toEqual({ status: "duplicate" });
|
||||
});
|
||||
|
||||
it(
|
||||
"never forgets a key regardless of how many other keys were committed since",
|
||||
async () => {
|
||||
// Regression: the old implementation evicted the oldest key past a
|
||||
// 512-entry cap, so a retry of an early key was wrongly re-accepted as
|
||||
// new instead of reported as a duplicate. There is no such cap now.
|
||||
const engine = new HeadroomContextEngine({ commitLogPath });
|
||||
|
||||
for (let i = 0; i < 600; i++) {
|
||||
await engine.commitTurn({ advancementKey: `turn-${i}`, messages: [] });
|
||||
}
|
||||
|
||||
await expect(
|
||||
engine.commitTurn({ advancementKey: "turn-0", messages: [] }),
|
||||
).resolves.toEqual({ status: "duplicate" });
|
||||
},
|
||||
20_000,
|
||||
);
|
||||
|
||||
it("persists the accepted messages together with the advancement key", async () => {
|
||||
const engine = new HeadroomContextEngine({ commitLogPath });
|
||||
const messages = [{ role: "user", content: "hello" }];
|
||||
|
||||
await expect(
|
||||
engine.commitTurn({ advancementKey: "turn-1", messages }),
|
||||
).resolves.toEqual({ status: "committed" });
|
||||
|
||||
const raw = await fs.readFile(commitLogPath, "utf8");
|
||||
const entries = JSON.parse(raw) as Record<string, { messages: unknown }>;
|
||||
expect(entries["turn-1"].messages).toEqual(messages);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
/**
|
||||
* Child-process worker for the cross-process DurableAdvancementKeyStore
|
||||
* regression tests (PR #3442 review, round 3).
|
||||
*
|
||||
* Run as a real, separate OS process (not a Worker thread, which shares the
|
||||
* parent's memory and wouldn't exercise inter-process file locking) via
|
||||
* `node --experimental-transform-types commit-turn-worker.ts <storePath>
|
||||
* <key> <goPath> <resultPath>`.
|
||||
*
|
||||
* Blocks until `goPath` exists (the parent test creates it only after every
|
||||
* worker has been spawned and had time to reach this poll loop), then
|
||||
* commits `key` through the real `DurableAdvancementKeyStore` and writes
|
||||
* the outcome to `resultPath` as JSON -- this is what lets two independently
|
||||
* launched processes race their read/check/write transactions against the
|
||||
* same commit-log file as closely to simultaneously as possible.
|
||||
*/
|
||||
|
||||
import { access, writeFile } from "node:fs/promises";
|
||||
|
||||
// Explicit `.ts` extension: this file runs directly under Node's native
|
||||
// TypeScript support (`--experimental-transform-types`), which -- unlike
|
||||
// the TypeScript compiler / bundler used to build the plugin itself --
|
||||
// resolves imports literally and does not map a `.js` specifier onto a
|
||||
// sibling `.ts` file.
|
||||
import { DurableAdvancementKeyStore } from "../../src/advancement-key-store.ts";
|
||||
|
||||
async function waitForGo(goPath: string): Promise<void> {
|
||||
for (;;) {
|
||||
try {
|
||||
await access(goPath);
|
||||
return;
|
||||
} catch {
|
||||
await new Promise((resolveDelay) => setTimeout(resolveDelay, 5));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const [, , storePath, key, goPath, resultPath] = process.argv;
|
||||
if (!storePath || !key || !goPath || !resultPath) {
|
||||
throw new Error("usage: commit-turn-worker.ts <storePath> <key> <goPath> <resultPath>");
|
||||
}
|
||||
|
||||
await waitForGo(goPath);
|
||||
|
||||
const store = new DurableAdvancementKeyStore(storePath);
|
||||
try {
|
||||
const status = await store.tryCommit(key, [key]);
|
||||
await writeFile(resultPath, JSON.stringify({ status }), "utf8");
|
||||
} catch (error) {
|
||||
await writeFile(
|
||||
resultPath,
|
||||
JSON.stringify({ error: error instanceof Error ? error.message : String(error) }),
|
||||
"utf8",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -0,0 +1,52 @@
|
||||
/**
|
||||
* Simulates a live-but-slow lock holder for the DurableAdvancementKeyStore
|
||||
* cross-process regression tests (PR #3442 review, round 4): acquires the
|
||||
* exact same lock file the real store uses, signals readiness, then holds
|
||||
* it until told to release -- proving a real, still-alive holder is never
|
||||
* force-preempted no matter how long its critical section runs, unlike the
|
||||
* round-3 age-based staleness heuristic this replaces.
|
||||
*
|
||||
* Deliberately does not import `DurableAdvancementKeyStore`: it only needs
|
||||
* to occupy the lock file at the path the real class would use, using the
|
||||
* same `open(path, "wx")` protocol, so the *other* worker's real
|
||||
* `tryCommit` call is what gets exercised and observed.
|
||||
*
|
||||
* Usage: node --experimental-transform-types hold-lock-worker.ts
|
||||
* <storePath> <readyPath> <releasePath>
|
||||
*/
|
||||
|
||||
import { access, mkdir, open, unlink, utimes, writeFile } from "node:fs/promises";
|
||||
import { dirname } from "node:path";
|
||||
|
||||
async function waitFor(p: string): Promise<void> {
|
||||
for (;;) {
|
||||
try {
|
||||
await access(p);
|
||||
return;
|
||||
} catch {
|
||||
await new Promise((resolveDelay) => setTimeout(resolveDelay, 5));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const [, , storePath, readyPath, releasePath] = process.argv;
|
||||
if (!storePath || !readyPath || !releasePath) {
|
||||
throw new Error("usage: hold-lock-worker.ts <storePath> <readyPath> <releasePath>");
|
||||
}
|
||||
|
||||
const lockPath = `${storePath}.lock`;
|
||||
await mkdir(dirname(storePath), { recursive: true });
|
||||
const handle = await open(lockPath, "wx");
|
||||
await handle.close();
|
||||
// Exercise the former 30-second reclaim threshold without a slow test.
|
||||
const oldTime = new Date(Date.now() - 60_000);
|
||||
await utimes(lockPath, oldTime, oldTime);
|
||||
|
||||
await writeFile(readyPath, "ready", "utf8");
|
||||
|
||||
await waitFor(releasePath);
|
||||
await unlink(lockPath).catch(() => undefined);
|
||||
}
|
||||
|
||||
main();
|
||||
Reference in New Issue
Block a user