mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description `assemble()` in the openclaw plugin passed `fallback: true` to the SDK's `compress()`. In `sdk/typescript/src/client.ts`, that flag makes a connection error, a timeout or a 5xx retry once and then **resolve** to `makeFallbackResult(messages)` — `tokensBefore: 0`, `tokensSaved: 0`, `compressed: false` — instead of throwing. That result never reached the `catch` that calls `tripCircuit()`. Worse, `assemble()` treated it as a successful no-op and called `resetCircuit()`, so the error counter went back to zero on every turn. Against a proxy that accepts connections but never answers, the circuit breaker could never open: every `assemble()` paid up to `2 × requestTimeoutMs` and then bypassed compression anyway. Closes #4738 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Pass `fallback: false` on the plugin's `compress()` call, so proxy failures throw into the existing `catch` → `tripCircuit()`. Graceful degradation is unchanged: that `catch` already returns the original messages. - Treat a zero `tokensBefore` on **nonempty** input as a failure too, so a result that resolved empty cannot keep the breaker closed either. Empty input is exempt, because the proxy legitimately answers `tokens_before: 0` for an empty `messages` array — caught in review, see the `minContextChars: 0` case below. - `README.md`: the post-cool-down turn does not call `/health`; nothing in the engine ever did. It re-probes by attempting a real compression. Fixed, and the threshold row now says which failures count. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, this is the TypeScript plugin; ran `vitest` instead - [ ] Linting passes (`ruff check .`) — N/A, no Python changed - [ ] Type checking passes (`mypy headroom`) — N/A, no Python changed - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ npx vitest run test/engine.test.ts Test Files 1 passed (1) Tests 41 passed (41) $ npm run typecheck > node node_modules/@typescript/native/bin/tsc --noEmit (clean) $ npm run build ESM Build success in 18ms DTS Build success in 515ms ``` The three new tests each fail on the unpatched tree (verified by stashing only `src/engine.ts` and re-running): ```text × passes fallback: false so proxy failures reach the circuit breaker × opens the circuit on zero-token results, even if the SDK swallows the error × does not count an empty context as a failure ``` Full suite: 158 passed, 2 failed — both pre-existing and unrelated, in `test/advancement-key-store.test.ts`, caused by this machine's Node rejecting `--experimental-transform-types` in the child-process fixture (`node: bad option: --experimental-transform-types`). They fail identically on `upstream/main` without this patch. ## Real Behavior Proof - Environment: Windows 11, Node v26.10.0, `plugins/openclaw` built with `npm run build`, in-tree `headroom-ai@0.42.0`. Stub proxy is a `node:http` server that accepts the connection and never responds. Engine constructed with `requestTimeoutMs: 500`, `circuitBreakerThreshold: 3`, `circuitBreakerCooldownMs: 60000`, `minContextChars: 0`, driven through five `assemble()` calls with a 2000-char message. - Exact command / steps: `npm run build` in `plugins/openclaw`, then `node proof-circuit.mjs`, which starts the hung proxy on an ephemeral port, points the engine at it, and makes five sequential `assemble()` calls. Run once on the unpatched build and once on the patched build. - Observed result: before the patch the breaker never opened and every call paid the full timeout. After the patch it opens after the third failure and calls 4 and 5 skip the proxy entirely (0 ms, 0 requests). Messages returned are the originals in every case, so model output is unchanged; only the latency is removed. Before the patch: ```text call 1: ms=1004 requestsSeenByProxy=2 totalProxyRequests=2 call 2: ms=1001 requestsSeenByProxy=2 totalProxyRequests=4 call 3: ms=1003 requestsSeenByProxy=2 totalProxyRequests=6 call 4: ms=1001 requestsSeenByProxy=2 totalProxyRequests=8 call 5: ms=1002 requestsSeenByProxy=2 totalProxyRequests=10 circuit-opened=false ``` After the patch: ```text call 1: ms=1004 requestsSeenByProxy=2 totalProxyRequests=2 messagesAreOriginal=true call 2: ms=1002 requestsSeenByProxy=2 totalProxyRequests=4 messagesAreOriginal=true call 3: ms=1002 requestsSeenByProxy=2 totalProxyRequests=6 messagesAreOriginal=true call 4: ms=0 requestsSeenByProxy=0 totalProxyRequests=6 messagesAreOriginal=true call 5: ms=0 requestsSeenByProxy=0 totalProxyRequests=6 messagesAreOriginal=true circuit-opened=true warn: [headroom] Circuit breaker opened after 3 errors (last: HeadroomConnectionError: Failed to connect to Headroom at http://127.0.0.1:53545: TimeoutError: The operation was aborted due to timeout); bypassing compression for 60000ms warn: [headroom] Circuit open — using uncompressed messages warn: [headroom] Circuit open — using uncompressed messages ``` - Not tested: a live OpenClaw host with a real provider (the engine needs a host to drive, so I drove `assemble()` directly against the built plugin), and the published `headroom-ai` beyond `0.42.0`. `fallback` has been in the SDK's `CompressOptions` since the first TS release, so the flag resolves against the plugin's `^0.40.0` range too. ## Runtime Rollout Safety - Rollout-managed feature(s): none — this is a plugin-local circuit breaker inside `headroom-openclaw`, with no rollout flag or channel gating involved. - Minimum rollout channel: N/A, no gating is introduced. - Stable/default behavior changed: yes, deliberately. The circuit breaker now opens on proxy failures instead of never opening, so an unhealthy proxy causes compression to be skipped for `circuitBreakerCooldownMs` (default 60 s) rather than being attempted on every turn. Messages are returned unchanged either way, so model output is unaffected; the change removes repeated timeout latency. - Kill switch / disable path: `enabled: false` disables the plugin entirely; `circuitBreakerThreshold` can be raised to make the breaker effectively unreachable; `circuitBreakerCooldownMs` bounds how long each bypass lasts. - Unsafe override required: no. - Qualification impact: none — no provider, transport or config surface changes, and no new dependency. - Rollback path: revert the commit. The prior behavior was a single-line `fallback: true`, and nothing here migrates or persists state, so reverting restores the old behavior exactly. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Review caught one defect in my first attempt, now fixed and covered by a test: with `minContextChars: 0`, `assemble()` forwards empty `messages: []` to the proxy, which answers `200` with `tokens_before: 0`. Counting that as a failure meant a handful of empty-context turns could open the breaker and then skip compression for a healthy proxy. The zero-token check is now scoped to nonempty input. Found with AI assistance (opencode) operated by me.
799 lines
30 KiB
TypeScript
799 lines
30 KiB
TypeScript
import * as fs from "node:fs/promises";
|
|
import * as os from "node:os";
|
|
import * as path from "node:path";
|
|
|
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
|
|
const mocked = vi.hoisted(() => ({
|
|
compress: vi.fn(),
|
|
delegateCompactionToRuntime: vi.fn(),
|
|
start: vi.fn(async () => "http://127.0.0.1:8787"),
|
|
stop: vi.fn(async () => undefined),
|
|
logger: {
|
|
debug: vi.fn(),
|
|
error: vi.fn(),
|
|
info: vi.fn(),
|
|
warn: vi.fn(),
|
|
},
|
|
}));
|
|
|
|
vi.mock("headroom-ai", () => ({
|
|
compress: mocked.compress,
|
|
}));
|
|
|
|
vi.mock("../src/openclaw-compaction.js", () => ({
|
|
delegateCompactionToRuntime: mocked.delegateCompactionToRuntime,
|
|
}));
|
|
|
|
vi.mock("../src/proxy-manager.js", () => ({
|
|
ProxyManager: class {
|
|
start = mocked.start;
|
|
stop = mocked.stop;
|
|
},
|
|
defaultLogger: mocked.logger,
|
|
}));
|
|
|
|
import { HeadroomContextEngine, type HeadroomEngineConfig } from "../src/engine.js";
|
|
import { compress } from "headroom-ai";
|
|
|
|
afterEach(() => {
|
|
mocked.compress.mockReset();
|
|
mocked.delegateCompactionToRuntime.mockReset();
|
|
mocked.start.mockReset();
|
|
mocked.start.mockResolvedValue("http://127.0.0.1:8787");
|
|
mocked.stop.mockClear();
|
|
mocked.logger.debug.mockClear();
|
|
mocked.logger.error.mockClear();
|
|
mocked.logger.info.mockClear();
|
|
mocked.logger.warn.mockClear();
|
|
});
|
|
|
|
describe("HeadroomContextEngine compaction", () => {
|
|
it("delegates persistent compaction to OpenClaw without claiming ownership", async () => {
|
|
const engine = new HeadroomContextEngine();
|
|
const params = {
|
|
sessionId: "session-1",
|
|
sessionKey: "agent:main:session-1",
|
|
tokenBudget: 12_000,
|
|
force: true,
|
|
runtimeContext: { workspaceDir: "/tmp/workspace" },
|
|
};
|
|
const delegatedResult = {
|
|
ok: true,
|
|
compacted: true,
|
|
result: {
|
|
tokensBefore: 20_000,
|
|
tokensAfter: 8_000,
|
|
},
|
|
};
|
|
mocked.delegateCompactionToRuntime.mockResolvedValueOnce(delegatedResult);
|
|
|
|
expect(engine.info.ownsCompaction).toBe(false);
|
|
await expect(engine.compact(params)).resolves.toEqual(delegatedResult);
|
|
|
|
expect(mocked.delegateCompactionToRuntime).toHaveBeenCalledWith(params);
|
|
expect(mocked.compress).not.toHaveBeenCalled();
|
|
expect(engine.getStats().compactions).toBe(1);
|
|
});
|
|
|
|
it("does not count a delegated no-op as a compaction", async () => {
|
|
const engine = new HeadroomContextEngine();
|
|
mocked.delegateCompactionToRuntime.mockResolvedValueOnce({
|
|
ok: true,
|
|
compacted: false,
|
|
reason: "Below compaction threshold",
|
|
});
|
|
|
|
await expect(
|
|
engine.compact({
|
|
sessionId: "session-1",
|
|
sessionKey: "agent:main:session-1",
|
|
}),
|
|
).resolves.toEqual({
|
|
ok: true,
|
|
compacted: false,
|
|
reason: "Below compaction threshold",
|
|
});
|
|
|
|
expect(engine.getStats().compactions).toBe(0);
|
|
});
|
|
|
|
it("propagates delegated compaction failures without reporting success", async () => {
|
|
const engine = new HeadroomContextEngine();
|
|
const failure = new Error("native compaction failed");
|
|
mocked.delegateCompactionToRuntime.mockRejectedValueOnce(failure);
|
|
|
|
await expect(
|
|
engine.compact({
|
|
sessionId: "session-1",
|
|
sessionKey: "agent:main:session-1",
|
|
}),
|
|
).rejects.toBe(failure);
|
|
|
|
expect(engine.getStats().compactions).toBe(0);
|
|
expect(mocked.logger.info).not.toHaveBeenCalled();
|
|
});
|
|
});
|
|
|
|
describe("HeadroomContextEngine proxy startup helpers", () => {
|
|
it("bootstraps by scheduling proxy startup when enabled", async () => {
|
|
const engine = new HeadroomContextEngine();
|
|
|
|
await expect(
|
|
engine.bootstrap({
|
|
sessionId: "session-1",
|
|
sessionFile: "session.jsonl",
|
|
}),
|
|
).resolves.toEqual({
|
|
bootstrapped: true,
|
|
reason: "proxy startup scheduled",
|
|
});
|
|
expect(mocked.start).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("removes unsubscribed proxy listeners before notifying readiness", async () => {
|
|
const engine = new HeadroomContextEngine();
|
|
const first = vi.fn();
|
|
const second = vi.fn();
|
|
|
|
const unsubscribeFirst = engine.onProxyReady(first);
|
|
engine.onProxyReady(second);
|
|
unsubscribeFirst();
|
|
|
|
engine.ensureProxyStarted();
|
|
await engine.ensureProxyUrl();
|
|
|
|
expect(first).not.toHaveBeenCalled();
|
|
expect(second).toHaveBeenCalledWith("http://127.0.0.1:8787");
|
|
});
|
|
|
|
it("returns the existing proxy URL without starting again", async () => {
|
|
const engine = new HeadroomContextEngine();
|
|
|
|
(engine as { proxyUrl: string | null }).proxyUrl = "http://127.0.0.1:8787";
|
|
|
|
await expect(engine.ensureProxyUrl()).resolves.toBe("http://127.0.0.1:8787");
|
|
expect(mocked.start).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("throws when proxy startup is disabled", async () => {
|
|
const engine = new HeadroomContextEngine({ enabled: false });
|
|
|
|
await expect(engine.ensureProxyUrl()).rejects.toThrow("Headroom proxy startup is disabled");
|
|
expect(mocked.start).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("does not emit an unhandledRejection when fire-and-forget startup fails", async () => {
|
|
mocked.start.mockReset();
|
|
mocked.start.mockRejectedValue(new Error("proxy boom"));
|
|
|
|
const engine = new HeadroomContextEngine();
|
|
const unhandled: unknown[] = [];
|
|
const onUnhandled = (reason: unknown) => unhandled.push(reason);
|
|
process.on("unhandledRejection", onUnhandled);
|
|
|
|
try {
|
|
// Fire-and-forget: caller intentionally does not await.
|
|
engine.ensureProxyStarted();
|
|
// Let the startup promise settle and any microtasks/macrotasks flush.
|
|
await new Promise((resolve) => setTimeout(resolve, 0));
|
|
|
|
expect(unhandled).toEqual([]);
|
|
expect(mocked.logger.warn).toHaveBeenCalledWith(
|
|
expect.stringContaining("Headroom proxy unavailable"),
|
|
);
|
|
} finally {
|
|
process.off("unhandledRejection", onUnhandled);
|
|
}
|
|
});
|
|
|
|
it("stores the startup failure in getProxyStartupError()", async () => {
|
|
const failure = new Error("proxy boom");
|
|
mocked.start.mockReset();
|
|
mocked.start.mockRejectedValue(failure);
|
|
|
|
const engine = new HeadroomContextEngine();
|
|
expect(engine.getProxyStartupError()).toBeNull();
|
|
|
|
engine.ensureProxyStarted();
|
|
await new Promise((resolve) => setTimeout(resolve, 0));
|
|
|
|
expect(engine.getProxyStartupError()).toBe(failure);
|
|
});
|
|
|
|
it("allows retrying startup after a failure", async () => {
|
|
mocked.start.mockReset();
|
|
mocked.start
|
|
.mockRejectedValueOnce(new Error("proxy boom"))
|
|
.mockResolvedValueOnce("http://127.0.0.1:8787");
|
|
|
|
const engine = new HeadroomContextEngine();
|
|
|
|
engine.ensureProxyStarted();
|
|
await new Promise((resolve) => setTimeout(resolve, 0));
|
|
expect(engine.getProxyStartupError()).toBeInstanceOf(Error);
|
|
|
|
// A second attempt is possible once the failed promise has cleared.
|
|
const url = await engine.ensureProxyUrl();
|
|
expect(url).toBe("http://127.0.0.1:8787");
|
|
expect(engine.getProxyStartupError()).toBeNull();
|
|
expect(mocked.start).toHaveBeenCalledTimes(2);
|
|
});
|
|
|
|
it("ensureProxyUrl rejects cleanly on startup failure without unhandledRejection", async () => {
|
|
const failure = new Error("proxy boom");
|
|
mocked.start.mockReset();
|
|
mocked.start.mockRejectedValue(failure);
|
|
|
|
const engine = new HeadroomContextEngine();
|
|
const unhandled: unknown[] = [];
|
|
const onUnhandled = (reason: unknown) => unhandled.push(reason);
|
|
process.on("unhandledRejection", onUnhandled);
|
|
|
|
try {
|
|
await expect(engine.ensureProxyUrl()).rejects.toBe(failure);
|
|
await new Promise((resolve) => setTimeout(resolve, 0));
|
|
expect(unhandled).toEqual([]);
|
|
} finally {
|
|
process.off("unhandledRejection", onUnhandled);
|
|
}
|
|
});
|
|
|
|
it("isolates and logs proxy-ready listener rejections", async () => {
|
|
const engine = new HeadroomContextEngine();
|
|
const failing = vi.fn(async () => {
|
|
throw new Error("listener boom");
|
|
});
|
|
const healthy = vi.fn();
|
|
|
|
engine.onProxyReady(failing);
|
|
engine.onProxyReady(healthy);
|
|
|
|
engine.ensureProxyStarted();
|
|
// ensureProxyUrl must still resolve despite the listener throwing.
|
|
await expect(engine.ensureProxyUrl()).resolves.toBe("http://127.0.0.1:8787");
|
|
|
|
expect(failing).toHaveBeenCalled();
|
|
expect(healthy).toHaveBeenCalledWith("http://127.0.0.1:8787");
|
|
expect(mocked.logger.warn).toHaveBeenCalledWith(
|
|
expect.stringContaining("Headroom proxy ready listener failed"),
|
|
);
|
|
expect(engine.getProxyStartupError()).toBeNull();
|
|
});
|
|
|
|
it("schedules startup and returns original messages when assembling before proxy readiness", async () => {
|
|
const engine = new HeadroomContextEngine();
|
|
const messages = [{ role: "user", content: "hello" }];
|
|
|
|
await expect(
|
|
engine.assemble({
|
|
sessionId: "session-1",
|
|
messages,
|
|
}),
|
|
).resolves.toEqual({
|
|
messages,
|
|
estimatedTokens: 0,
|
|
});
|
|
expect(mocked.start).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("passes the configured request timeout to the aborting SDK client", async () => {
|
|
vi.mocked(compress).mockResolvedValue({
|
|
compressed: false,
|
|
messages: [{ role: "user", content: "hello" }],
|
|
tokensBefore: 5,
|
|
tokensAfter: 5,
|
|
tokensSaved: 0,
|
|
});
|
|
|
|
const engine = new HeadroomContextEngine({ requestTimeoutMs: 1_234, minContextChars: 0 });
|
|
(engine as { proxyUrl: string | null }).proxyUrl = "http://127.0.0.1:8787";
|
|
|
|
await engine.assemble({
|
|
sessionId: "session-1",
|
|
messages: [{ role: "user", content: "hello" }],
|
|
});
|
|
|
|
expect(compress).toHaveBeenCalledWith(
|
|
expect.any(Array),
|
|
expect.objectContaining({ timeout: 1_234 }),
|
|
);
|
|
});
|
|
|
|
it("skips proxy compression for short contexts with no meaningful savings", async () => {
|
|
const messages = [{ role: "user", content: "hello" }];
|
|
const engine = new HeadroomContextEngine();
|
|
(engine as { proxyUrl: string | null }).proxyUrl = "http://127.0.0.1:8787";
|
|
|
|
await expect(engine.assemble({ sessionId: "session-1", messages })).resolves.toEqual({
|
|
messages,
|
|
estimatedTokens: 0,
|
|
});
|
|
expect(compress).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("counts a single large tool result as eligible context", async () => {
|
|
const messages = [{ role: "toolResult", toolCallId: "call-1", content: "x".repeat(800) }];
|
|
vi.mocked(compress).mockResolvedValue({
|
|
compressed: false,
|
|
messages: [{ role: "tool", tool_call_id: "call-1", content: "x".repeat(800) }],
|
|
tokensBefore: 200,
|
|
tokensAfter: 200,
|
|
tokensSaved: 0,
|
|
});
|
|
const engine = new HeadroomContextEngine();
|
|
(engine as { proxyUrl: string | null }).proxyUrl = "http://127.0.0.1:8787";
|
|
|
|
await engine.assemble({ sessionId: "session-1", messages });
|
|
expect(compress).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("allows the short-context gate to be disabled", async () => {
|
|
vi.mocked(compress).mockResolvedValue({
|
|
compressed: false,
|
|
messages: [{ role: "user", content: "hello" }],
|
|
tokensBefore: 2,
|
|
tokensAfter: 2,
|
|
tokensSaved: 0,
|
|
});
|
|
const engine = new HeadroomContextEngine({ minContextChars: 0 });
|
|
(engine as { proxyUrl: string | null }).proxyUrl = "http://127.0.0.1:8787";
|
|
|
|
await engine.assemble({ sessionId: "session-1", messages: [{ role: "user", content: "hello" }] });
|
|
expect(compress).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("opens the circuit after consecutive compression failures", async () => {
|
|
vi.mocked(compress).mockRejectedValue(new Error("proxy stalled"));
|
|
const messages = [{ role: "user", content: "hello" }];
|
|
const engine = new HeadroomContextEngine({
|
|
minContextChars: 0,
|
|
circuitBreakerThreshold: 2,
|
|
circuitBreakerCooldownMs: 60_000,
|
|
});
|
|
(engine as { proxyUrl: string | null }).proxyUrl = "http://127.0.0.1:8787";
|
|
|
|
await engine.assemble({ sessionId: "session-1", messages });
|
|
await engine.assemble({ sessionId: "session-1", messages });
|
|
await expect(engine.assemble({ sessionId: "session-1", messages })).resolves.toEqual({
|
|
messages,
|
|
estimatedTokens: 0,
|
|
});
|
|
|
|
expect(compress).toHaveBeenCalledTimes(2);
|
|
expect(mocked.logger.warn).toHaveBeenCalledWith(
|
|
expect.stringContaining("Circuit breaker opened"),
|
|
);
|
|
});
|
|
|
|
// Regression: engine.ts used to pass `fallback: true`, so the SDK resolved a
|
|
// connection error, timeout or 5xx to a zero-token stub instead of throwing.
|
|
// That stub looked like a successful no-op, so every assemble() reset the
|
|
// breaker and it could never open on an unhealthy proxy.
|
|
it("passes fallback: false so proxy failures reach the circuit breaker", async () => {
|
|
vi.mocked(compress).mockResolvedValue({
|
|
compressed: false,
|
|
messages: [{ role: "user", content: "hello" }],
|
|
tokensBefore: 0,
|
|
tokensAfter: 0,
|
|
tokensSaved: 0,
|
|
});
|
|
const engine = new HeadroomContextEngine({ minContextChars: 0 });
|
|
(engine as { proxyUrl: string | null }).proxyUrl = "http://127.0.0.1:8787";
|
|
|
|
await engine.assemble({ sessionId: "session-1", messages: [{ role: "user", content: "hello" }] });
|
|
|
|
expect(vi.mocked(compress).mock.calls[0][1]).toMatchObject({ fallback: false });
|
|
});
|
|
|
|
it("opens the circuit on zero-token results, even if the SDK swallows the error", async () => {
|
|
vi.mocked(compress).mockResolvedValue({
|
|
compressed: false,
|
|
messages: [{ role: "user", content: "hello" }],
|
|
tokensBefore: 0,
|
|
tokensAfter: 0,
|
|
tokensSaved: 0,
|
|
});
|
|
const messages = [{ role: "user", content: "hello" }];
|
|
const engine = new HeadroomContextEngine({
|
|
minContextChars: 0,
|
|
circuitBreakerThreshold: 2,
|
|
circuitBreakerCooldownMs: 60_000,
|
|
});
|
|
(engine as { proxyUrl: string | null }).proxyUrl = "http://127.0.0.1:8787";
|
|
|
|
await engine.assemble({ sessionId: "session-1", messages });
|
|
await engine.assemble({ sessionId: "session-1", messages });
|
|
await expect(engine.assemble({ sessionId: "session-1", messages })).resolves.toEqual({
|
|
messages,
|
|
estimatedTokens: 0,
|
|
});
|
|
|
|
// The third call is served from the open circuit, not the proxy.
|
|
expect(compress).toHaveBeenCalledTimes(2);
|
|
expect(mocked.logger.warn).toHaveBeenCalledWith(
|
|
expect.stringContaining("Circuit breaker opened"),
|
|
);
|
|
});
|
|
|
|
it("does not count an empty context as a failure", async () => {
|
|
// The proxy answers 200 with tokens_before: 0 for an empty message list, so
|
|
// zero tokens there is a real answer, not a failure.
|
|
vi.mocked(compress).mockResolvedValue({
|
|
compressed: false,
|
|
messages: [],
|
|
tokensBefore: 0,
|
|
tokensAfter: 0,
|
|
tokensSaved: 0,
|
|
});
|
|
const engine = new HeadroomContextEngine({
|
|
minContextChars: 0,
|
|
circuitBreakerThreshold: 1,
|
|
});
|
|
(engine as { proxyUrl: string | null }).proxyUrl = "http://127.0.0.1:8787";
|
|
|
|
await engine.assemble({ sessionId: "session-1", messages: [] });
|
|
await engine.assemble({ sessionId: "session-1", messages: [] });
|
|
|
|
// The breaker must stay closed, so the next call still reaches the proxy.
|
|
expect(compress).toHaveBeenCalledTimes(2);
|
|
expect(mocked.logger.warn).not.toHaveBeenCalledWith(
|
|
expect.stringContaining("Circuit breaker opened"),
|
|
);
|
|
});
|
|
|
|
it("does not count a genuine no-savings result as a failure", async () => {
|
|
vi.mocked(compress).mockResolvedValue({
|
|
compressed: false,
|
|
messages: [{ role: "user", content: "hello" }],
|
|
tokensBefore: 5,
|
|
tokensAfter: 5,
|
|
tokensSaved: 0,
|
|
});
|
|
const engine = new HeadroomContextEngine({
|
|
minContextChars: 0,
|
|
circuitBreakerThreshold: 1,
|
|
});
|
|
(engine as { proxyUrl: string | null }).proxyUrl = "http://127.0.0.1:8787";
|
|
|
|
await engine.assemble({ sessionId: "session-1", messages: [{ role: "user", content: "hello" }] });
|
|
await engine.assemble({ sessionId: "session-1", messages: [{ role: "user", content: "hello" }] });
|
|
|
|
// Nothing was saved, but the proxy answered — the breaker must stay closed.
|
|
expect(compress).toHaveBeenCalledTimes(2);
|
|
expect(mocked.logger.warn).not.toHaveBeenCalledWith(
|
|
expect.stringContaining("Circuit breaker opened"),
|
|
);
|
|
});
|
|
});
|
|
|
|
describe("HeadroomContextEngine transcriptSemantics contract", () => {
|
|
let commitLogDir: string;
|
|
let commitLogPath: string;
|
|
|
|
beforeEach(async () => {
|
|
commitLogDir = await fs.mkdtemp(path.join(os.tmpdir(), "headroom-commit-log-"));
|
|
commitLogPath = path.join(commitLogDir, "commit-log.json");
|
|
});
|
|
|
|
afterEach(async () => {
|
|
await fs.rm(commitLogDir, { recursive: true, force: true });
|
|
});
|
|
|
|
it("declares the durable-commit transcript semantics OpenClaw requires", () => {
|
|
const engine = new HeadroomContextEngine({ commitLogPath });
|
|
|
|
expect(engine.info.transcriptSemantics).toEqual({
|
|
currentTurnFence: "before-current-turn-entry-v1",
|
|
turnAdvancementIdempotency: "atomic-idempotent-v1",
|
|
});
|
|
});
|
|
|
|
it("commits a new advancement key", async () => {
|
|
const engine = new HeadroomContextEngine({ commitLogPath });
|
|
|
|
await expect(
|
|
engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
|
|
).resolves.toEqual({ status: "committed" });
|
|
});
|
|
|
|
it("reports duplicate on a retried advancement key", async () => {
|
|
const engine = new HeadroomContextEngine({ commitLogPath });
|
|
|
|
await expect(
|
|
engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
|
|
).resolves.toEqual({ status: "committed" });
|
|
await expect(
|
|
engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
|
|
).resolves.toEqual({ status: "duplicate" });
|
|
});
|
|
|
|
it("treats distinct advancement keys independently", async () => {
|
|
const engine = new HeadroomContextEngine({ commitLogPath });
|
|
|
|
await expect(
|
|
engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
|
|
).resolves.toEqual({ status: "committed" });
|
|
await expect(
|
|
engine.commitTurn({ advancementKey: "turn-2", messages: [] }),
|
|
).resolves.toEqual({ status: "committed" });
|
|
});
|
|
|
|
it("reports duplicate for a key committed before a process restart", async () => {
|
|
// Simulate a restart: a brand new engine instance (no shared in-memory
|
|
// state) pointed at the same durable commit-log path.
|
|
const before = new HeadroomContextEngine({ commitLogPath });
|
|
await expect(
|
|
before.commitTurn({ advancementKey: "turn-restart", messages: [] }),
|
|
).resolves.toEqual({ status: "committed" });
|
|
|
|
const after = new HeadroomContextEngine({ commitLogPath });
|
|
await expect(
|
|
after.commitTurn({ advancementKey: "turn-restart", messages: [] }),
|
|
).resolves.toEqual({ status: "duplicate" });
|
|
});
|
|
|
|
it(
|
|
"never forgets a key regardless of how many other keys were committed since",
|
|
async () => {
|
|
// Regression: the old implementation evicted the oldest key past a
|
|
// 512-entry cap, so a retry of an early key was wrongly re-accepted as
|
|
// new instead of reported as a duplicate. There is no such cap now.
|
|
const engine = new HeadroomContextEngine({ commitLogPath });
|
|
|
|
for (let i = 0; i < 600; i++) {
|
|
await engine.commitTurn({ advancementKey: `turn-${i}`, messages: [] });
|
|
}
|
|
|
|
await expect(
|
|
engine.commitTurn({ advancementKey: "turn-0", messages: [] }),
|
|
).resolves.toEqual({ status: "duplicate" });
|
|
},
|
|
20_000,
|
|
);
|
|
|
|
it("records only the advancement key, so the store doesn't grow with the turn's size", async () => {
|
|
// Regression: every accepted turn's messages used to be kept forever in one
|
|
// file that was rewritten on each commit. OpenClaw's transcript already holds
|
|
// them, and this engine never reads them back.
|
|
const engine = new HeadroomContextEngine({ commitLogPath });
|
|
const toolOutput = "x".repeat(1_000_000);
|
|
|
|
await expect(
|
|
engine.commitTurn({
|
|
advancementKey: "turn-1",
|
|
messages: [{ role: "toolResult", content: [{ type: "text", text: toolOutput }] }],
|
|
}),
|
|
).resolves.toEqual({ status: "committed" });
|
|
|
|
let storedBytes = 0;
|
|
for (const entry of await fs.readdir(commitLogDir, { recursive: true, withFileTypes: true })) {
|
|
if (entry.isFile()) {
|
|
storedBytes += (await fs.stat(path.join(entry.parentPath, entry.name))).size;
|
|
}
|
|
}
|
|
expect(storedBytes).toBeGreaterThan(0);
|
|
expect(storedBytes).toBeLessThan(1_000);
|
|
});
|
|
});
|
|
|
|
describe("HeadroomContextEngine assemble() compression notice", () => {
|
|
const HEADROOM_COMPRESSION_NOTICE =
|
|
"[Headroom is compressing tool outputs in this session. Use headroom_retrieve if you need the original, uncompressed content.]";
|
|
const messages = [{ role: "user", content: "hello" }];
|
|
|
|
function mockCompressResult(overrides: Partial<{
|
|
compressed: boolean;
|
|
tokensSaved: number;
|
|
tokensBefore: number;
|
|
tokensAfter: number;
|
|
}>) {
|
|
const tokensSaved = overrides.tokensSaved ?? 0;
|
|
return {
|
|
compressed: overrides.compressed ?? tokensSaved > 0,
|
|
messages: [{ role: "user", content: "hello" }],
|
|
tokensBefore: overrides.tokensBefore ?? 1000,
|
|
tokensAfter: overrides.tokensAfter ?? 1000 - tokensSaved,
|
|
tokensSaved,
|
|
};
|
|
}
|
|
|
|
function readyEngine(config?: HeadroomEngineConfig) {
|
|
const engine = new HeadroomContextEngine({ minContextChars: 0, ...config });
|
|
(engine as unknown as { proxyUrl: string | null }).proxyUrl = "http://127.0.0.1:8787";
|
|
return engine;
|
|
}
|
|
|
|
it("returns the static notice, with no interpolated count, once tokensSaved crosses the threshold", async () => {
|
|
vi.mocked(compress).mockResolvedValueOnce(mockCompressResult({ tokensSaved: 150 }));
|
|
|
|
const engine = readyEngine();
|
|
const result = await engine.assemble({ sessionId: "s1", messages });
|
|
|
|
expect(result.systemPromptAddition).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
});
|
|
|
|
it("returns byte-identical notices across turns with different tokensSaved amounts", async () => {
|
|
vi.mocked(compress)
|
|
.mockResolvedValueOnce(mockCompressResult({ tokensSaved: 150 }))
|
|
.mockResolvedValueOnce(mockCompressResult({ tokensSaved: 9000 }));
|
|
|
|
const engine = readyEngine();
|
|
const first = await engine.assemble({ sessionId: "s1", messages });
|
|
const second = await engine.assemble({ sessionId: "s1", messages });
|
|
|
|
expect(first.systemPromptAddition).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
expect(second.systemPromptAddition).toBe(first.systemPromptAddition);
|
|
});
|
|
|
|
it("keeps the notice present on a later turn that has nothing to compress", async () => {
|
|
vi.mocked(compress)
|
|
.mockResolvedValueOnce(mockCompressResult({ tokensSaved: 150 }))
|
|
.mockResolvedValueOnce(mockCompressResult({ compressed: false, tokensSaved: 0 }));
|
|
|
|
const engine = readyEngine();
|
|
const first = await engine.assemble({ sessionId: "s1", messages });
|
|
const second = await engine.assemble({ sessionId: "s1", messages });
|
|
|
|
expect(first.systemPromptAddition).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
expect(second.systemPromptAddition).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
});
|
|
|
|
it("keeps an earned notice on a later context below the compression threshold", async () => {
|
|
vi.mocked(compress).mockResolvedValueOnce(mockCompressResult({ tokensSaved: 150 }));
|
|
const engine = readyEngine({ minContextChars: 800 });
|
|
const first = await engine.assemble({
|
|
sessionId: "s1", messages: [{ role: "user", content: "x".repeat(1000) }],
|
|
});
|
|
const second = await engine.assemble({ sessionId: "s1", messages });
|
|
expect(first.systemPromptAddition).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
expect(second.systemPromptAddition).toBe(first.systemPromptAddition);
|
|
expect(compress).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it("never returns a notice when announceCompression is false", async () => {
|
|
vi.mocked(compress).mockResolvedValueOnce(mockCompressResult({ tokensSaved: 500 }));
|
|
|
|
const engine = readyEngine({ announceCompression: false });
|
|
const result = await engine.assemble({ sessionId: "s1", messages });
|
|
|
|
expect(result.systemPromptAddition).toBeUndefined();
|
|
});
|
|
|
|
it("does not announce for compression below the noise threshold", async () => {
|
|
vi.mocked(compress).mockResolvedValueOnce(mockCompressResult({ tokensSaved: 50 }));
|
|
|
|
const engine = readyEngine();
|
|
const result = await engine.assemble({ sessionId: "s1", messages });
|
|
|
|
expect(result.systemPromptAddition).toBeUndefined();
|
|
});
|
|
|
|
it("does not leak a session's announcement to a different session on the same engine", async () => {
|
|
vi.mocked(compress)
|
|
.mockResolvedValueOnce(mockCompressResult({ tokensSaved: 150 }))
|
|
.mockResolvedValueOnce(mockCompressResult({ compressed: false, tokensSaved: 0 }));
|
|
|
|
const engine = readyEngine();
|
|
const sessionA = await engine.assemble({ sessionId: "a", messages });
|
|
const sessionB = await engine.assemble({ sessionId: "b", messages });
|
|
|
|
expect(sessionA.systemPromptAddition).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
expect(sessionB.systemPromptAddition).toBeUndefined();
|
|
});
|
|
|
|
it("keeps announcing for the session that earned the notice even after another session is seen", async () => {
|
|
vi.mocked(compress)
|
|
.mockResolvedValueOnce(mockCompressResult({ tokensSaved: 150 }))
|
|
.mockResolvedValueOnce(mockCompressResult({ compressed: false, tokensSaved: 0 }))
|
|
.mockResolvedValueOnce(mockCompressResult({ compressed: false, tokensSaved: 0 }));
|
|
|
|
const engine = readyEngine();
|
|
await engine.assemble({ sessionId: "a", messages });
|
|
await engine.assemble({ sessionId: "b", messages });
|
|
const sessionAAgain = await engine.assemble({ sessionId: "a", messages });
|
|
|
|
expect(sessionAAgain.systemPromptAddition).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
});
|
|
|
|
it("evicts the oldest tracked session once the announced-session bound is exceeded", async () => {
|
|
vi.mocked(compress).mockResolvedValue(mockCompressResult({ tokensSaved: 150 }));
|
|
|
|
const engine = readyEngine();
|
|
for (let i = 0; i < 1000; i++) {
|
|
await engine.assemble({ sessionId: `session-${i}`, messages });
|
|
}
|
|
const overflow = await engine.assemble({ sessionId: "session-overflow", messages });
|
|
|
|
vi.mocked(compress).mockResolvedValueOnce(mockCompressResult({ compressed: false, tokensSaved: 0 }));
|
|
const oldest = await engine.assemble({ sessionId: "session-0", messages });
|
|
vi.mocked(compress).mockResolvedValueOnce(mockCompressResult({ compressed: false, tokensSaved: 0 }));
|
|
const newest = await engine.assemble({ sessionId: "session-overflow", messages });
|
|
|
|
expect(overflow.systemPromptAddition).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
expect(oldest.systemPromptAddition).toBeUndefined();
|
|
expect(newest.systemPromptAddition).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
});
|
|
|
|
it("keeps the earned notice byte-identical through a timeout, a circuit-open turn, and recovery", async () => {
|
|
vi.useFakeTimers();
|
|
try {
|
|
const requestTimeoutMs = 5_000;
|
|
const circuitBreakerCooldownMs = 10_000;
|
|
const engine = readyEngine({
|
|
requestTimeoutMs,
|
|
circuitBreakerThreshold: 1,
|
|
circuitBreakerCooldownMs,
|
|
});
|
|
|
|
vi.mocked(compress).mockResolvedValueOnce(mockCompressResult({ tokensSaved: 150 }));
|
|
const turn1 = await engine.assemble({ sessionId: "s1", messages });
|
|
const firstNotice = turn1.systemPromptAddition;
|
|
expect(firstNotice).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
|
|
vi.mocked(compress).mockImplementationOnce(() => new Promise((_, reject) => {
|
|
setTimeout(() => reject(new Error("SDK request timed out")), requestTimeoutMs);
|
|
}));
|
|
const turn2Promise = engine.assemble({ sessionId: "s1", messages });
|
|
await vi.advanceTimersByTimeAsync(requestTimeoutMs);
|
|
const turn2 = await turn2Promise;
|
|
expect(turn2.systemPromptAddition).toBe(firstNotice);
|
|
|
|
const turn3 = await engine.assemble({ sessionId: "s1", messages });
|
|
expect(turn3.systemPromptAddition).toBe(firstNotice);
|
|
expect(compress).toHaveBeenCalledTimes(2);
|
|
|
|
await vi.advanceTimersByTimeAsync(circuitBreakerCooldownMs);
|
|
vi.mocked(compress).mockResolvedValueOnce(mockCompressResult({ compressed: false, tokensSaved: 0 }));
|
|
const turn4 = await engine.assemble({ sessionId: "s1", messages });
|
|
expect(turn4.systemPromptAddition).toBe(firstNotice);
|
|
expect(turn4.systemPromptAddition).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
} finally {
|
|
vi.useRealTimers();
|
|
}
|
|
});
|
|
|
|
it("preserves an earned notice when the proxy becomes unavailable", async () => {
|
|
vi.mocked(compress).mockResolvedValueOnce(mockCompressResult({ tokensSaved: 150 }));
|
|
|
|
const engine = readyEngine();
|
|
const first = await engine.assemble({ sessionId: "s1", messages });
|
|
expect(first.systemPromptAddition).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
|
|
(engine as unknown as { proxyUrl: string | null }).proxyUrl = null;
|
|
const second = await engine.assemble({ sessionId: "s1", messages });
|
|
|
|
expect(second.systemPromptAddition).toBe(HEADROOM_COMPRESSION_NOTICE);
|
|
});
|
|
|
|
it("never surfaces a notice in any fallback path when announceCompression is false", async () => {
|
|
vi.useFakeTimers();
|
|
try {
|
|
const requestTimeoutMs = 5_000;
|
|
const engine = readyEngine({
|
|
announceCompression: false,
|
|
requestTimeoutMs,
|
|
circuitBreakerThreshold: 1,
|
|
circuitBreakerCooldownMs: 10_000,
|
|
});
|
|
|
|
vi.mocked(compress).mockResolvedValueOnce(mockCompressResult({ tokensSaved: 150 }));
|
|
const turn1 = await engine.assemble({ sessionId: "s1", messages });
|
|
expect(turn1.systemPromptAddition).toBeUndefined();
|
|
|
|
vi.mocked(compress).mockImplementationOnce(() => new Promise((_, reject) => {
|
|
setTimeout(() => reject(new Error("SDK request timed out")), requestTimeoutMs);
|
|
}));
|
|
const turn2Promise = engine.assemble({ sessionId: "s1", messages });
|
|
await vi.advanceTimersByTimeAsync(requestTimeoutMs);
|
|
const turn2 = await turn2Promise;
|
|
expect(turn2.systemPromptAddition).toBeUndefined();
|
|
|
|
const turn3 = await engine.assemble({ sessionId: "s1", messages });
|
|
expect(turn3.systemPromptAddition).toBeUndefined();
|
|
} finally {
|
|
vi.useRealTimers();
|
|
}
|
|
});
|
|
});
|