Files
headroom/plugins/openclaw/openclaw.plugin.json
24acbe8f32 fix(openclaw): record turn advancements as one marker file per key (#4212)
## Description

OpenClaw 2026.9.9 started calling the context engine's `commitTurn`
after every accepted turn. While a commit is pending, it runs every
later turn of that session on its legacy context path, which means no
Headroom compression. The plugin's turn-commit store (from #3442) has
three ways to leave a commit pending, and this PR removes all three.
Each committed key becomes one small marker file, created with exclusive
creation (`O_CREAT | O_EXCL`). There's no lock and no shared file to
rewrite, and only the key is stored.

**What we saw.** We run the plugin on Kubernetes with a read-only root
filesystem. After upgrading OpenClaw to 2026.9.9, every turn logged:

```
[context-engine] durable turn advancement remains queued: <key>: ENOENT: no such file or directory, mkdir '/home/openclaw/.headroom'
```

On the session's next turn, OpenClaw retries the pending commit before
assembling context. When the retry fails, it degrades that turn to the
legacy engine (`degradeBeforeStart("pending durable turn advancement
could not be completed before the next turn")` in OpenClaw's
`context-engine-turn-attempt`). A failed commit only increments
`attempt_count`, so the row stays pending and every later turn of the
session degrades too.

**The three problems in `advancement-key-store.ts`:**

1. **Read-only home.** The default
`~/.headroom/openclaw/commit-log.json` can't be created on a read-only
root. Node reports the failed `mkdir` as `ENOENT`, which reads like a
bug rather than a setting. `commitLogPath` exists on the engine but not
in the manifest's `configSchema` (`additionalProperties: false`), so
`HEADROOM_WORKSPACE_DIR` is the only way to move it.
2. **Unbounded growth.** Every accepted turn's `messages` (OpenClaw caps
a turn at 8 MiB) are kept forever in one JSON file. Each commit
re-reads, parses, copies and rewrites the whole file on the gateway's
event loop, so the file and each commit's latency grow with every turn
ever committed. Nothing reads the messages back (`get` is test-only),
and OpenClaw's transcript already holds them.
3. **A lock nothing reclaims.** By design, the lock is never
force-broken. A gateway killed mid-commit (an OOM kill, a node drain)
leaves `commit-log.json.lock` behind, and every later commit waits 60 s
and then fails until someone deletes it. With the retry above, every
turn then starts 60 s late before degrading.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **One marker per committed key**, at
`commit-log.d/<sha256[0:2]>/<sha256(key)>` next to the old log path,
created with `open(path, "wx")`. The first creator in any process gets
`"committed"`; every later caller gets `EEXIST`, i.e. `"duplicate"`.
Only that exclusive create can mean `"duplicate"`, and only when what
exists at the path is a marker file. An `EEXIST` from creating the shard
directory (a file where the directory belongs), or anything but a file
at the marker's path, is a storage error. The marker holds
`{advancementKey, committedAt}`: 99 bytes for OpenClaw's UUID keys.
- **Only the key is recorded.** For this engine, the advancement *is*
the key: `assemble()` works from the messages OpenClaw passes in, and
the plugin keeps no turn state. The contract asks for "one atomic,
idempotent write keyed by `advancementKey`" (OpenClaw
`docs/concepts/context-engine.md`), and creating the marker is that
write.
- **Keys are still never evicted**, for the reason #3442 gives.
- **Migration.** Before every commit and lookup, a store checks for a
`commit-log.json` from an earlier version. With no log there, that check
is one failed `open()`. If a log is there and it's a version the store
hasn't migrated, the store moves its keys into markers and records which
version that was (size, mtime, inode) in
`commit-log.d/migrated-legacy-log.json`. That record only saves work, so
a store that can't write it carries on and reads the log again later.
The log's version and its contents come from one open handle. The log
itself is never moved or changed, so an older version still running
against the same workspace keeps its own record, and nothing has to
write to the log's directory (which may be read-only, with a writable
marker volume mounted beneath it). An unchanged log is skipped by its
metadata alone, even if it can no longer be read. A changed one is
migrated again, so a key an older version commits at any moment, even
while a store is migrating the log, is never accepted as new. Only keys
the store hasn't seen get markers, so a rewrite costs only the keys it
adds. Both the object and the older array format are read. Several
processes can migrate at once safely, since creating an existing marker
is a no-op. A corrupt log still fails loudly rather than being read as
empty, and the store recovers once it's fixed.
- **An actionable error.** When the marker directory can't be created,
the message says what to do: `Cannot record turn advancements under
<dir>: <error>. Point HEADROOM_WORKSPACE_DIR (or the plugin's
commitLogPath) at a writable directory.` A lookup in a store whose path
a file blocks fails the same way on every platform, although Windows
reports such a path as `ENOENT`, which is also what a missing marker
gives.
- **`commitLogPath` is settable.** It's declared in
`openclaw.plugin.json`, with a UI hint and a README row.
- **Tests.** The store suite is rewritten for the new design. It keeps
the existing guarantees: committed, then duplicate; survives a restart;
never evicts across 600 keys; exactly one winner in-process and across
two OS processes; a failed write is retried, not remembered. New tests
cover:
  - a commit stores one small marker holding only the key;
- committing more keys never rewrites an earlier marker (same inode and
mtime);
  - a leftover `.lock` from a crashed process doesn't block a commit;
- a file where the marker directory or a shard directory belongs, and a
directory where a marker belongs: each is an error, never `"duplicate"`,
and the same key commits once the path is cleared;
- a file blocking the store's directory or its marker directory,
reported the way Windows reports it (`ENOENT`, not `ENOTDIR`): a lookup
is a storage error, not an absent key;
  - both legacy formats, including two stores migrating concurrently;
- a legacy log in a read-only directory: commits go through; a restart
doesn't read the unchanged log again; a migrated log that can no longer
be read doesn't block commits; and once the directory is writable, a key
an older version commits the old way (temp file + rename) is migrated
before a running store's next commit;
- a key an older version commits while a store is migrating the log: the
store's next lookup finds it, and its next commit, like a new store's,
is `"duplicate"`;
- an older version rewriting the log: only the key it added gets a
marker;
- a migration record that can't be written: commits go through, in that
store and in one started later;
  - a corrupt legacy log.

In `engine.test.ts`, a commit carrying 1 MB of tool output stores under
1 KB. The lock-holder fixture is removed, since there is no lock.

## Testing

- [ ] Unit tests pass (`pytest`) (n/a, TypeScript plugin only)
- [ ] Linting passes (`ruff check .`) (n/a)
- [ ] Type checking passes (`mypy headroom`) (n/a; `npm run typecheck`
passes)
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)

### Test Output

```text
$ cd plugins/openclaw && npm run typecheck        # exit 0
$ npx vitest run                                  # Node v24.21.0, macOS
 Test Files  9 passed (9)
      Tests  139 passed (139)
$ npx vitest run                                  # Node v22.22.0, GitHub Actions windows-latest
 Test Files  9 passed (9)
      Tests  137 passed | 2 skipped (139)
```

## Real Behavior Proof

- Environment: macOS, Node v24.21.0, for the benchmark below: "old" is
`advancement-key-store.ts` from `main`, "new" is this PR, and the same
script drives both. In production: OpenClaw 2026.9.9 on Kubernetes with
a read-only root filesystem and `HEADROOM_WORKSPACE_DIR` on a writable
volume, running the plugin at v0.40.0 from a fork tag, with
`advancement-key-store.ts` identical to a9e9ea6e.
- Exact command / steps: one script drives both stores through the first
three cases, and the last compares production logs:
- commit 300 accepted turns, each carrying 100 KB of tool output, and
time commits #1, #100 and #300;
- commit once with a `commit-log.json.lock` left behind, as a killed
process would leave it;
- commit with the workspace's parent directory not writable (mode 0555);
- in production, compare one gateway's OpenClaw logs over a 12-hour
night on v0.40.0's store with its first 3 hours on this store.
- Observed result: the new store stays small and fast and ignores a
leftover lock. In production, no commit failed and no turn fell back to
legacy.

  | | Old store | New store |
  |---|---|---|
  | Store on disk after 300 turns | 30.0 MB | 0.02 MB |
| Commit #1 / #100 / #300 | 3.3 / 14.3 / 40.4 ms | 2.6 / 0.1 / 0.2 ms |
| Leftover lock file from a crashed process | fails after 60.0 s |
committed immediately |
| Workspace not writable | `EACCES: permission denied, mkdir
'…/.headroom'` | same error, plus which setting fixes it |

In production, v0.40.0's store failed all 53 commits overnight with the
`remains queued` line above, and all 8 later turns of a session fell
back to legacy. On this store, the same gateway made 38 commits in its
first 3 hours, none failed, and no turn fell back. The 5 later turns we
could trace to their session all ran on headroom. The 38 markers take
296 KiB on disk, shard directories included.
- Not tested: the two permission-bit tests (a read-only directory, an
unwritable migration record) on Windows or as root, where they are
skipped, since permission bits don't restrict either. Our production
build predates the last three commits, which change only how a legacy
log is migrated and how a blocked path is reported, and that deployment
never had a legacy log.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: the commit store's on-disk format (an
existing log is migrated on first use). The default location is
unchanged.
- Kill switch / disable path: none needed. Pointing `commitLogPath` or
`HEADROOM_WORKSPACE_DIR` elsewhere starts a fresh store.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert. The old log stays where it was, so an older
version still finds every key it wrote. Keys committed after the upgrade
live only in markers, so an older version could re-accept a turn that
was mid-retry during the downgrade.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (on Node
24; see Additional Notes)
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- A version older than this one, running at the same moment against the
same workspace, keeps writing the old file, and this store migrates
whatever it adds. The older version doesn't read markers, though, so it
could still accept a key only this version committed. That's the same
mixed-version caveat any format change has.
- Unrelated to this change: the cross-process tests run `node
--experimental-transform-types`, which Node 26 no longer accepts (`bad
option`). They pass on Node 24, which CI uses.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4212"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-03 22:46:13 -06:00

151 lines
4.9 KiB
JSON

{
"id": "headroom",
"kind": "context-engine",
"uiHints": {
"proxyUrl": {
"label": "Proxy URL",
"help": "Optional. URL Headroom proxy (example: http://127.0.0.1:8787 or https://headroom.example.com). Configured URLs probe-gated before provider routing. Auto-start opt-in only works local addresses."
},
"proxyPort": {
"label": "Proxy Port",
"help": "Default port used for auto-detect/auto-start when proxyUrl is not set (default: 8787)."
},
"pythonPath": {
"label": "Python Path",
"help": "Optional explicit python executable for python fallback launcher (for example: python, python3, py, or full path)."
},
"retryMaxAttempts": {
"label": "Retry Max Attempts",
"help": "Optional maximum number of upstream retry attempts for connection/read/5xx failures when the plugin auto-starts a local Headroom proxy. Lower values fail faster for interactive chat."
},
"connectTimeoutSeconds": {
"label": "Connect Timeout Seconds",
"help": "Optional upstream connection timeout for the auto-started local Headroom proxy. Lower values surface network failures sooner."
},
"requestTimeoutMs": {
"label": "Compression Request Timeout",
"help": "Maximum milliseconds to wait for one compression request before returning the original messages."
},
"minContextChars": {
"label": "Minimum Context Size",
"help": "Skip the proxy for smaller contexts that cannot produce meaningful savings. Set to 0 to disable."
},
"circuitBreakerThreshold": {
"label": "Circuit Breaker Threshold",
"help": "Consecutive compression failures before temporarily bypassing Headroom."
},
"circuitBreakerCooldownMs": {
"label": "Circuit Breaker Cooldown",
"help": "Milliseconds to bypass compression after the circuit breaker opens."
},
"routeCodexViaProxy": {
"label": "Route OpenAI Codex Via Headroom",
"help": "When enabled, OpenClaw will use the active Headroom proxy as the in-memory upstream base URL for the built-in openai-codex provider so provider traffic flows through Headroom."
},
"gatewayProviderIds": {
"label": "Gateway Provider IDs",
"help": "Optional list of OpenClaw provider ids to route through the active Headroom proxy in memory. Friendly aliases codex, claude, copilot, and gemini are also accepted. When set, this overrides the default openai-codex-only routing."
},
"announceCompression": {
"label": "Announce Compression In System Prompt",
"help": "When enabled, a static, cache-stable notice is added to the system prompt the first time a session compresses more than 100 tokens. Disable to guarantee the plugin never modifies the system prompt."
},
"commitLogPath": {
"label": "Turn Commit Log Path",
"help": "Where to record committed turn advancements (the markers go in a sibling directory). Must be writable. Defaults to $HEADROOM_WORKSPACE_DIR/openclaw/commit-log.json, else ~/.headroom/openclaw/commit-log.json."
}
},
"configSchema": {
"type": "object",
"additionalProperties": false,
"properties": {
"enabled": {
"type": "boolean"
},
"proxyUrl": {
"type": "string",
"pattern": "^https?:\\/\\/.+(:\\d+)?\\/?$"
},
"proxyPort": {
"type": "integer",
"minimum": 1,
"maximum": 65535,
"default": 8787
},
"pythonPath": {
"type": "string"
},
"autoStart": {
"type": "boolean",
"default": false
},
"startupTimeoutMs": {
"type": "integer",
"minimum": 1000,
"maximum": 120000,
"default": 20000
},
"retryMaxAttempts": {
"type": "integer",
"minimum": 1
},
"connectTimeoutSeconds": {
"type": "integer",
"minimum": 1
},
"requestTimeoutMs": {
"type": "integer",
"minimum": 1,
"default": 2000
},
"minContextChars": {
"type": "integer",
"minimum": 0,
"default": 800
},
"circuitBreakerThreshold": {
"type": "integer",
"minimum": 1,
"default": 3
},
"circuitBreakerCooldownMs": {
"type": "integer",
"minimum": 1,
"default": 60000
},
"routeCodexViaProxy": {
"type": "boolean",
"default": true
},
"gatewayProviderIds": {
"type": "array",
"items": {
"type": "string"
},
"default": []
},
"announceCompression": {
"type": "boolean",
"default": true
},
"commitLogPath": {
"type": "string",
"minLength": 1
}
}
},
"capabilities": {
"network": {
"allow": [
"http://*:*",
"https://*:*"
]
}
},
"contracts": {
"tools": [
"headroom_retrieve"
]
}
}