mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1372f95524 |
fix(openclaw): let the circuit breaker open on proxy failures (#4755)
## Description `assemble()` in the openclaw plugin passed `fallback: true` to the SDK's `compress()`. In `sdk/typescript/src/client.ts`, that flag makes a connection error, a timeout or a 5xx retry once and then **resolve** to `makeFallbackResult(messages)` — `tokensBefore: 0`, `tokensSaved: 0`, `compressed: false` — instead of throwing. That result never reached the `catch` that calls `tripCircuit()`. Worse, `assemble()` treated it as a successful no-op and called `resetCircuit()`, so the error counter went back to zero on every turn. Against a proxy that accepts connections but never answers, the circuit breaker could never open: every `assemble()` paid up to `2 × requestTimeoutMs` and then bypassed compression anyway. Closes #4738 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Pass `fallback: false` on the plugin's `compress()` call, so proxy failures throw into the existing `catch` → `tripCircuit()`. Graceful degradation is unchanged: that `catch` already returns the original messages. - Treat a zero `tokensBefore` on **nonempty** input as a failure too, so a result that resolved empty cannot keep the breaker closed either. Empty input is exempt, because the proxy legitimately answers `tokens_before: 0` for an empty `messages` array — caught in review, see the `minContextChars: 0` case below. - `README.md`: the post-cool-down turn does not call `/health`; nothing in the engine ever did. It re-probes by attempting a real compression. Fixed, and the threshold row now says which failures count. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, this is the TypeScript plugin; ran `vitest` instead - [ ] Linting passes (`ruff check .`) — N/A, no Python changed - [ ] Type checking passes (`mypy headroom`) — N/A, no Python changed - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ npx vitest run test/engine.test.ts Test Files 1 passed (1) Tests 41 passed (41) $ npm run typecheck > node node_modules/@typescript/native/bin/tsc --noEmit (clean) $ npm run build ESM Build success in 18ms DTS Build success in 515ms ``` The three new tests each fail on the unpatched tree (verified by stashing only `src/engine.ts` and re-running): ```text × passes fallback: false so proxy failures reach the circuit breaker × opens the circuit on zero-token results, even if the SDK swallows the error × does not count an empty context as a failure ``` Full suite: 158 passed, 2 failed — both pre-existing and unrelated, in `test/advancement-key-store.test.ts`, caused by this machine's Node rejecting `--experimental-transform-types` in the child-process fixture (`node: bad option: --experimental-transform-types`). They fail identically on `upstream/main` without this patch. ## Real Behavior Proof - Environment: Windows 11, Node v26.10.0, `plugins/openclaw` built with `npm run build`, in-tree `headroom-ai@0.42.0`. Stub proxy is a `node:http` server that accepts the connection and never responds. Engine constructed with `requestTimeoutMs: 500`, `circuitBreakerThreshold: 3`, `circuitBreakerCooldownMs: 60000`, `minContextChars: 0`, driven through five `assemble()` calls with a 2000-char message. - Exact command / steps: `npm run build` in `plugins/openclaw`, then `node proof-circuit.mjs`, which starts the hung proxy on an ephemeral port, points the engine at it, and makes five sequential `assemble()` calls. Run once on the unpatched build and once on the patched build. - Observed result: before the patch the breaker never opened and every call paid the full timeout. After the patch it opens after the third failure and calls 4 and 5 skip the proxy entirely (0 ms, 0 requests). Messages returned are the originals in every case, so model output is unchanged; only the latency is removed. Before the patch: ```text call 1: ms=1004 requestsSeenByProxy=2 totalProxyRequests=2 call 2: ms=1001 requestsSeenByProxy=2 totalProxyRequests=4 call 3: ms=1003 requestsSeenByProxy=2 totalProxyRequests=6 call 4: ms=1001 requestsSeenByProxy=2 totalProxyRequests=8 call 5: ms=1002 requestsSeenByProxy=2 totalProxyRequests=10 circuit-opened=false ``` After the patch: ```text call 1: ms=1004 requestsSeenByProxy=2 totalProxyRequests=2 messagesAreOriginal=true call 2: ms=1002 requestsSeenByProxy=2 totalProxyRequests=4 messagesAreOriginal=true call 3: ms=1002 requestsSeenByProxy=2 totalProxyRequests=6 messagesAreOriginal=true call 4: ms=0 requestsSeenByProxy=0 totalProxyRequests=6 messagesAreOriginal=true call 5: ms=0 requestsSeenByProxy=0 totalProxyRequests=6 messagesAreOriginal=true circuit-opened=true warn: [headroom] Circuit breaker opened after 3 errors (last: HeadroomConnectionError: Failed to connect to Headroom at http://127.0.0.1:53545: TimeoutError: The operation was aborted due to timeout); bypassing compression for 60000ms warn: [headroom] Circuit open — using uncompressed messages warn: [headroom] Circuit open — using uncompressed messages ``` - Not tested: a live OpenClaw host with a real provider (the engine needs a host to drive, so I drove `assemble()` directly against the built plugin), and the published `headroom-ai` beyond `0.42.0`. `fallback` has been in the SDK's `CompressOptions` since the first TS release, so the flag resolves against the plugin's `^0.40.0` range too. ## Runtime Rollout Safety - Rollout-managed feature(s): none — this is a plugin-local circuit breaker inside `headroom-openclaw`, with no rollout flag or channel gating involved. - Minimum rollout channel: N/A, no gating is introduced. - Stable/default behavior changed: yes, deliberately. The circuit breaker now opens on proxy failures instead of never opening, so an unhealthy proxy causes compression to be skipped for `circuitBreakerCooldownMs` (default 60 s) rather than being attempted on every turn. Messages are returned unchanged either way, so model output is unaffected; the change removes repeated timeout latency. - Kill switch / disable path: `enabled: false` disables the plugin entirely; `circuitBreakerThreshold` can be raised to make the breaker effectively unreachable; `circuitBreakerCooldownMs` bounds how long each bypass lasts. - Unsafe override required: no. - Qualification impact: none — no provider, transport or config surface changes, and no new dependency. - Rollback path: revert the commit. The prior behavior was a single-line `fallback: true`, and nothing here migrates or persists state, so reverting restores the old behavior exactly. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Review caught one defect in my first attempt, now fixed and covered by a test: with `minContextChars: 0`, `assemble()` forwards empty `messages: []` to the proxy, which answers `200` with `tokens_before: 0`. Counting that as a failure meant a handful of empty-context turns could open the breaker and then skip compression for a healthy proxy. The zero-token check is now scoped to nonempty input. Found with AI assistance (opencode) operated by me. |
||
|
|
24acbe8f32 |
fix(openclaw): record turn advancements as one marker file per key (#4212)
## Description
OpenClaw 2026.9.9 started calling the context engine's `commitTurn`
after every accepted turn. While a commit is pending, it runs every
later turn of that session on its legacy context path, which means no
Headroom compression. The plugin's turn-commit store (from #3442) has
three ways to leave a commit pending, and this PR removes all three.
Each committed key becomes one small marker file, created with exclusive
creation (`O_CREAT | O_EXCL`). There's no lock and no shared file to
rewrite, and only the key is stored.
**What we saw.** We run the plugin on Kubernetes with a read-only root
filesystem. After upgrading OpenClaw to 2026.9.9, every turn logged:
```
[context-engine] durable turn advancement remains queued: <key>: ENOENT: no such file or directory, mkdir '/home/openclaw/.headroom'
```
On the session's next turn, OpenClaw retries the pending commit before
assembling context. When the retry fails, it degrades that turn to the
legacy engine (`degradeBeforeStart("pending durable turn advancement
could not be completed before the next turn")` in OpenClaw's
`context-engine-turn-attempt`). A failed commit only increments
`attempt_count`, so the row stays pending and every later turn of the
session degrades too.
**The three problems in `advancement-key-store.ts`:**
1. **Read-only home.** The default
`~/.headroom/openclaw/commit-log.json` can't be created on a read-only
root. Node reports the failed `mkdir` as `ENOENT`, which reads like a
bug rather than a setting. `commitLogPath` exists on the engine but not
in the manifest's `configSchema` (`additionalProperties: false`), so
`HEADROOM_WORKSPACE_DIR` is the only way to move it.
2. **Unbounded growth.** Every accepted turn's `messages` (OpenClaw caps
a turn at 8 MiB) are kept forever in one JSON file. Each commit
re-reads, parses, copies and rewrites the whole file on the gateway's
event loop, so the file and each commit's latency grow with every turn
ever committed. Nothing reads the messages back (`get` is test-only),
and OpenClaw's transcript already holds them.
3. **A lock nothing reclaims.** By design, the lock is never
force-broken. A gateway killed mid-commit (an OOM kill, a node drain)
leaves `commit-log.json.lock` behind, and every later commit waits 60 s
and then fails until someone deletes it. With the retry above, every
turn then starts 60 s late before degrading.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **One marker per committed key**, at
`commit-log.d/<sha256[0:2]>/<sha256(key)>` next to the old log path,
created with `open(path, "wx")`. The first creator in any process gets
`"committed"`; every later caller gets `EEXIST`, i.e. `"duplicate"`.
Only that exclusive create can mean `"duplicate"`, and only when what
exists at the path is a marker file. An `EEXIST` from creating the shard
directory (a file where the directory belongs), or anything but a file
at the marker's path, is a storage error. The marker holds
`{advancementKey, committedAt}`: 99 bytes for OpenClaw's UUID keys.
- **Only the key is recorded.** For this engine, the advancement *is*
the key: `assemble()` works from the messages OpenClaw passes in, and
the plugin keeps no turn state. The contract asks for "one atomic,
idempotent write keyed by `advancementKey`" (OpenClaw
`docs/concepts/context-engine.md`), and creating the marker is that
write.
- **Keys are still never evicted**, for the reason #3442 gives.
- **Migration.** Before every commit and lookup, a store checks for a
`commit-log.json` from an earlier version. With no log there, that check
is one failed `open()`. If a log is there and it's a version the store
hasn't migrated, the store moves its keys into markers and records which
version that was (size, mtime, inode) in
`commit-log.d/migrated-legacy-log.json`. That record only saves work, so
a store that can't write it carries on and reads the log again later.
The log's version and its contents come from one open handle. The log
itself is never moved or changed, so an older version still running
against the same workspace keeps its own record, and nothing has to
write to the log's directory (which may be read-only, with a writable
marker volume mounted beneath it). An unchanged log is skipped by its
metadata alone, even if it can no longer be read. A changed one is
migrated again, so a key an older version commits at any moment, even
while a store is migrating the log, is never accepted as new. Only keys
the store hasn't seen get markers, so a rewrite costs only the keys it
adds. Both the object and the older array format are read. Several
processes can migrate at once safely, since creating an existing marker
is a no-op. A corrupt log still fails loudly rather than being read as
empty, and the store recovers once it's fixed.
- **An actionable error.** When the marker directory can't be created,
the message says what to do: `Cannot record turn advancements under
<dir>: <error>. Point HEADROOM_WORKSPACE_DIR (or the plugin's
commitLogPath) at a writable directory.` A lookup in a store whose path
a file blocks fails the same way on every platform, although Windows
reports such a path as `ENOENT`, which is also what a missing marker
gives.
- **`commitLogPath` is settable.** It's declared in
`openclaw.plugin.json`, with a UI hint and a README row.
- **Tests.** The store suite is rewritten for the new design. It keeps
the existing guarantees: committed, then duplicate; survives a restart;
never evicts across 600 keys; exactly one winner in-process and across
two OS processes; a failed write is retried, not remembered. New tests
cover:
- a commit stores one small marker holding only the key;
- committing more keys never rewrites an earlier marker (same inode and
mtime);
- a leftover `.lock` from a crashed process doesn't block a commit;
- a file where the marker directory or a shard directory belongs, and a
directory where a marker belongs: each is an error, never `"duplicate"`,
and the same key commits once the path is cleared;
- a file blocking the store's directory or its marker directory,
reported the way Windows reports it (`ENOENT`, not `ENOTDIR`): a lookup
is a storage error, not an absent key;
- both legacy formats, including two stores migrating concurrently;
- a legacy log in a read-only directory: commits go through; a restart
doesn't read the unchanged log again; a migrated log that can no longer
be read doesn't block commits; and once the directory is writable, a key
an older version commits the old way (temp file + rename) is migrated
before a running store's next commit;
- a key an older version commits while a store is migrating the log: the
store's next lookup finds it, and its next commit, like a new store's,
is `"duplicate"`;
- an older version rewriting the log: only the key it added gets a
marker;
- a migration record that can't be written: commits go through, in that
store and in one started later;
- a corrupt legacy log.
In `engine.test.ts`, a commit carrying 1 MB of tool output stores under
1 KB. The lock-holder fixture is removed, since there is no lock.
## Testing
- [ ] Unit tests pass (`pytest`) (n/a, TypeScript plugin only)
- [ ] Linting passes (`ruff check .`) (n/a)
- [ ] Type checking passes (`mypy headroom`) (n/a; `npm run typecheck`
passes)
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ cd plugins/openclaw && npm run typecheck # exit 0
$ npx vitest run # Node v24.21.0, macOS
Test Files 9 passed (9)
Tests 139 passed (139)
$ npx vitest run # Node v22.22.0, GitHub Actions windows-latest
Test Files 9 passed (9)
Tests 137 passed | 2 skipped (139)
```
## Real Behavior Proof
- Environment: macOS, Node v24.21.0, for the benchmark below: "old" is
`advancement-key-store.ts` from `main`, "new" is this PR, and the same
script drives both. In production: OpenClaw 2026.9.9 on Kubernetes with
a read-only root filesystem and `HEADROOM_WORKSPACE_DIR` on a writable
volume, running the plugin at v0.40.0 from a fork tag, with
`advancement-key-store.ts` identical to
|
||
|
|
33f204406c |
fix(openclaw/release): ship responsive npm integrations (#1458)
## Description Fixes #977 and #2353. The v0.35.0 release built all three JavaScript packages successfully, but npm rejected every publish with ENEEDAUTH. Because each publish step used continue-on-error, the publish-npm job still concluded success. npm therefore remains on headroom-ai 0.22.4 and headroom-openclaw 0.1.0, while headroom-opencode is absent. The checked-in OpenClaw source had already corrected the old strict-schema mismatch and added a circuit breaker, but its assemble hook still called compression for every tiny conversation and allowed a 30-second default wait. This PR completes both the runtime correction and the delivery path instead of merely exposing one failure. ## Changes ### OpenClaw runtime - Skip proxy compression below an 800-character context floor (roughly 200 tokens), avoiding synchronous HTTP work on short conversations that cannot produce meaningful savings. - Measure actual content, including tool-call arguments, so a single large tool result remains eligible. - Allow operators to disable the floor with minContextChars=0. - Lower the interactive request ceiling from 30 seconds to 2 seconds. - Pass that timeout into the SDK, whose AbortSignal cancels the HTTP request, instead of racing a promise while the fetch continues in the background. - Preserve circuit-breaker fail-open behavior and document/schema the new defaults. ### npm release integrity - Authenticate with npm before publishing any package, so a missing or invalid NPM_TOKEN fails before a partial release begins. - Make SDK, OpenClaw, and OpenCode npm publish failures fatal when npm publishing is enabled. - Preserve NPM_SKIP=true as the explicit opt-out. - Make retries safe after partial publication by skipping only the exact package version already present, then continuing to missing packages. - Remove the false-success notice path. ## Existing #977 corrections verified - The current OpenClaw schema declares startupTimeoutMs, routeCodexViaProxy, gatewayProviderIds, request timeout, and circuit-breaker fields written/used by the installer and runtime. - wrap defaults to the real published package name, headroom-openclaw. - unwrap removes managed values while preserving unrelated user configuration. - The coding savings profile uses protect_recent=0 paired with type-based recent-read protection, avoiding the old blanket short-session protection without weakening tool-result fidelity. ## Testing - 80 OpenClaw plugin tests pass. - TypeScript typecheck and production build pass. - 123 OpenClaw wrap/provider, savings-profile, compress endpoint, and MCP contract tests pass. - 47 release-workflow and MCP contract tests pass. - The npm release asset builder installed/imported and verified the actual 0.35.1 SDK and OpenClaw tarballs. - Workflow YAML parses; Ruff, format, and git diff checks pass. ## Required repository action A maintainer must replace or restore the npm publishing credential in NPM_TOKEN, then rerun the v0.35.0 release workflow (or publish the next version). This PR makes that failure visible and safely rerunnable; repository code cannot provision the external npm credential. Co-authored with @liuyongcheng0000, whose original PR identified the masked npm failures. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/1458" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Jerrett Davis <mxjerrett@gmail.com> |
||
|
|
f8851878a4 |
fix(plugins/openclaw): keep compression system-prompt notice cache-stable (#3811)
## Description `HeadroomContextEngine.assemble()` in the OpenClaw plugin returns a `systemPromptAddition` string that OpenClaw prepends to the system prompt every turn. It interpolated the per-turn `tokensSaved` count, so the system prompt — the start of the provider's cached prefix — changed on every compressing turn, defeating prompt caching. Reported cost: a turn saving 267 tokens caused a ~93K-token cache miss on Bedrock/Claude Opus 5.5, roughly 4,000x the tokens the compression saved. Refs #3809 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `plugins/openclaw/src/engine.ts`: replaced the interpolated `${result.tokensSaved}` notice with a fixed constant (`HEADROOM_COMPRESSION_NOTICE`), so any turn that includes it is byte-identical to any other. Added a `hasAnnouncedCompression` instance flag so the notice, once shown (tokensSaved > 100), stays present and unchanged on later turns even when a given turn has nothing new to compress - `plugins/openclaw/src/engine.ts`: added `announceCompression` (boolean, default `true`) to `HeadroomEngineConfig` to disable the notice entirely. - `plugins/openclaw/openclaw.plugin.json`: added `announceCompression` to `configSchema.properties` (required — schema has `additionalProperties: false`) and a matching `uiHints` entry. - `plugins/openclaw/README.md`: documented the new option in the configuration table. - `plugins/openclaw/test/engine.test.ts`: 5 new tests, static notice with no interpolated count, byte-identical notices across differing `tokensSaved`, notice staying present on a turn with nothing to compress (the flap-prevention regression case), `announceCompression: false` opt-out, and the existing below-threshold no-op case. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ npx vitest run test/engine.test.ts RUN v4.1.10 /Users/sarah_ling/Documents/GitHub/headroom/plugins/openclaw Test Files 1 passed (1) Tests 27 passed (27) Start at 18:17:43 Duration 527ms $ npx vitest run Test Files 1 failed | 6 passed (7) Tests 3 failed | 104 passed (107) # The 3 failures are in test/advancement-key-store.test.ts, unrelated to this change ``` ## Real Behavior Proof - Environment: local vitest run against `plugins/openclaw` (mocked `compress()`/proxy — no live OpenClaw host or real Bedrock/Anthropic request). - Exact command / steps: added a temporary test that mocks a 6-turn session mirroring the reported scenario (tokensSaved: 0, 267, 0, 9000, 0, 50) and recorded `assemble()`'s `systemPromptAddition` per turn. - Observed result: notice absent on turn 1, appears static and identical on turns 2–6 (including turns 3/5 with nothing to compress, and turn 6's below-threshold 50-token save) — 1 total transition across 6 turns, versus the old behavior's distinct string on every compressing turn. - Not tested: no live OpenClaw + real provider run to directly observe the Bedrock cache-read/cache-write fields improve, as the original issue's report did. The fix is verified at the unit level (the exact value the plugin hands to OpenClaw), not against a real provider cache. ## Runtime Rollout Safety - Rollout-managed feature(s): None - Minimum rollout channel: N/A - Stable/default behavior changed: yes, the notice text changes (no longer reports a live token count) and stays present for the rest of the session instead of tracking that turn's compression state. Default (`announceCompression` unset) still shows the notice, matching prior default-on behavior. - Kill switch / disable path: set `announceCompression: false` in the plugin config to fully disable the notice. - Unsafe override required: None - Qualification impact: None - Rollback path: Revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3811" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
a28dd4eab7 |
fix(openclaw): declare transcriptSemantics and implement commitTurn (#3442)
## Description Adds OpenClaw transcript-semantics declarations and an accepted-turn log for admitted-turn advancement. Fixes #3441. Messages and advancement keys are written together in `HEADROOM_WORKSPACE_DIR/openclaw/commit-log.json` (default workspace: `~/.headroom`). Each commit re-reads the log inside an exclusive sibling-file lock and replaces the log atomically. Keys are retained without eviction; prior key-only logs are recognized for migration. Failed writes do not create an in-memory phantom duplicate. ## Locking and recovery The lock serializes processes sharing the workspace and is never reclaimed merely because it is old. A waiting caller times out after 60 seconds instead of committing without ownership. Release checks the acquisition token. A crashed holder can leave an orphan `.lock` file. Recovery is operational: stop writers using that workspace, establish that the holder is no longer running, remove the orphan lock, then resume. Do not delete a live writer's lock. Automatic crash recovery and power-loss durability are not claimed. ## Testing - 99 plugin tests pass, including distinct-key and same-key child-process commits and an aged live-lock regression. - TypeScript checking passes. - The aged-lock regression fails against the prior age-reclaim implementation and passes against the current source. Test cleanup releases the child even on assertion failure. - Current-head CI is pending. ## Real Behavior Proof - Environment: macOS, two real Node child processes importing `DurableAdvancementKeyStore`. - Exact command / steps: A acquires the lock and reads the empty log; pause A while alive for 31 seconds. Start B committing another key and verify it remains pending, then resume A. - Observed result: B does not finish while A holds the lock. After release, both return `committed` and both message payloads remain in the log. The prior implementation acknowledged both but lost one turn. - Not tested: automatic recovery from crashed holders (requires the operator procedure above); power-loss durability. ## Runtime Rollout Safety - Rollout-managed feature(s): OpenClaw transcript-semantics declaration. - Minimum rollout channel: Host capability negotiation. - Stable/default behavior changed: Accepted messages and keys persist in a workspace-shared log. The log grows without eviction, and an orphan lock requires operator recovery. - Kill switch / disable path: Revert the plugin change or use the host's legacy context-engine path. - Unsafe override required: No. - Qualification impact: Admission uses atomic/idempotent commit behavior; contention fails closed. - Rollback path: Revert the plugin change. Preserve the log if retry history must remain available. ## Review Readiness - [x] Prior message persistence, retry, cross-process, and stale-lock findings resolved. - [x] Local tests and type checking pass. - [ ] Current-head CI complete. - Final human review is required. --------- Signed-off-by: Radhakrishnan P <gingeekrishna@gmail.com> Co-authored-by: Jerrett Davis <mxjerrett@gmail.com> |
||
|
|
eb4da64774 |
fix(openclaw): delegate native compaction (#2304)
## Description
Fix the OpenClaw context engine claiming ownership of persistent
compaction while its `compact()` implementation performed no compaction.
Previously, `ownsCompaction` was `true`, which disabled OpenClaw's
built-in automatic compaction. However, `compact()` did not read or
rewrite
the session transcript and still returned `{ ok: true, compacted: true
}`.
Headroom now declares that it does not own persistent compaction and
delegates `/compact` and overflow-recovery requests through OpenClaw's
official `delegateCompactionToRuntime(...)` API. Headroom continues to
perform zero-cost compression while assembling model input.
OpenClaw's context-engine contract requires non-owning engines to
delegate `compact()` explicitly; setting `ownsCompaction: false` alone
does
not restore manual or overflow-recovery compaction:
https://docs.openclaw.ai/concepts/context-engine#ownscompaction
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Changed `HeadroomContextEngine.info.ownsCompaction` from `true` to
`false`.
- Replaced the successful no-op `compact()` implementation with OpenClaw
native-runtime delegation.
- Added a small runtime adapter that dynamically loads
`openclaw/plugin-sdk/core` without bundling the optional OpenClaw peer
dependency.
- Return the real delegated compaction result instead of manufacturing a
successful result.
- Increment compaction statistics only when OpenClaw reports that
compaction occurred.
- Propagate native compaction failures so OpenClaw can apply its normal
failure-isolation behavior.
- Removed stale claims that RollingWindow drops messages during Headroom
compression.
- Updated the OpenClaw README to distinguish input compression from
persistent compaction.
- Added an Unreleased changelog entry.
- Added regression coverage for delegated success, delegated no-op, and
delegated failure behavior.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cd plugins/openclaw
$ npx vitest run test/engine.test.ts
Test Files 1 passed (1)
Tests 13 passed (13)
$ npm run typecheck
> tsc --noEmit
Passed
$ npm run build
ESM Build success
DTS Build success
$ npm pack --dry-run
npm notice total files: 13
npm notice package size: 35.2 kB
$ uv run --extra dev --frozen ruff check .
All checks passed!
$ uv run --extra dev --frozen mypy headroom --ignore-missing-imports
$ uv run --extra dev --frozen pytest
9360 passed, 565 skipped, 4 failed
- Anthropic compaction transform ordering
- macOS AF_UNIX temporary socket path length
- Dashboard cached-stats expectation
- Content-router log fallback expectation
The repository-wide format check also flags pre-existing formatting in the untouched `headroom/proxy/handlers/anthropic.py`.
The complete OpenClaw plugin suite reports:
```text
Test Files 1 failed | 5 passed (6)
Tests 1 failed | 77 passed (78)
```
The remaining failure expects `sh -lc` while the existing implementation uses `sh -c`. The identical failure was reproduced in a detached,
untouched `origin/main` worktree, where it reported `1 failed | 74 passed`.
## Real Behavior Proof
- Environment: macOS 26.5.1 on arm64, Node.js v24.14.1, npm 11.11.0.
- Exact command / steps: Ran the focused Vitest suite, TypeScript type-check, production build, and package dry run. Inspected the emitted
`dist/index.js` to verify that it contains `ownsCompaction: false`, preserves the runtime import of `openclaw/plugin-sdk/core`, and calls
`delegateCompactionToRuntime(params)` from `compact()`.
- Observed result: A delegated successful result containing `tokensBefore: 20000` and `tokensAfter: 8000` was returned unchanged and
incremented the compaction counter. A delegated `{ compacted: false }` result did not increment the counter. A rejected native compaction
propagated the original error and did not report or count a successful compaction. Headroom's `compress()` was not invoked as a substitute for
persistent compaction.
- Not tested: A live installed OpenClaw gateway, an end-to-end `/compact` command against a persisted production transcript, or provider-
triggered overflow recovery. Those paths enter the same official OpenClaw delegation API exercised by the engine boundary.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did not edit CHANGELOG.md; release notes are generated automatically
## Additional Notes
The complete-suite checkbox remains unchecked because of the unrelated baseline failures documented above. The focused OpenClaw compaction
tests, TypeScript type-check, production build, Ruff lint, and mypy all pass.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
6576ef639c |
fix(openclaw-plugin): circuit breaker + per-request timeout for proxy resilience (#639)
## Description This change adds bounded timeout and circuit-breaker behavior so OpenClaw can degrade safely when Headroom or the upstream stream stalls, while returning structured proxy errors instead of hanging. Closes #638 by improving OpenClaw/proxy resilience when the Headroom proxy stalls or Anthropic resets a stream. The PR adds proxy-side handling for `httpx.RemoteProtocolError`, returns structured 502 responses for otherwise unhandled proxy middleware errors, and adds OpenClaw plugin timeout/circuit-breaker fallback behavior. ## Type of Change - [x] Bug fix - [ ] New feature - [x] Documentation - [ ] Refactor - [x] Tests only ## Changes Made - Added OpenClaw plugin per-request compression timeout and circuit breaker fallback. - Cleared timeout timers after successful or failed compression so successful calls do not leave pending timers. - Added a focused Vitest regression for timeout cleanup. - Added `contracts.tools` for `headroom_retrieve` without whole-file manifest reformatting. - Added proxy handling for mid-stream `httpx.RemoteProtocolError` and structured 502 fallback behavior. - Documented the new OpenClaw resilience configuration fields. ## Testing - [x] Unit tests - [x] Integration-style proxy tests - [x] Typecheck/build - [ ] Manual testing ### Test Output ```text cd plugins/openclaw && npm test Test Files 6 passed (6), Tests 55 passed (55) cd plugins/openclaw && npm run typecheck passed cd plugins/openclaw && npm run build Build success UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --with pytest --with pytest-asyncio --with fastapi --with httpx --with uvicorn --with h2 python -m pytest tests/test_proxy_streaming_resilience.py -q 24 passed in 2.16s ``` ## Real Behavior Proof - Environment: Windows 11, Node/npm from local plugin worktree, Python 3.13.3, focused local worktree for PR #639. - Exact command / steps: Installed plugin dependencies, ran OpenClaw plugin tests/typecheck/build, and ran the proxy streaming resilience suite with required async/FastAPI/httpx extras. - Observed result: Plugin tests, typecheck, build, and proxy resilience tests all passed. - Not tested: Live OpenClaw gateway session in this pass; original reporter previously verified patched files in a container and OpenClaw degraded/recovered cleanly. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Sergei Baikin <sergei.baikin@fotograf.de> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
|
6d116b15f1 |
Harden OpenClaw plugin proxy routing (#1074)
## Description Hardens the bundled OpenClaw plugin so configured proxy routing is fail-closed and `autoStart` is opt-in. Closes: N/A This follow-up is intentionally separate from the ContentRouter cache fix because it changes plugin/gateway behavior rather than core compression routing. The plugin should not mutate upstream provider routing unless a configured proxy URL is reachable and looks like Headroom. It should also avoid unhandled startup promise rejections when proxy startup is fire-and-forget. Why this shape: - `autoStart: false` by default matches deployments where Headroom is supervised externally, for example by systemd. The plugin should not silently start or assume ownership of a proxy unless the operator opted in. - Provider routing is fail-closed: a configured URL must first respond like Headroom, not merely expose a generic liveness endpoint. This prevents accidentally routing model traffic through the wrong local service. - `/readyz` is treated as liveness, not identity. Identity comes from Headroom-shaped stats endpoints (`/v1/retrieve/stats` or `/stats`) because those are harder for unrelated services to satisfy by accident. - Startup remains asynchronous, but errors are captured and exposed instead of becoming unhandled promise rejections. - This is a separate PR because the core cache fix is about compression correctness, while this patch is about integration safety around OpenClaw gateway routing. ## Type of Change - [x] Bug fix (non-breaking change fixes issue) - [ ] New feature (non-breaking change adds functionality) - [ ] Breaking change (fix or feature would cause existing functionality change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Make proxy `autoStart` opt-in (`default: false`). - Probe configured `proxyUrl` before applying provider routing. - Treat `/readyz` as liveness only; require Headroom-shaped `/v1/retrieve/stats` or `/stats` for identity. - Observe fire-and-forget startup promise rejection and expose startup error for callers. - Isolate proxy-ready listener failures. - Keep provider routing deferred when no active/probed Headroom proxy exists. - Register retrieve tool with explicit `headroom_retrieve` name. - Extend plugin/unit tests for configured proxy failures, generic non-Headroom endpoints, path collisions, and routing behavior. Changed files: - `plugins/openclaw/README.md` - `plugins/openclaw/openclaw.plugin.json` - `plugins/openclaw/src/engine.ts` - `plugins/openclaw/src/plugin/index.ts` - `plugins/openclaw/src/proxy-manager.ts` - `plugins/openclaw/test/engine.test.ts` - `plugins/openclaw/test/gateway-config.test.ts` - `plugins/openclaw/test/plugin-runtime-routing.test.ts` - `plugins/openclaw/test/proxy-manager.test.ts` ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added new functionality - [x] Manual testing performed ### Test Output ```text $ npm test Test Files 6 passed (6) Tests 74 passed (74) $ npm run typecheck tsc --noEmit $ npm run build tsup && node prepare-dist.mjs Build success ``` ## Real Behavior Proof - Environment: local OpenClaw plugin package in the Headroom repo. - Exact command / steps: - Run plugin test suite. - Run TypeScript typecheck. - Run plugin build. - Observed result: - Tests passed: `74/74`. - Typecheck passed. - Build passed. - Not tested: - Full OpenClaw Gateway integration as part of this standalone PR prep. ## Review Readiness - [x] I performed self-review - [x] This PR ready for human review ## Checklist - [x] My code follows project's style guidelines - [x] I performed self-review my code - [ ] I commented my code, particularly in hard-to-understand areas - [x] I made corresponding changes documentation - [x] My changes generate no new warnings - [x] I added tests prove fix is effective or feature works - [x] New and existing unit tests pass locally my changes - [ ] I updated CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes Checklist items left unchecked intentionally: - No CHANGELOG update included. - No extra comments were needed beyond existing code structure. Co-authored-by: Björn-Christian Bönkost <bjoern@v2202603344248440850.hotsrv.de> |
||
|
|
37f32a8922 |
test(openclaw): cover branch routing paths
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
0da1a7b4de | Add OpenClaw unwrap flow and non-blocking proxy startup | ||
|
|
2edff08668 | fix(openclaw): normalize engine messages for discord | ||
|
|
5f075dee81 | fix(openclaw): preserve tool call linkage in converted history | ||
|
|
3e368f13c8 |
refactor(openclaw): DRY up duplicate code across plugin
- Extract shared defaultLogger constant, use in both ProxyManager and HeadroomContextEngine (was duplicated inline in both constructors) - Extract parseProxyUrl() helper, reused by normalizeAndValidateProxyUrl and withDefaultPort (eliminates redundant URL parsing) - Add test helpers: stubProbeSuccess/NonHeadroom/Unreachable to replace 6 identical inline fetch mock constructions - Remove duplicate probe tests from engine.test.ts (already covered in proxy-manager.test.ts) - Clean up unused probeHeadroomProxy import from engine.test.ts Net: -26 lines, same coverage (28 tests) |
||
|
|
3e11dc6316 | Remove skipped integration test registration | ||
|
|
205ec54df2 | Harden openclaw plugin proxy handling and metadata | ||
|
|
a41c7f0c89 |
OpenClaw plugin fixes, telemetry fix, cost tracker improvement, tokenBudget support
OpenClaw plugin: - Fix assistant content always returned as array (fixes flatMap crash) - Clean up debug logging, restore clean plugin entry point - Add test for text-only assistant round-trip - Set headroom-ai dependency to npm (not local path) Telemetry: - Switch from sb_publishable_ to JWT anon key (PostgREST needs JWT for RLS) - Switch from upsert to plain INSERT (anon role upsert was failing with 42501) - Verified: full payload with all JSONB columns writes successfully (201) Cost tracker: - Use output_buffer instead of hardcoded 500 for cost estimation in client.py - CostTracker counterfactual: value removed tokens at avg effective $/token (actual_spend / billed_tokens) instead of uncached list price — aligns savings with real billing mix (cache reads, writes, uncached) - Add debug log for /stats summary payload - Add 3 tests for new cost savings calculation TypeScript SDK: - Add tokenBudget param to compress(), HeadroomClient, CompressOptions - Proxy /v1/compress accepts optional token_budget to override model limit |
||
|
|
053c4a06af |
feat: add OpenClaw ContextEngine plugin (@headroom-ai/openclaw)
- HeadroomContextEngine: implements ContextEngine interface (bootstrap, ingest, assemble, compact, afterTurn, prepareSubagentSpawn, dispose) - ProxyManager: auto-detects running proxy or spawns one as child process, health checks, restart on crash, graceful shutdown - AgentMessage ↔ OpenAI format conversion (user, assistant/tool_use, toolResult) - headroom_retrieve CCR tool: agent can retrieve original uncompressed content - Plugin manifest (openclaw.plugin.json) with config schema - 11 tests (6 unit + 5 integration), all passing - Integration tested: 100 servers compressed, 3,735 tokens saved (82%) |