mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 12:05:34 +08:00
feat/https-compose-caddy
3781
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
6712d432c2 | Merge branch 'main' into feat/https-compose-caddy | ||
|
|
18067b1d19 |
fix(content-router): protect cache_control markers nested in tool_result content (#4898)
## Description
`ContentRouter` never rewrites a block that carries `cache_control`
(contract 1 in `_process_content_blocks`), but the guard only checks
`"cache_control" in block`. The Messages API also accepts the marker on
a `text` sub-block of a `tool_result`'s list content. That shape is not
seen as marked, so the `tool_result` is compressed and re-wrapped as
`[{"type": "text", "text": compressed}]`: the marker is dropped and the
bytes under it change, even for a tool result earlier in the
conversation. The cross-turn dedup `protected` check has the same gap.
`compress()` and `POST /v1/compress` return the router output as-is, so
callers get the rewritten block back.
This PR treats a `tool_result` whose list content carries a marker as
marked, the same shapes `count_cache_breakpoints` in
`headroom/proxy/helpers.py` already counts.
Fixes #4890
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/content_router.py`: new helper
`_carries_cache_control(block)`, true when the block has `cache_control`
or is a `tool_result` whose list content has a sub-block with
`cache_control`.
- Used in the per-block contract 1 guard of `_process_content_blocks`
and in the `protected` check of the cross-turn dedup. The contract 1
docstring now names the nested shape. The fresh-turn release
(`fresh_turn` and a `tool_result` in the final message) is unchanged.
- `tests/test_transforms_content_router.py`: the existing
`test_tool_result_cache_control_protected` is parametrized over
`block_marker` / `nested_marker`, plus
`test_marked_tool_result_in_history_survives_apply`, which goes through
`apply()` with the marked tool result in history.
- `tests/test_cross_turn_dedup.py`:
`test_dedup_never_rewrites_nested_cache_control_tool_result`, with an
unmarked control that must still be folded.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Source: one commit `916e8395` on `main` `5023c84e`. Runs below were
built from `git archive` of that commit in a `python:3.13-slim-bookworm`
image with `uv sync --frozen --extra proxy --extra dev`
(`headroom._core` built by maturin). ruff ran on the three touched
files, mypy on `headroom`. The new and changed tests were run 5 times in
a row. The module run covers the two touched test files plus
`test_content_router_detection_dedup.py`,
`test_responses_cross_turn_dedup.py` and
`test_openai_chat_dedup_recoverability.py`. For the reverse check,
`content_router.py` was replaced with the copy from `5023c84e` and the
tests kept.
### Test Output
```text
== RUFF CHECK RC 0
== RUFF FORMAT RC 0
== MYPY RC 0
# new and changed tests, run 1 and run 5 (runs 2-4 identical):
====================== 9 passed, 124 deselected in 0.97s =======================
====================== 9 passed, 124 deselected in 0.37s =======================
# related modules:
======================= 149 passed, 1 skipped in 12.32s ========================
# with content_router.py restored to
|
||
|
|
8da7e17e4e |
fix(kompress): honour toin_read_only in the Kompress CCR store (#4895)
## Description
Fixes #4801
The second TOIN write path of #4474. #4493 gated the router →
`SmartCrusher.crush()` path, but `store_kompress_in_ccr` still recorded
a TOIN observation unconditionally, so a Subscription request
(`CompressionPolicy.toin_read_only=True` — "never writes new pattern
observations from this request") that went through Kompress added an
observation to the shared TOIN store anyway. This change threads the
same per-call gate through the Kompress CCR store, both compressors, and
the router seam.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `store_kompress_in_ccr` gains a keyword-only `record_toin: bool =
True`; when `False` the CCR store write still happens (compressed output
stays retrievable, marker unchanged) but the
`get_toin().record_compression(...)` call is skipped.
- `KompressCompressor.compress()` / `compress_batch()` accept and
forward `record_toin` on every internal path (single, batched
reconstruction, sequential fallback); `apply()` derives it from a
`compression_policy` kwarg, the way `SmartCrusher.apply()` and
`CacheAligner` do.
- `RemoteKompressCompressor.compress()` accepts the same keyword
(keeping the drop-in parity contract) and forwards it to the shared
store function.
- `ContentRouter._try_ml_compressor` — the single seam both compressors
are called through — sends `record_toin=False` only when the request's
`_runtime_compression_policy.toin_read_only` is set, mirroring
`_invoke_smart_crusher`; the kwarg is omitted otherwise, so compressors
and test doubles without it keep working and default store calls keep
the exact pre-change 3-argument shape.
- Six regression tests in `tests/test_compression_policy_toin_gate.py`
covering the store function, local `compress()`, `compress_batch()`,
`apply()` under a Subscription policy, the remote client, and router
forwarding.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_compression_policy_toin_gate.py tests/test_remote_kompress_dropin.py tests/test_ccr_marker_policy.py tests/test_compression_policy.py
58 passed in the combined related run (gate file alone: 8 passed, 8 skipped — skips are the suite's own headroom._core-gated tests, no Rust wheel on this machine)
# Same five runnable new tests against unmodified main: 5 failed (they pass with this change)
$ ruff check headroom/transforms/kompress_compressor.py headroom/transforms/kompress_remote.py headroom/transforms/content_router.py tests/test_compression_policy_toin_gate.py
All checks passed!
$ ruff format --check (same four files)
4 files already formatted
# Issue repro, extended with the new gate (TOIN at a temp file):
store_kompress_in_ccr("word " * 400, "word " * 100, 400) -> TOIN total 0 -> 1 (reported bug; kept as the default for policy-less callers)
store_kompress_in_ccr("word " * 400, "word " * 100, 400, record_toin=False) -> TOIN total 1 -> 1, cache key still returned
store_kompress_in_ccr("word " * 400, "word " * 100, 400) -> TOIN total 1 -> 2
```
Type checking was not run locally (no mypy environment on this machine);
CI runs it.
## Real Behavior Proof
- Environment: Linux, Python 3.12, checkout of `main` at
|
||
|
|
f9627eb462 |
fix(compress): verify scalar types in lossless message comparisons (#4884)
## Description
Agent mode can report a message as lossless after a pipeline extension
changes a tool argument from JSON true to 1. Python considers those
values equal even inside nested dicts. The same issue affects integers
and floats.
## Type of Change
- [x] Bug fix
## Changes Made
- Verify aligned message structures before accepting equal-valued
scalars and require their types to match. Identical objects
short-circuit during traversal.
- Exercise changes both with and without simultaneous reversible string
compression. The existing token recount test now simulates a failed
string round trip at the leaf being verified.
## Testing
- [x] Ran the agent-mode, codec, compress API and canonical pipeline
suites. The regression test failed before the fix.
- [x] Ruff lint and format checks passed on changed files.
### Test Output
```text
72 passed, 7 skipped, 1 warning in 1.56s
```
## Real Behavior Proof
- Environment: macOS, Python 3.12.13, native extension from headroom-ai
0.44.0, offline.
- Exact command / steps: Run the following from the checkout with the
Python environment active.
- Observed result: The output changed from `True 0 int 1` to `True 1
bool True`. The changed tool call is now reverted.
- Not tested: Live providers or the full repository suite.
```sh
HEADROOM_OFFLINE=1 python - <<'PYTHON'
from dataclasses import replace
from headroom import compress
from headroom.hooks import CompressionHooks
from headroom.pipeline import PipelineStage
class NumericHook(CompressionHooks):
def on_pipeline_event(self, event):
if event.stage == PipelineStage.INPUT_COMPRESSED:
msg = dict(event.messages[0])
msg["content"] = [dict(msg["content"][0], input={"enabled": 1})]
return replace(event, messages=[msg])
return None
messages = [{"role": "assistant", "content": [
{"type": "tool_use", "id": "call", "name": "run", "input": {"enabled": True}}
]}]
r = compress(messages, mode="agent", hooks=NumericHook())
v = r.messages[0]["content"][0]["input"]["enabled"]
print(r.lossless, r.reverted_messages, type(v).__name__, v)
PYTHON
```
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Lossless verification reverts
messages with scalar type changes.
- Kill switch / disable path: Existing verify_lossless option outside
agent mode.
- Unsafe override required: No.
- Qualification impact: Adds regression coverage for nested booleans,
integers and floats.
- Rollback path: Revert this patch.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
303f521812 |
fix(smart-crusher): keep container-shaped strings as strings in lossless tables (#4875)
## Description
A string cell in a `json` column whose text is a JSON **container**
(`"[]"`, `"{}"`, `"[1,2]"`) is
classified as `StringifiedJson` and un-escaped into
`CellValue::Scalar(parsed)`, so the cell renders
byte-identically to the real value it spells. The csv-schema decoder
cannot tell the two apart
afterwards: `expand_compacted` returns `[]` where the original had
`"[]"`, and with
`verify_lossless=True` (also forced by `mode="agent"`) the round-trip
check fails and the whole
message is reverted, silently dropping its compression.
This is the container-shaped remainder of #4053, which fixed the
scalar-looking case (`"null"`,
`"true"`, `"1"`). The formatter already knows how to render these
strings unambiguously
(`csv_formatter_json_column_quotes_strings_that_parse_as_json` covers
`"[1]"` and `'{"a":1}'`), but
`compact()` never lets them reach it as strings.
Closes #4874
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `compactor::cell_from_value`: for `StringifiedJson` that is not an
array of objects, keep the
original string (`CellValue::Scalar(v.clone())`) instead of the parsed
value. The formatter's
existing ambiguous-text rule JSON-quotes it, so the decoder returns the
string.
- Nested behaviour is unchanged: a stringified **array of objects**
still becomes a nested table, so
`test_nested_table_markers_resolve_to_source_bytes` and its CCR markers
keep working.
- Rust regression test pinning the exact wire cells (`"[]"` for the
array, `"""[1]"""`-style for the
string).
- Python regression test at the caller boundary (`_crush` +
`expand_compacted`).
## Testing
- [x] Unit tests pass (`cargo test -p headroom-core --lib` -> 1044
passed, 1 ignored)
- [x] `cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo test -p headroom-core --lib compaction::
test result: ok. 74 passed; 0 failed; 0 ignored
$ cargo test -p headroom-core --lib
test result: ok. 1044 passed; 0 failed; 1 ignored
# with the fix reverted, the new Rust test fails:
left: "\"[]\""
right: "\"\"\"[]\"\"\""
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, Rust 1.100
`x86_64-pc-windows-gnu`, repo `main` at
`fda75de5`
- Exact command / steps: `cargo test -p headroom-core --lib
compaction::compactor::tests::stringified_container_strings_keep_their_identity`;
the test builds
`[{id:0,v:[]},{id:1,v:"[]"},{id:2,v:[1,2]},{id:3,v:"[1,2]"},{id:4,v:{}},{id:5,v:"{}"}]`,
compacts
it, and asserts the rendered `v` cells.
- Observed result: without the change the array and the string render
the same cell (`"[]"`); with
it the string renders as the JSON string literal (`"""[]"""`), which
`expand_compacted` maps back to `"[]"`.
- Not tested: the Python regression test could not be executed locally
because this machine cannot
build the native extension (Rust build stopped in `icu_segmenter` under
the GNU toolchain); it is
included for CI, where the wheel is built normally.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; the change is in the lossless table
codec only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: a stringified JSON container in a
table cell now stays a string
in the densified text (it was displayed as the parsed value). This is
required for the documented
reversibility of `csv-schema`; the model sees a JSON string literal
instead of nested JSON.
- Kill switch / disable path: n/a (lossless path)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the single `cell_from_value` branch.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4875"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
cb8414cc73 |
perf(bm25): lower batch allocations -- 93.4% less peak memory (#4893)
## TL;DR - Reduce peak memory in `BM25Scorer.score_batch` for 1,000 search results. - Keep corpus counts between two passes and reuse reason strings within each batch. - Peak allocation falls from 2,471,190 to 164,222 bytes, saving 2,306,968 bytes; runtime rises from 15.41 to 22.56 ms. - `pytest tests/test_relevance.py tests/test_relevance_extra.py -k 'bm25 or BM25' -q`: 16 passed, 32 deselected. ## Description The batch scorer kept every document's tokens until scoring finished. It now keeps query-term document counts, then reads each document again for scoring. The score formula, document lengths, and query-term counts stay the same. A small local dictionary also reuses equal result descriptions. ## Changes Made - `headroom/relevance/bm25.py`: `BM25Scorer.score_batch` keeps only query-term document counts in the first pass, then tokenizes each document again in the scoring pass. - `headroom/relevance/bm25.py`: a small per-batch dictionary reuses equal reason strings. - No public interface, setting, or test file changes. ## Validation These results use the repository's search-result generator on Linux x86_64 with Python 3.12.13 and `HEADROOM_OFFLINE=1`. Memory means peak Python allocations during one batch, not process RSS. Runtime rows show median times without allocation tracing. | Situation | Before | After | Change | |---|---|---|---| | 1,000 search results, peak allocation | 2,471,190 bytes | 164,222 bytes | 93.4% less | | 100 search results, runtime | 1.46824 ms | 2.28269 ms | 55.5% slower | | 1,000 search results, runtime | 15.40643 ms | 22.55516 ms | 46.4% slower | | 100 results with empty query, runtime | 0.03821 ms | 0.03777 ms | Unchanged path; no observed slowdown | | Single result, runtime | 0.01296 ms | 0.01295 ms | Unchanged path; no material difference | Memory measurements across seven fixed inputs and three runs give: | Statistic | Value | |---|---| | Old -> new (mean) | 2.4 MiB -> 160.4 KiB (93.4% less peak memory) | | 95% CI of the new/old ratio (bootstrap, 10,000 resamples) | 0.066–0.067 (93.3–93.4% less peak memory) | | Mean ± SD of the change per input | -2.2 MiB ± 1.7 KiB | | 95% CI of the mean change (t-interval) | -2.2 MiB to -2.2 MiB | | Number of fixed inputs (n) | 7 (3 runs) | This is a memory-for-time tradeoff: repeated tokenization makes nonempty batches slower. The later pass releases each token list after scoring its document. Run the repository benchmarks with: ```bash HEADROOM_OFFLINE=1 python -m pytest benchmarks/bench_relevance.py::TestBM25Benchmarks -q ``` All seven benchmarks passed. An additional local comparison checked 8,568 combinations against the original implementation with exact results. Those combinations cover empty documents, repeated words, UUIDs, Unicode, and different scoring settings. ## Why this is safe and easy to maintain The change adds six net code lines and keeps one shared scoring formula. Scores, result order, reasons, and matched terms remain unchanged for the checked inputs. Existing BM25 tests cover corpus IDF, matching, and normalization. ## Type of Change - [x] Performance improvement ## Testing - [x] Focused BM25 unit tests pass. - [x] `ruff check headroom/relevance/bm25.py` passes. - [x] `ruff format --check headroom/relevance/bm25.py` passes. - [ ] Full unit suite passes. - [ ] Type checking passes (`mypy headroom`). - [ ] New repository tests added. The broader relevance run reports 37 passed, eight skipped, and three failed on both versions. All three failures require the unavailable `headroom._core` extension in SmartCrusher integration tests. No new behavior or public interface is added; existing tests exercise the changed path. ### Test Output ```text $ HEADROOM_OFFLINE=1 pytest tests/test_relevance.py tests/test_relevance_extra.py -k 'bm25 or BM25' -q ====================== 16 passed, 32 deselected in 0.62s ======================= $ ruff check headroom/relevance/bm25.py && ruff format --check headroom/relevance/bm25.py All checks passed! 1 file already formatted $ HEADROOM_OFFLINE=1 python -m pytest benchmarks/bench_relevance.py::TestBM25Benchmarks -q ============================== 7 passed in 5.81s =============================== ``` ## Real Behavior Proof - Environment: Linux x86_64, Python 3.12.13, offline mode, no provider or model needed. - Exact command / steps: Run this direct scoring call after applying the patch: ```bash HEADROOM_OFFLINE=1 python -c 'from headroom.relevance.bm25 import BM25Scorer; s = BM25Scorer(); print([(r.score, r.matched_terms) for r in s.score_batch(["shared zeta", "shared alpha", "shared beta", "shared gamma"], "shared zeta")])' ``` - Observed result: The rare term ranks the first document higher: ```text [(0.13093333199837626, ['shared', 'zeta']), (0.010536051565782637, ['shared']), (0.010536051565782637, ['shared']), (0.010536051565782637, ['shared'])] ``` - Not tested: Full application traffic, native-extension integration, and other operating systems. ## Runtime Rollout Safety - Rollout-managed feature(s): None; this is the pure Python BM25 scorer. - Minimum rollout channel: Not applicable to this local scoring path. - Stable/default behavior changed: Lower temporary memory use with higher batch latency; scoring values are preserved. - Kill switch / disable path: No new setting; use the previous implementation if needed. - Unsafe override required: No override is required. - Qualification impact: No provider qualification changes. - Rollback path: Revert this single-file change. ## Review Readiness - [x] I have performed a self-review (automated source and result review by a separate reviewer step; see AI assistance below). - [x] Automated source and result checks completed. - [x] This PR is ready for human review, with the latency tradeoff stated above. ## Checklist - [x] The modified file follows the project's style guidelines. - [x] Source reviewed for duplicated scoring logic and unnecessary complexity. - [x] `CHANGELOG.md` was not edited. - [ ] Full repository checks completed. ## AI assistance This PR was prepared with AI assistance from Aiden. Candidates were tracked with Weco Observe. Weco share link: https://dashboard.weco.ai/share/Y27llB_ia4h2OzafJay18VxKVJ0L4m8I <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4893" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Aiden <aiden@weco.ai> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
c8168a0aa0 |
fix(cache): invalidate cached CCR markers after store eviction (#4872)
## Description When `ContentRouter` caches a compressed output that carries a CCR retrieval marker (`Retrieve more: hash=...`, `Retrieve original: hash=...`, or `<<ccr:...>>`), a subsequent `compress()` / `ContentRouter.apply()` call could hit Tier 2 `CompressionCache` and return that marker even after the underlying `CompressionStore` entry was evicted by `max_entries`, cleared, or expired. Any follow-up retrieval for that marker then returned `None`. This change: 1. Adds `CompressionCache.invalidate(key)` to evict a stale cached entry without marking it skipped, and tracks verified CCR marker hashes per cached entry (`_marker_hashes`) cleaned up alongside `_results`. 2. Records only marker hashes that actually existed in `CompressionStore` at `put()` time via `ContentRouter._collect_verified_marker_hashes`, preserving compatibility with synthetic/mocked router tests that do not write to `CompressionStore`. 3. Validates tracked marker hashes via `ContentRouter._cached_markers_retrievable` on both Tier 2 cache hit paths (`ContentRouter.apply` and `ContentRouter._compress_block_content`), invalidating stale entries and falling through to fresh compression so the original content is re-stored and retrievable. Closes #4796 *Disclosure: This fix and PR description were written by an AI agent (沈寻 / Shen Xun, `@xunWhale`) on behalf of `@e0_7`, and verified locally with pre-fix failure / post-fix pass pytest outputs, repo-wide ruff/mypy, and the issue reproduction.* ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/content_router.py`: added `_CCR_MARKER_HASH_RE`, `CompressionCache.invalidate()`, `CompressionCache.get_marker_hashes()`, `_marker_hashes` tracking inside `CompressionCache`, and `ContentRouter._collect_verified_marker_hashes()` / `_cached_markers_retrievable()` checks on both string and content-block Tier 2 cache hit paths. - `tests/test_router_result_cache.py`: added 4 regression tests covering `CompressionCache.invalidate()`, capacity eviction under `compress()` for both `role="tool"` string messages and Anthropic `tool_result` content blocks, and custom `CompressionStore.clear()` invalidation. ## Testing - [x] Unit tests pass (`pytest` on router/cache/compression suites; see scope note below) - [x] Linting passes (`ruff check .` & `ruff format --check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output Pre-fix baseline (`0368a87`, without `content_router.py` fix): ```text $ PYTHONPATH=. pytest tests/test_router_result_cache.py -k "capacity_eviction or store_cleared or invalidate_drops" -v FAILED tests/test_router_result_cache.py::test_invalidate_drops_cached_result_without_marking_skipped - AttributeError: 'CompressionCache' object has no attribute 'invalidate' FAILED tests/test_router_result_cache.py::test_compress_repopulates_ccr_store_after_capacity_eviction - assert None is not None FAILED tests/test_router_result_cache.py::test_compress_block_content_repopulates_ccr_store_after_capacity_eviction - assert None is not None FAILED tests/test_router_result_cache.py::test_router_invalidates_cached_marker_when_store_cleared - TypeError: CompressionCache.put() got an unexpected keyword argument 'marker_hashes' ======================= 4 failed, 28 deselected in 0.39s ======================= ``` Post-fix checks: ```text $ ruff check . All checks passed! $ ruff format --check . 1940 files already formatted $ mypy headroom Success: no issues found in 583 source files $ PYTHONPATH=. pytest tests/test_router_result_cache.py -v ... tests/test_router_result_cache.py::test_invalidate_drops_cached_result_without_marking_skipped PASSED [ 90%] tests/test_router_result_cache.py::test_compress_repopulates_ccr_store_after_capacity_eviction PASSED [ 93%] tests/test_router_result_cache.py::test_compress_block_content_repopulates_ccr_store_after_capacity_eviction PASSED [ 96%] tests/test_router_result_cache.py::test_router_invalidates_cached_marker_when_store_cleared PASSED [100%] ============================== 32 passed in 1.93s ============================== $ PYTHONPATH=. pytest tests/test_router_result_cache.py tests/test_router_cache_policy.py tests/test_transforms_content_router.py tests/test_compression_store.py tests/test_compression_safety_rails.py tests/test_compression_determinism.py tests/test_compression_decision.py tests/test_compression_units.py tests/test_compression_batches.py tests/test_compression_cache.py tests/test_ccr_batch_store.py -q ======================= 423 passed, 2 skipped in 23.02s ======================== ``` *(Scope note: The full repository test suite was not run locally because optional extras such as `[proxy]`/`fastapi` are not installed in this environment; `ruff check .`, `ruff format --check .`, and `mypy headroom` ran across the entire repository.)* ## Real behavior proof - Environment: Debian Linux (Python 3.11.2), Headroom `main` (`0368a87`) with `headroom._core` from `headroom-ai==0.44.0`, `InMemoryBackend(max_entries=2)`, `kompress_model="disabled"`. - Exact command / steps: Ran the issue #4796 reproduction (`compress` on 3 distinct 251-line tool log payloads so entry 0 is capacity-evicted from `CompressionStore`, followed by a repeat `compress(originals[0])` and `store.retrieve(repeat_key)`). - Observed result: - Before patch: `before_repeat_retrieve: None`, `repeat_key == keys[0]: True`, `store.retrieve(repeat_key): None` - After patch: `before_repeat_retrieve: None`, `after_repeat_key_match: True retrieved_present: True exact_match: True` (`store.retrieve(repeat_key).original_content` equals `originals[0][-1]["content"]`). - Not tested: Distributed Redis `CompressionStore` backend, `[proxy]` extra tests (`fastapi` not installed), or live upstream LLM API calls; `kompress_model="disabled"` was used as in the issue reproduction. ## Runtime Rollout Safety - Rollout-managed feature(s): None (bug fix in `ContentRouter` Tier 2 cache hit validation). - Minimum rollout channel: N/A - Stable/default behavior changed: Only invalidates cached CCR marker hits when their store-backed CCR entry is no longer present in `CompressionStore`. - Kill switch / disable path: Standard `HEADROOM_COMPRESSION_CACHE` disable / `invalidate`. - Unsafe override required: No. - Qualification impact: Prevents unrecoverable CCR markers on repeat compression after store capacity eviction. - Rollback path: Revert this single commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A — internal cache bug fix, no public API or doc change) - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4872" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
f4e3ad4b11 |
fix(content-router): keep list-form tool_result text blocks on separate lines (#4739)
## Description
A `tool_result` whose `content` is a list of several `text` blocks gets
flattened with `"".join(...)` in `ContentRouter` before the
read-protection check and before compression. MCP tools that return more
than one `TextContent` and litellm-style clients send this shape. When a
block does not end with a newline, the last line of one block and the
first line of the next one become a single line. The compressed result
is written back as one block, so the model sees a line the tool never
produced.
This PR joins the blocks with a newline unless the block already ends
with one, at both flatten sites.
Fixes #4729
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/content_router.py`: new helper
`_join_tool_result_text_blocks()` that joins list-form `text` blocks
with `\n`, skipping the separator when a block already ends with a
newline. Empty blocks add no extra line.
- Used at both places that flattened with `""`: the read-protection
check and the compression path. The wire shape does not change (the
existing `[{"type": "text", ...}]` re-wrap is kept), and results that
are not compressed keep their original blocks.
- `tests/test_transforms/test_content_router.py`: one `router.apply`
test with a two-block log `tool_result` that asserts that the output was
compressed and that every log line in it is a line the tool actually
returned, plus four parametrized cases for the helper.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Exact source: head `5305d8acb5f505e635d1287cae30ed7f8d9eb595`, base
`c05fbdddc873078dd97cb9ede841e75ef7ed0bef`. The PR has three commits:
`f273fbb9` (the fix, written on `1da8930a`) and two merges of `main`
(`2cba6c5b`, `5305d8ac`), both without conflicts. `git diff
c05fbddd...5305d8ac` still touches only the two files listed above.
Upstream CI on `5305d8ac`, all `completed` / `success`: CI
[43338122484](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122484),
Merge Conflicts
[43338122336](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122336),
Changelog Guard
[43338122370](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122370),
rust
[43338122318](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122318),
Evaluation Suite
[43338122592](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122592),
Security
[43338122460](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122460).
Local runs below were built from `git archive 5305d8ac` in a
`python:3.13-slim-bookworm` image with `uv sync --frozen --extra proxy
--extra dev`. ruff ran on the two touched files and mypy on
`content_router.py`. pytest ran on the content router test modules. For
the reverse check, `content_router.py` was replaced with the copy from
`c05fbddd` and the tests kept.
A wider run over the 122 test files that touch the content router gave 9
failed / 2669 passed on `5305d8ac` and 9 failed / 2664 passed on
`c05fbddd`. The 9 failures are the same set on both (memory_bridge,
offline_egress docs, tabular; they come from the test image, not this
change), and the extra 5 passes are the new tests.
### Test Output
```text
== RUFF CHECK RC 0
== RUFF FORMAT RC 0
== MYPY RC 0
# new tests, 5 consecutive runs, all identical:
======================= 5 passed, 66 deselected in 5.01s =======================
# test_content_router.py + test_content_router_ccr_retrieve_exemption.py + test_content_router_shell_small_results.py:
============================= 125 passed in 5.61s ==============================
# with content_router.py restored to
|
||
|
|
3aafdf2dcc |
fix(router): keep source code out of the prose (Kompress) fallbacks (#4787)
## Description When the code compressor declined a block of source code, the router passed the block to Kompress. Kompress is a prose model. It drops low-salience words, and in code those words are keywords such as `for`, `in`, `if` and `import`. The result no longer parses. #4214 shows a fenced Python script that comes back with `for` and `in` removed. This PR stops that in the Python router and in the Rust live-zone dispatcher. When the code compressor declines, the code is forwarded unchanged. Closes #4214 Credit: the Python change is oxura's work from https://github.com/oxura/headroom/tree/fix/fenced-code-routing-4214-20261018 (277ca1f). Their commit is the first commit here, rebased onto `main`, and oxura is its author. The test, docs and Rust commits were added on top. oxura asked us to open this PR from our fork ([comment](https://github.com/headroomlabs-ai/headroom/issues/4214#issuecomment-6797572655)), and said they have not run this branch, so the model and native results below are ours, not theirs. ## Behaviour change to review With `enable_code_aware=False` (the default), source code now stays unchanged. Before this PR, it was sent to Kompress and lost keywords. With `enable_code_aware=True`, code that the code compressor declines also stays unchanged. Before, Kompress ran on it. A code block the code compressor does compress is unaffected. Prose, logs, JSON and tables are unaffected. Side effect: a code block that Kompress used to shrink (and break) now keeps its full size. This is the intended trade, but it means fewer tokens saved on code when `enable_code_aware=False`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update - [ ] New feature - [ ] Breaking change (the default-config behaviour change above is intended, and it only stops damage to code) ## Changes Made Five commits: - `596764c3` `fix(router): keep source code out of prose fallbacks`. Author: oxura. Removes the CODE_AWARE to Kompress fallbacks in `headroom/transforms/content_router.py`. This is 277ca1f rebased onto `main`, and it differs from 277ca1f in two ways only: the two test-file conflicts are resolved, and its `wiki/proxy.md` hunk is dropped, because `AGENTS.md` says not to add content under `wiki/`. - `7705a4e6` `test(router): repin the code-to-Kompress tests and add the #4214 fence regression`. Author: Tao Hansen, with a `Co-authored-by: oxura` trailer. Five tests asserted the old fallback. They are rewritten to assert the new behaviour (code unchanged, Kompress never sees it) instead of being deleted. Adds a regression test for the script from #4214, with `enable_code_aware` both off and on. Updates `docs/content/docs/code-compression.mdx` and `docs/content/docs/how-compression-works.mdx`, which said that detected code goes to Kompress. - `6631668a` `fix(live-zone): leave source code unchanged when the code compressor declines`. Author: Tao Hansen. Same fix in `crates/headroom-core/src/transforms/live_zone.rs`: a declined `SourceCode` block is now a NoOp, not a Kompress input. Tests: `tests/live_zone_source_code_decline.rs` (single test, needs no model), and new cases in `tests/live_zone_dispatch.rs`. - `d8776283` `fix(router): turn off the code compressor's internal Kompress fallback`. Author: Tao Hansen. Follows review: the router's code compressor was still built with `fallback_to_kompress=self.config.enable_kompress` (#4719), so unknown languages, Perl and AST failures could build a `KompressCompressor` inside `CodeAwareCompressor`. The router now always sets `fallback_to_kompress=False`. The #4719 test that pinned `True` (`test_enabled_kompress_keeps_code_aware_fallback`) is repinned as `test_enabled_kompress_still_turns_off_code_aware_fallback`, and the unknown-language, Perl (by language hint, since `detect_language()` reports Perl-dominant code as unknown) and AST-failure paths are tested with spies on `is_kompress_available` and `KompressCompressor`, with `enable_kompress` both off and on. `docs/content/docs/code-compression.mdx` notes that the router always turns the option off. - `c1f8464e` `fix(router): keep passthrough code from external compressors and stale caches`. Author: Tao Hansen. Follows review: `_try_external_compressor` skips `PASSTHROUGH` blocks (code with `prefer_code_aware_for_code=False`, whole or as a mixed section), which would otherwise be offered to a prose-only external compressor as `text/plain`. The disk cache namespace now also hashes `_ROUTING_POLICY_REVISION`, so a same-version restart does not replay code that Kompress damaged before this change. Tests: two external-dispatch cases with a `text/plain` external, and one namespace test. All three fail on the previous head. I did not touch `CHANGELOG.md`. ## Testing - [x] Unit tests pass (`pytest`): see below, the failures are environment failures that also fail on `main` - [x] Linting passes (`ruff check .`, `ruff format --check .`) - [x] Type checking passes (`mypy headroom/transforms/content_router.py`) - [x] New tests added for new behaviour - [x] Manual testing performed (real model replay, below) ### Test Output macOS, Python 3.14. Head is now `c1f8464e`. ```text After the second review fix ( |
||
|
|
cf3284675e |
fix(lossless): never grep-fold Go, nginx, klog or access-log timestamps (#4901)
## Description The lossless search fold splits each line as `path:line:content` and factors a repeated `path` into a header. `_TIMESTAMP_ROW_RE` (#3419) keeps timestamped log lines out of it, because a log line has the same colon shape: folding it hoists the date and hour into a fake file header and every row starts mid-clock. The fold round-trips byte-exactly, so the inverse check in `compact_lossless` cannot catch it. The guard knew ISO, `MM/DD/YYYY`, syslog and bare-clock prefixes, but not these four common formats: | Format | Line | Split as | |---|---|---| | Go `log` package | `2026/09/02 14:30:00 ERROR ...` | `2026/09/02 14` / `30` / `00 ERROR ...` | | nginx error log | `2026/09/02 14:30:00 [error] ...` | same | | klog (Kubernetes components) | `E0902 14:30:00.000000 1 reflector.go:100] ...` | `E0902 14` / `30` / `00.000000 ...` | | common/combined access log | `10.0.0.7 - - [02/Sep/2026:14:30:00 +0000] ...` | `10.0.0.7 - - [02/Sep/2026` / `14` / `30:00 ...` | They reach the fold two ways: the bash-search fold (`grep ERROR api.log`) and the router's lossless-first stage, which tries the search fold on any tool output (`cat`, `tail`, `kubectl logs`). This adds those three prefix shapes to `_TIMESTAMP_ROW_RE`, the same place #3419 put the guard. The content detector (#3748) and the mixed-content splitter (#4463) share the regex; neither claimed these lines before, so for them only false positives could change, and there are none on real grep output (below). The access-log alternative differs from the one I suggested in #4900. That one (`\S+ \S+ \S+ \[...`) also matched a real `grep -n` row over an access log (`/var/log/nginx/access.log:12:10.0.0.7 - - [02/Sep/...`), because `\S+` swallows `path:12:10.0.0.7`. The fields now exclude `:`, which a mis-split line cannot contain before its first colon anyway. The host field takes name/IPv4 characters only (`[\w.-]`), capped at 255 (a DNS name is at most 253). With `[^\s:]` the check scanned every grep row's whole path, and fold + detection on 10k grep rows took 10% and 32% longer than main in two interleaved runs; with this class and cap it stops early on a path and on a long unbroken line such as a base64 blob. Closes #4900 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/lossless_compaction.py`: three alternatives in `_TIMESTAMP_ROW_RE`: `YYYY/MM/DD hh:mm` (Go log, nginx error log), `[IWEF]MMDD hh:mm` (klog/glog), and the access-log prefix `host ident user [dd/Mon/yyyy:hh:mm` with a `[\w.-]{1,255}` host. - `tests/test_lossless_timestamp_guard.py`: the four formats, plus an access log with a 253-character host, join `test_timestamped_logs_are_never_folded`. New `test_grep_rows_over_those_logs_still_fold`: real `grep -n` rows over the same logs (including a dated `2026/09/02/api.log` path) still fold under their file header. - `tests/test_bash_search_lossless_fold.py`: `test_go_log_keeps_its_timestamps` runs a Go log through `ContentRouter` as the output of `grep ERROR /var/log/api.log` (bash-search fold) and of `cat /var/log/api.log` (lossless-first stage). ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # this branch's tests against main's lossless_compaction.py (main |
||
|
|
0d203db6b8 |
fix(copilot): keep new VS Code chats on the Local harness so they reach the proxy (#4899)
## Description
VS Code can run a Copilot chat on its "Copilot harness" (Agent Host, the
bundled Copilot SDK runtime) instead of the Local harness. That runtime
ignores `github.copilot.advanced.debug.overrideCapiUrl` /
`overrideProxyUrl`, so the chat goes straight to GitHub. Copilot's small
background calls (title, intent) still reach the proxy, so the proxy
looks connected while the main conversation never arrives. The harness
default (`chat.defaultToCopilotHarness`,
`chat.editor.preferCopilotHarness`) is tagged `experiment: startup` in
VS Code, so it can be flipped per user by rollout.
`headroom wrap vscode` now pins new chats to the Local harness.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/copilot/vscode.py`: the managed block also writes
`"chat.defaultToCopilotHarness": false` and
`"chat.editor.preferCopilotHarness": false`. Because they sit inside the
marker block, `headroom unwrap vscode` removes them. A harness setting
the user already set outside the block is left alone (no duplicate key).
New helper `copilot_harness_settings_enabled(path)` reports a user value
of `true`.
- `headroom/cli/wrap.py`: one line saying new chats stay on the Local
harness and existing chats keep their engine; a warning instead when the
user's own setting is `true`. `--no-configure` prints the two extra
settings.
- `docs/content/docs/vscode-copilot.mdx`: documents the harness settings
and the troubleshooting case.
- Blocks written by older versions (including ones with
`overrideAuthType`) are replaced by marker as before.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
tests/test_provider_copilot_vscode_config.py + tests/test_cli/test_wrap_vscode.py: 32 passed.
Against origin/main source: 7 of those fail (block contents, old-block upgrade, user-set key [True]/[False], wrap output, --no-configure output).
ruff 0.17.4 check + format --check on the 5 changed Python files: clean.
mypy on the changed source files: 8 errors, all in headroom/cache/prefix_tracker.py and headroom/ccr/mcp_server.py, identical on origin/main; none in changed files.
```
## Real Behavior Proof
- Environment: macOS 27.2.1, Python 3.12.13, VS Code 1.145.2 with GitHub
Copilot signed in, core run from this branch, real GitHub Copilot
upstream. Real user `settings.json` (backed up and restored afterwards).
- Exact command / steps:
1. `headroom wrap vscode -p 8795` from this branch.
2. `code chat -n -m agent "Reply with the single word OK and nothing
else."` (opens a new VS Code window and sends one Agent-mode chat).
3. Read the proxy's `/stats`, VS Code's `agenthost.log`, then `headroom
unwrap vscode`.
- Observed result:
```text
VS Code Copilot proxy settings updated: ~/Library/Application
Support/Code/User/settings.json
Keep using Copilot's normal model picker; the selected model is
preserved.
New chats stay on VS Code's Local harness; a chat keeps the engine it
started with, so start a new chat.
settings.json after wrap:
{
// --- Headroom Copilot proxy ---
"github.copilot.advanced.debug.overrideProxyUrl":
"http://127.0.0.1:8795/p/win-local",
"github.copilot.advanced.debug.overrideCapiUrl":
"http://127.0.0.1:8795/p/win-local",
"chat.defaultToCopilotHarness": false,
"chat.editor.preferCopilotHarness": false
// --- end Headroom Copilot proxy ---
}
proxy requests before chat: 0 {}
proxy requests after one Agent-mode chat in a new VS Code window: 4
{'gpt-4o-mini-2024-07-18': 3, 'claude-opus-5.5': 1}
agent host turns during the chat: 0
Removed Headroom Copilot proxy settings from ~/Library/Application
Support/Code/User/settings.json
settings.json after unwrap:
{
}
```
Separately, on the same machine with the settings edited by hand (same
VS Code, same proxy, new window each time):
`"chat.defaultToCopilotHarness": true` -> the Agent-mode chat ran in the
Agent Host (provider `copilotcli`) and 0 requests reached the proxy;
`"chat.defaultToCopilotHarness": false` +
`"chat.editor.preferCopilotHarness": false` -> Local harness, 4 requests
reached the proxy including the main `/v1/messages` call.
`"chat.temporaryDefaultToLocalHarness": true` as a user setting had no
effect, so it is not written.
- Not tested: a VS Code where Microsoft's rollout has flipped the
default (the explicit user value is expected to win, but that could not
be reproduced); an organization that enforces managed sandbox settings,
where VS Code can require the Copilot harness regardless; VS Code
profiles, workspace-level settings, Insiders/VSCodium; Windows and
Linux; the persistent `headroom start --persistent --vscode` path in the
Enterprise client.
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: `headroom wrap vscode` writes two
more VS Code user settings inside its managed block. No proxy behaviour
changes.
- Kill switch / disable path: `headroom unwrap vscode` removes them; a
user who sets either key themselves keeps their value.
- Unsafe override required: no.
- Qualification impact: none expected.
- Rollback path: revert this commit; `unwrap` removes blocks of either
format.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- This keeps users off the Copilot harness; it does not route that
harness through Headroom. Supporting it is separate work.
- Only the configured settings file is checked for a user value. A
`true` in workspace settings, another profile, or policy is not
detected.
- After `unwrap` the keys return to VS Code's default, which a rollout
may move to the Copilot harness.
- Existing chats keep the harness they started with; the wrap output
says to start a new chat.
- `e2e/copilot_live.py` and `e2e/wrap/run.py` were not run.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4899"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
b560cba927 |
fix(memory): parse frontmatter delimiters on complete lines (#4886)
## Description A memory beginning with `Use foo---bar as the separator` writes that text into the generated name/description. The reader takes the embedded `---` as the closing delimiter, mixes frontmatter into the body and changes the content hash. The next sync imports a polluted duplicate. Recognize opening and closing frontmatter delimiters only on complete lines, including LF and CRLF input. Preserve the original text if the header is malformed or has no closing delimiter. Related to #4832's full-store sync work; this fixes a separate contract on the current main snapshot. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Require complete delimiter lines; preserve bodies containing embedded dashes and malformed headers. - Add regressions against the unmodified base `5023c84e527e28281dd73d1ba04184fe738aac0f` and real file/component coverage. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook with httpx installed) - [x] New tests added - [ ] Manual testing performed by the human author ### Test Output 7 new regressions; sync, CLI index and supersession suites: 77 passed. The existing 35 sync cases are included. Existing datetime deprecation warnings remain. Original pre-commit and commit-msg hooks passed. Original full pre-push `make ci-precheck` passed; native extension rebuilt/import-verified. Full gate summaries: ```text cargo fmt / clippy --workspace: passed Rust workspace: 1755 passed, 3 ignored Native extension: rebuilt and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing development environment; base `5023c84e527e28281dd73d1ba04184fe738aac0f`, patch `1fda45a711b4ce38848e8615148c74dc9c1bd6ab`. Real LocalBackend factory, SQLite/FTS/vector store, agent files and Ollama HTTP adapter; the loopback service returns controlled 384D vectors. - Exact command / steps: save the following script as `proof.py`. Run `python proof.py frontmatter <fresh-output-directory> before` from the base checkout, then `python proof.py frontmatter <another-fresh-directory> after` from the patch. The identical script ran outside pytest. - Observed result: first_exported=1 on both; round_trip_equal: main=false, patch=true; repeat_imported/repeat_exported: main=1/1, patch=0/0; primary_count: main=2, patch=1. The patch uses one actual embedding HTTP request; main uses a second request to embed the polluted duplicate. - Not tested: trained-model inference or answer quality, paid/provider APIs, non-local backends, production workloads, other operating systems or the entire Python suite. Local focused, broader and original-gate results are reported separately. <details> <summary>Standalone runtime reproduction</summary> ```python """Real LocalBackend/Ollama HTTP/SQLite/agent-file before-after replay.""" import asyncio import json import os from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path import sys import threading from headroom.memory.backends.local import LocalBackend, LocalBackendConfig from headroom.memory.sync import sync from headroom.memory.sync_adapters.codex_agent import CodexAdapter from headroom.memory.sync_adapters.claude_code import ClaudeCodeAdapter case, folder, expected = sys.argv[1:] out = Path(folder).resolve() out.mkdir(parents=True, exist_ok=False) os.environ.update(HEADROOM_STATELESS="0", HEADROOM_WORKSPACE_DIR=str(out/'workspace'), HEADROOM_OFFLINE="0") class Endpoint(BaseHTTPRequestHandler): requests = [] def log_message(self, *_): pass def do_POST(self): payload=json.loads(self.rfile.read(int(self.headers['content-length']))) self.requests.append({'path':self.path,'body':payload}) assert self.path=='/api/embeddings' body=json.dumps({'embedding':[1.0]+[0.0]*383}).encode() self.send_response(200) self.send_header('Content-Type','application/json') self.send_header('Content-Length',str(len(body))) self.end_headers() self.wfile.write(body) server=ThreadingHTTPServer(('127.0.0.1',0),Endpoint) threading.Thread(target=server.serve_forever,daemon=True).start() async def run(): backend=LocalBackend(LocalBackendConfig(db_path=str(out/'memory.db'),embedder_backend='ollama',embedder_model='controlled-vector',vector_dimension=384,ollama_base_url=f'http://127.0.0.1:{server.server_port}')) state=out/'sync-state.json' try: if case=='fingerprint': saved=await backend.save_memory('Original orchard fact',user_id='alice') adapter=CodexAdapter(out/'AGENTS.md') first=await sync(backend,adapter,'alice',state) unchanged=await sync(backend,adapter,'alice',state) await backend._hierarchical_memory.update(saved.id,content='Corrected orchard fact') result=await sync(backend,adapter,'alice',state) text=adapter._path.read_text(encoding='utf-8') data={'first_exported':first.exported,'unchanged_exported':unchanged.exported,'edited_exported':result.exported,'corrected_in_file':'Corrected orchard fact' in text,'edited_imported':result.imported} assert data['first_exported']==1 and data['unchanged_exported']==0 assert data['corrected_in_file']==(expected=='after'),data assert result.exported==(1 if expected=='after' else 0),data else: content='Use foo---bar as the separator\nKeep the second line intact.' saved=await backend.save_memory(content,user_id='alice') adapter=ClaudeCodeAdapter(out/'claude-memory') first=await sync(backend,adapter,'alice',state,force=True) readback=await adapter.read_memories() repeated=await sync(backend,adapter,'alice',state,force=True) rows=await backend.get_user_memories('alice') data={'first_exported':first.exported,'round_trip_equal':readback[0].content==content,'reread_content':readback[0].content,'repeat_imported':repeated.imported,'repeat_exported':repeated.exported,'primary_count':len(rows)} assert data['round_trip_equal']==(expected=='after'),data assert data['repeat_imported']==(0 if expected=='after' else 1),data data['requests']=Endpoint.requests data['source_file']=__import__('headroom').__file__ (out/'result.json').write_text(json.dumps(data,indent=2),encoding='utf-8',newline='\n') print(json.dumps(data,ensure_ascii=False)) finally: await backend.close() try: asyncio.run(run()) finally: server.shutdown();server.server_close() ``` </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none; existing memory sync. - Minimum rollout channel: existing memory sync users. - Stable/default behavior changed: metadata containing triple dashes remains separate from the memory body. - Kill switch / disable path: stop invoking memory sync; no new toggle. - Unsafe override required: no. - Qualification impact: memory synchronization correctness; no dependency, native or workflow change. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented the non-obvious behavior - [x] I have added regression tests - [x] I did not edit CHANGELOG.md ## Additional Notes Implementation and local test execution were completed with Codex assistance. The author personally reviewed the two patches and the before/after evidence; this does not claim the author personally ran the tests. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4886" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
00b38736d9 |
fix(memory): detect content changes in incremental sync (#4885)
## Description Incremental sync hashes only the row count and the first eight characters of the newest five IDs. An in-place content edit keeps those values unchanged, so sync returns early and never exports the corrected fact. Hash the content, complete IDs and fields used for export/lineage across the already-fetched snapshot. Unchanged snapshots still take the no-op path. The persisted fingerprint is opaque; existing states receive one normal sync after this change. Related to #4832's full-store sync work; this fixes a separate contract on the current main snapshot. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Include full IDs, content and export/lineage fields from every returned row; exclude volatile access counters. - Add regressions against the unmodified base `5023c84e527e28281dd73d1ba04184fe738aac0f` and real file/component coverage. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook with httpx installed) - [x] New tests added - [ ] Manual testing performed by the human author ### Test Output 6 new regressions; focused sync suite: 41 passed. Broader memory run: 675 passed, 82 skipped, 8 failed. The same 8 failures reproduce on the exact unmodified base (275 passed in the two affected files): six Windows SQLite temporary-directory cleanup failures and two POSIX-path assumptions. No old fixture changed. Original pre-commit and commit-msg hooks passed. Original full pre-push `make ci-precheck` passed; native extension rebuilt/import-verified. Full gate summaries: ```text cargo fmt / clippy --workspace: passed Rust workspace: 1755 passed, 3 ignored Native extension: rebuilt and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing development environment; base `5023c84e527e28281dd73d1ba04184fe738aac0f`, patch `c4ffa08665c26491b2379af0f4d953ceb203ee6b`. Real LocalBackend factory, SQLite/FTS/vector store, agent files and Ollama HTTP adapter; the loopback service returns controlled 384D vectors. - Exact command / steps: save the following script as `proof.py`. Run `python proof.py fingerprint <fresh-output-directory> before` from the base checkout, then `python proof.py fingerprint <another-fresh-directory> after` from the patch. The identical script ran outside pytest. - Observed result: first_exported=1; unchanged_exported=0; edited_exported: main=0, patch=1; corrected_in_file: main=false, patch=true. The existing additive bidirectional semantics also reimport the old file fact (edited_imported=1 on the patch); this change does not implement deletion propagation or conflict resolution. - Not tested: trained-model inference or answer quality, paid/provider APIs, non-local backends, production workloads, other operating systems or the entire Python suite. Local focused, broader and original-gate results are reported separately. <details> <summary>Standalone runtime reproduction</summary> ```python """Real LocalBackend/Ollama HTTP/SQLite/agent-file before-after replay.""" import asyncio import json import os from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path import sys import threading from headroom.memory.backends.local import LocalBackend, LocalBackendConfig from headroom.memory.sync import sync from headroom.memory.sync_adapters.codex_agent import CodexAdapter from headroom.memory.sync_adapters.claude_code import ClaudeCodeAdapter case, folder, expected = sys.argv[1:] out = Path(folder).resolve() out.mkdir(parents=True, exist_ok=False) os.environ.update(HEADROOM_STATELESS="0", HEADROOM_WORKSPACE_DIR=str(out/'workspace'), HEADROOM_OFFLINE="0") class Endpoint(BaseHTTPRequestHandler): requests = [] def log_message(self, *_): pass def do_POST(self): payload=json.loads(self.rfile.read(int(self.headers['content-length']))) self.requests.append({'path':self.path,'body':payload}) assert self.path=='/api/embeddings' body=json.dumps({'embedding':[1.0]+[0.0]*383}).encode() self.send_response(200) self.send_header('Content-Type','application/json') self.send_header('Content-Length',str(len(body))) self.end_headers() self.wfile.write(body) server=ThreadingHTTPServer(('127.0.0.1',0),Endpoint) threading.Thread(target=server.serve_forever,daemon=True).start() async def run(): backend=LocalBackend(LocalBackendConfig(db_path=str(out/'memory.db'),embedder_backend='ollama',embedder_model='controlled-vector',vector_dimension=384,ollama_base_url=f'http://127.0.0.1:{server.server_port}')) state=out/'sync-state.json' try: if case=='fingerprint': saved=await backend.save_memory('Original orchard fact',user_id='alice') adapter=CodexAdapter(out/'AGENTS.md') first=await sync(backend,adapter,'alice',state) unchanged=await sync(backend,adapter,'alice',state) await backend._hierarchical_memory.update(saved.id,content='Corrected orchard fact') result=await sync(backend,adapter,'alice',state) text=adapter._path.read_text(encoding='utf-8') data={'first_exported':first.exported,'unchanged_exported':unchanged.exported,'edited_exported':result.exported,'corrected_in_file':'Corrected orchard fact' in text,'edited_imported':result.imported} assert data['first_exported']==1 and data['unchanged_exported']==0 assert data['corrected_in_file']==(expected=='after'),data assert result.exported==(1 if expected=='after' else 0),data else: content='Use foo---bar as the separator\nKeep the second line intact.' saved=await backend.save_memory(content,user_id='alice') adapter=ClaudeCodeAdapter(out/'claude-memory') first=await sync(backend,adapter,'alice',state,force=True) readback=await adapter.read_memories() repeated=await sync(backend,adapter,'alice',state,force=True) rows=await backend.get_user_memories('alice') data={'first_exported':first.exported,'round_trip_equal':readback[0].content==content,'reread_content':readback[0].content,'repeat_imported':repeated.imported,'repeat_exported':repeated.exported,'primary_count':len(rows)} assert data['round_trip_equal']==(expected=='after'),data assert data['repeat_imported']==(0 if expected=='after' else 1),data data['requests']=Endpoint.requests data['source_file']=__import__('headroom').__file__ (out/'result.json').write_text(json.dumps(data,indent=2),encoding='utf-8',newline='\n') print(json.dumps(data,ensure_ascii=False)) finally: await backend.close() try: asyncio.run(run()) finally: server.shutdown();server.server_close() ``` </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none; existing memory sync. - Minimum rollout channel: existing memory sync users. - Stable/default behavior changed: edited sync snapshots trigger comparison/export instead of a false no-op. - Kill switch / disable path: stop invoking memory sync; no new toggle. - Unsafe override required: no. - Qualification impact: memory synchronization correctness; no dependency, native or workflow change. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented the non-obvious behavior - [x] I have added regression tests - [x] I did not edit CHANGELOG.md ## Additional Notes Implementation and local test execution were completed with Codex assistance. The author personally reviewed the two patches and the before/after evidence; this does not claim the author personally ran the tests. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4885" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
0d9155019f |
fix(memory): dispatch the advertised memory_list tool (#4830)
## Description `MemorySystem.get_tools()` advertises `memory_list`, but `process_tool_call()` has no handler for it. An agent can see the tool and still receive `Unknown tool: memory_list` when trying to browse stored memories or obtain an ID for an update. Dispatch the existing tool to the backend's scoped listing API. A session-bound MemorySystem lists only that user's current session; an unscoped one retains user-wide browsing. Return recent memories with their IDs and readable fields, without embeddings or a semantic query. Backends without the requested listing scope return an explicit unsupported-operation error rather than falling back to a wider scope. Refs #4362, item 1 only. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Add the `memory_list` dispatcher and public handler. - Keep the schema's default limit of 10 and bounds of 1–100. Filter by user and, when bound, session before applying the limit, using the existing HierarchicalMemory.get_session_memories API through LocalBackend. - Cover normal and optimized tool inventories, empty results, bounds, unsupported backends and backend failures. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook) - [x] Regression tests added - [x] Standalone runtime validation completed ### Test Output ```text Initial regressions against original production code: 8 failed Session regressions against initial PR head: 4 failed, 9 passed Final list regressions on the corrected head: 13 passed pytest tests/test_memory_list_dispatch.py tests/test_memory_system.py tests/test_memory_tool_project_scope.py -q: 113 passed, 13 skipped Original pre-commit and commit-msg hooks: passed Whole-tree Ruff check and format --check: passed Original pre-push make ci-precheck: passed Rust workspace: 1749 passed, 3 ignored Native extension: rebuilt, installed and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` The owning-suite skips require Hugging Face model files unavailable offline. Existing datetime/optional-model deprecation warnings remain visible in the logs. ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing dev venv; clean production baseline `9f2cc017863f540c61f8747b3c744a97776ebaf2`, patch `42aca035dde6734d1a9d7a0b87f1b99cf8b27e69`. Explicit checkout on PYTHONPATH; no paid/provider endpoint. The loopback service returns controlled vectors, not trained-model inference. - Exact command / steps: Save the script below as `proof.py`. From each checkout run `python proof.py list <fresh-output-directory> before` on main, or `python proof.py list <fresh-output-directory> after` on the patch. Identical scripts ran outside pytest. The script initializes a real LocalBackend, saves three Alice memories and one Bob memory through MemorySystem, then calls the advertised listing tool with limit=2. The Ollama HTTP adapter sends actual requests to a bound loopback receiver returning controlled 384-dimensional vectors; SQLite, FTS5 and the configured local vector index are real. <details> <summary>Standalone reproduction script</summary> ```python """Standalone real SQLite/vector/FTS/HTTP and agent-file boundary reproductions. Usage: python memory_http_proof.py {list,vectors,sync} FRESH_OUTPUT {before,after} The loopback embedding service returns controlled vectors, not model inference. """ import asyncio from datetime import datetime, timedelta, timezone import hashlib from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import json import os from pathlib import Path import platform import sqlite3 import sys import threading import headroom import headroom._core from headroom.memory.backends.local import LocalBackend, LocalBackendConfig from headroom.memory.config import EmbedderBackend, MemoryConfig, VectorBackend from headroom.memory.core import HierarchicalMemory from headroom.memory.models import Memory from headroom.memory.sync import sync, sync_import from headroom.memory.sync_adapters.codex_agent import CodexAdapter from headroom.memory.system import MemorySystem sys.stdout.reconfigure(encoding="utf-8") case, directory, expected = sys.argv[1:] assert case == "list" assert expected in ("before", "after") out = Path(directory).resolve() out.mkdir(parents=True, exist_ok=False) os.environ["HEADROOM_STATELESS"] = "0" os.environ["HEADROOM_WORKSPACE_DIR"] = str(out / "workspace") os.environ["HEADROOM_CCR_BACKEND"] = "memory" os.environ["HEADROOM_OFFLINE"] = "0" # Only the ephemeral loopback service is called. class Endpoint(BaseHTTPRequestHandler): dimension = 384 requests = [] def log_message(self, *_): pass def do_POST(self): payload = json.loads(self.rfile.read(int(self.headers["content-length"]))) Endpoint.requests.append({"path": self.path, "body": payload, "dimension": self.dimension}) assert self.path == "/api/embeddings", self.path body = json.dumps({"embedding": [1.0] + [0.0] * (self.dimension - 1)}).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) server = ThreadingHTTPServer(("127.0.0.1", 0), Endpoint) worker = threading.Thread(target=server.serve_forever, daemon=True) worker.start() url = f"http://127.0.0.1:{server.server_port}" def backend_config(name): return LocalBackendConfig( db_path=str(out / (name + ".db")), embedder_backend="ollama", embedder_model="probe-fixed", ollama_base_url=url, vector_dimension=384, ) async def list_case(): backend = LocalBackend(backend_config("list")) try: system = MemorySystem(backend, user_id="alice") ids = [] for content in ("First scoped fact", "Second scoped fact", "Latest scoped fact"): response = await system.process_tool_call("memory_save", {"content": content, "importance": 0.4}) assert response["success"], response ids.append(response["memory_id"]) response = await MemorySystem(backend, "bob").process_tool_call("memory_save", {"content": "Bob's private fact", "importance": 0.4}) assert response["success"], response before_requests = len(Endpoint.requests) result = await system.process_tool_call("memory_list", {"limit": 2}) result["embedding_requests_for_listing"] = len(Endpoint.requests) - before_requests result["stored_alice_count"] = len(await backend.get_user_memories("alice")) if expected == "after": assert result["success"] and result["count"] == 2, result assert [m["memory_id"] for m in result["memories"]] == [ids[2], ids[1]] assert all("Bob" not in m["content"] for m in result["memories"]) else: assert result["success"] is False and result["error"] == "Unknown tool: memory_list", result assert result["stored_alice_count"] == 3 and result["embedding_requests_for_listing"] == 0 return result finally: await backend.close() try: result = asyncio.run(list_case()) result.update({ "case": case, "expected": expected, "python": platform.python_version(), "source": str(Path(headroom.__file__).resolve()), "native": str(Path(headroom._core.__file__).resolve()), "http_requests": Endpoint.requests, "not_tested": "Trained model inference, paid APIs, provider answer quality, non-Windows runtime, non-local memory backends.", }) (out / "result.json").write_text(json.dumps(result, indent=2), encoding="utf-8", newline="\n") print(json.dumps(result, indent=2)) finally: server.shutdown() server.server_close() worker.join(timeout=5) ``` </details> - Observed result: On main the tool returns Unknown tool: memory_list even though Alice has three stored rows. With the patch it returns Alice's two newest memories, in order, with the saved IDs. Bob's memory is absent, and listing makes zero additional embedding requests. Session-scope follow-up: run the script below as `python session_proof.py <fresh-output-directory> before` on the initial PR head `908762a8705dea9a878f28dca4cb25b3fc5136ee`, or with `after` on the corrected head. It saves real memories for Alice/session-a, Alice/session-b and Bob/session-a through LocalBackend and its actual loopback Ollama HTTP adapter. With limit=1, the initial head returns the newer Alice/session-b fact to session-a; the correction returns session-a's fact. A missing session returns zero results, unscoped browsing still returns both Alice facts, Bob remains excluded, and listing sends no embedding request. <details> <summary>Session isolation reproduction</summary> ```python """Real LocalBackend listing must filter user and session before LIMIT.""" import asyncio from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import json import os from pathlib import Path import sys import threading from headroom.memory.backends.local import LocalBackend, LocalBackendConfig from headroom.memory.system import MemorySystem sys.stdout.reconfigure(encoding="utf-8") directory, expected = sys.argv[1:] assert expected in ("before", "after") out = Path(directory).resolve() out.mkdir(parents=True, exist_ok=False) os.environ.update(HEADROOM_OFFLINE="0", HEADROOM_STATELESS="0", HEADROOM_CCR_BACKEND="memory") class Endpoint(BaseHTTPRequestHandler): calls = [] def log_message(self, *_): pass def do_POST(self): request = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) assert self.path == "/api/embeddings" Endpoint.calls.append(request["prompt"]) response = json.dumps({"embedding": [1.0] + [0.0] * 383}).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(response))) self.end_headers() self.wfile.write(response) server = ThreadingHTTPServer(("127.0.0.1", 0), Endpoint) worker = threading.Thread(target=server.serve_forever, daemon=True) worker.start() async def main(): backend = LocalBackend(LocalBackendConfig( db_path=str(out / "memory.db"), embedder_backend="ollama", embedder_model="probe-fixed", ollama_base_url=f"http://127.0.0.1:{server.server_port}", vector_dimension=384, )) try: current = MemorySystem(backend, "alice", session_id="session-a") other_session = MemorySystem(backend, "alice", session_id="session-b") other_user = MemorySystem(backend, "bob", session_id="session-a") saved = [] for system, content in ((current,"Current-session fact"),(other_session,"Newer other-session fact"),(other_user,"Newest other-user fact")): result = await system.process_tool_call("memory_save", {"content":content,"importance":0.4}) assert result["success"],result saved.append(result["memory_id"]) count = len(Endpoint.calls) listed = await current.process_tool_call("memory_list", {"limit":1}) assert listed["success"] and listed["count"] == 1,listed scope_preserved = listed["memories"][0]["memory_id"] == saved[0] assert scope_preserved == (expected == "after"),listed assert listed["memories"][0]["memory_id"] != saved[2] missing = await MemorySystem(backend,"alice",session_id="missing").process_tool_call("memory_list", {"limit":1}) assert missing["count"] == (0 if expected == "after" else 1),missing unscoped = await MemorySystem(backend,"alice").process_tool_call("memory_list", {"limit":2}) assert [m["memory_id"] for m in unscoped["memories"]] == [saved[1],saved[0]] assert len(Endpoint.calls) == count result = {"session_isolated":scope_preserved,"scoped_content":listed["memories"][0]["content"],"missing_session_count":missing["count"],"unscoped_count":unscoped["count"],"other_user_excluded":True,"embedding_calls_for_listing":len(Endpoint.calls)-count,"save_http_calls":count,"expected":expected,"source":str(Path(__import__('headroom').__file__).resolve())} (out/"result.json").write_text(json.dumps(result,indent=2),encoding="utf-8",newline="\n") print(json.dumps(result,indent=2)) finally: await backend.close() try: asyncio.run(main()) finally: server.shutdown() server.server_close() worker.join(timeout=5) ``` </details> - Not tested: Remote memory backends, trained embedding inference, paid APIs, model answer quality, non-Windows runtime, or the complete Python suite. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing memory behavior; no new feature flag or channel expansion. - Minimum rollout channel: Normal release after maintainer review and CI qualification. - Stable/default behavior changed: The advertised tool can enumerate recent memories on backends that expose get_user_memories. Other memory operations retain their dispatch paths. - Kill switch / disable path: Use the existing memory-disable path, or revert this commit. - Unsafe override required: No. - Qualification impact: Related tests and original local gates passed; hosted CI is tracked separately. - Rollback path: Revert this commit; no dependency or storage-schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove the fix - [x] Relevant new and existing tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Listing is supported through LocalBackend's user and session APIs; the session adapter delegates to an existing core query that applies both filters before LIMIT. This patch does not add an enumeration API to DirectMem0. The author's review covered the initial patch and evidence; the session-isolation review correction was additionally reviewed and validated with Codex. Developed and validated with Codex assistance. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4830" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
cd6f438fec |
fix(memory): validate embedding dimensions before writes (#4831)
## Description A mismatched embedding reaches the vector index after the primary memory row has already been saved. A rejected add leaves a half-written row; a rejected supersession also ends the old version and removes its index entry before failing. This occurs with a vector_dimension that differs from the embedder's actual output. Validate each newly produced embedding's shape before the first primary-store mutation. Apply this to add, the whole add_batch, content updates and supersession. An invalid replacement leaves the old row, cached object and version chain intact. Refs #4362, item 2's half-write boundary. The configured vector dimension remains explicit; this does not automatically resize or migrate an index. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Reject non-vector and wrong-width embeddings before saving rows. - Validate an entire batch before saving any member. - Validate a content update before mutating a store-returned object, preserving update()'s existing object identity and metadata-only refresh behavior. - Validate supersession before ending the old version or removing its indexes. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook) - [x] Regression tests added - [x] Standalone runtime validation completed ### Test Output ```text Final regressions against original production code: 7 failed, 1 passed pytest tests/test_memory_embedding_validation.py tests/test_memory/test_core_operations.py tests/test_memory/test_hierarchical.py tests/test_memory/test_core_metadata_index_sync.py tests/test_memory/test_clear_scope_consistency.py -q: 65 passed, 51 skipped Original pre-commit and commit-msg hooks: passed Whole-tree Ruff check and format --check: passed Original pre-push make ci-precheck: passed Rust workspace: 1749 passed, 3 ignored Native extension: rebuilt, installed and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` The owning-suite skips require Hugging Face model files unavailable offline. Existing datetime/optional-model deprecation warnings remain visible in the logs. ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing dev venv; clean production baseline `9f2cc017863f540c61f8747b3c744a97776ebaf2`, patch `50d126b1b8379390e72d3e47216519f9d297b4a1`. Explicit checkout on PYTHONPATH; no paid/provider endpoint. The loopback service returns controlled vectors, not trained-model inference. - Exact command / steps: Save the script below as `proof.py`. From each checkout run `python proof.py vectors <fresh-output-directory> before` on main, or `python proof.py vectors <fresh-output-directory> after` on the patch. Identical scripts ran outside pytest. The script creates real HierarchicalMemory with SQLite, FTS5, HNSW(dimension=384) and the Ollama HTTP adapter. A bound loopback server first returns a 768-dimensional embedding for a rejected add, then 384 dimensions for a valid original, then 768 dimensions for a rejected supersession. It inspects primary rows, the original version and the live vector index. <details> <summary>Standalone reproduction script</summary> ```python """Standalone real SQLite/vector/FTS/HTTP and agent-file boundary reproductions. Usage: python memory_http_proof.py {list,vectors,sync} FRESH_OUTPUT {before,after} The loopback embedding service returns controlled vectors, not model inference. """ import asyncio from datetime import datetime, timedelta, timezone import hashlib from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import json import os from pathlib import Path import platform import sqlite3 import sys import threading import headroom import headroom._core from headroom.memory.backends.local import LocalBackend, LocalBackendConfig from headroom.memory.config import EmbedderBackend, MemoryConfig, VectorBackend from headroom.memory.core import HierarchicalMemory from headroom.memory.models import Memory from headroom.memory.sync import sync, sync_import from headroom.memory.sync_adapters.codex_agent import CodexAdapter from headroom.memory.system import MemorySystem sys.stdout.reconfigure(encoding="utf-8") case, directory, expected = sys.argv[1:] assert case == "vectors" assert expected in ("before", "after") out = Path(directory).resolve() out.mkdir(parents=True, exist_ok=False) os.environ["HEADROOM_STATELESS"] = "0" os.environ["HEADROOM_WORKSPACE_DIR"] = str(out / "workspace") os.environ["HEADROOM_CCR_BACKEND"] = "memory" os.environ["HEADROOM_OFFLINE"] = "0" # Only the ephemeral loopback service is called. class Endpoint(BaseHTTPRequestHandler): dimension = 384 requests = [] def log_message(self, *_): pass def do_POST(self): payload = json.loads(self.rfile.read(int(self.headers["content-length"]))) Endpoint.requests.append({"path": self.path, "body": payload, "dimension": self.dimension}) assert self.path == "/api/embeddings", self.path body = json.dumps({"embedding": [1.0] + [0.0] * (self.dimension - 1)}).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) server = ThreadingHTTPServer(("127.0.0.1", 0), Endpoint) worker = threading.Thread(target=server.serve_forever, daemon=True) worker.start() url = f"http://127.0.0.1:{server.server_port}" async def new_memory(name): return await HierarchicalMemory.create(MemoryConfig( db_path=out / (name + ".db"), embedder_backend=EmbedderBackend.OLLAMA, embedder_model="nomic-embed-text", ollama_base_url=url, vector_backend=VectorBackend.HNSW, vector_dimension=384, auto_bubble=False, )) async def vectors_case(): memory = await new_memory("vectors") Endpoint.dimension = 768 result = {} try: try: await memory.add("Rejected incompatible embedding", user_id="alice") raise AssertionError("mismatch was accepted") except ValueError as exc: result["add_error"] = str(exc) result["rows_after_rejected_add"] = len(await memory.get_user_memories("alice")) Endpoint.dimension = 384 original = await memory.add("Valid original fact", user_id="alice") Endpoint.dimension = 768 try: await memory.supersede(original.id, "Rejected replacement fact") raise AssertionError("replacement mismatch was accepted") except ValueError as exc: result["supersede_error"] = str(exc) stored = await memory._store.get(original.id) result["original_still_active"] = stored.valid_until is None and stored.superseded_by is None result["vector_entries_after_rejected_supersede"] = memory._vector_index.size with sqlite3.connect(out / "vectors.db") as db: result["all_primary_rows"] = db.execute("SELECT COUNT(*) FROM memories WHERE user_id = 'alice'").fetchone()[0] assert result["rows_after_rejected_add"] == (0 if expected == "after" else 1), result assert result["original_still_active"] == (expected == "after"), result assert result["vector_entries_after_rejected_supersede"] == (1 if expected == "after" else 0), result assert result["all_primary_rows"] == (1 if expected == "after" else 3), result return result finally: await memory.close() try: result = asyncio.run(vectors_case()) result.update({ "case": case, "expected": expected, "python": platform.python_version(), "source": str(Path(headroom.__file__).resolve()), "native": str(Path(headroom._core.__file__).resolve()), "http_requests": Endpoint.requests, "not_tested": "Trained model inference, paid APIs, provider answer quality, non-Windows runtime, non-local memory backends.", }) (out / "result.json").write_text(json.dumps(result, indent=2), encoding="utf-8", newline="\n") print(json.dumps(result, indent=2)) finally: server.shutdown() server.server_close() worker.join(timeout=5) ``` </details> - Observed result: Main leaves one row after the rejected add; after the rejected replacement it has three primary rows, the original is no longer active and its vector was removed. The patch leaves zero rows after the rejected add, then retains exactly the valid original row and its one vector after the rejected replacement. - Not tested: General cross-component transactions for arbitrary storage/index I/O failures, automatic dimension discovery or index migration, remote vector backends, trained model inference, paid APIs, non-Windows runtime, or the complete Python suite. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing memory behavior; no new feature flag or channel expansion. - Minimum rollout channel: Normal release after maintainer review and CI qualification. - Stable/default behavior changed: Embedding shape errors are reported before primary writes. Matching vectors and explicit auto_embed=False operations retain their existing behavior. - Kill switch / disable path: Use the existing memory-disable path, or revert this commit. - Unsafe override required: No. - Qualification impact: Related tests and original local gates passed; hosted CI is tracked separately. - Rollback path: Revert this commit; no dependency or storage-schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove the fix - [x] Relevant new and existing tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Broader regression exposed update()'s object-identity contract; the final fix preserves it without changing the existing fixture. The author's review covered the initial patch and evidence; this compatibility adjustment was additionally reviewed and tested with Codex. Developed and validated with Codex assistance. The author reviewed the changes and before/after evidence. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4831" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
a6a825c55d |
fix(memory): include older memories in agent sync (#4832)
## Description The memory sync engine builds its deduplication and export sets from only the latest 500 user memories. Older rows are omitted from agent files, and an existing older fact can be imported again as a duplicate. Use the user's full snapshot for synchronization, including the refetch after an import and the standalone import/export entry points. Carry the existing MemoryFilter limit=None semantics through LocalBackend and HierarchicalMemory; ordinary queries keep their default limit of 100. Refs #4362, item 9 only. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Request the full user-scoped snapshot at each sync entry point. - Document and type the existing unbounded query option without changing default query limits. - Cover a 501-row store, older-fact deduplication, import-triggered refetch, forced/ordinary sync, idempotence and user isolation. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook) - [x] Regression tests added - [x] Standalone runtime validation completed ### Test Output ```text Final regressions against original production code: 5 failed, 2 passed pytest tests/test_memory_sync_full_store.py tests/test_memory_sync.py tests/test_memory/test_core_operations.py tests/test_memory/test_clear_scope_consistency.py -q: 57 passed, 48 skipped Original pre-commit and commit-msg hooks: passed Whole-tree Ruff check and format --check: passed Original pre-push make ci-precheck: passed Rust workspace: 1749 passed, 3 ignored Native extension: rebuilt, installed and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` The owning-suite skips require Hugging Face model files unavailable offline. Existing datetime/optional-model deprecation warnings remain visible in the logs. ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing dev venv; clean production baseline `9f2cc017863f540c61f8747b3c744a97776ebaf2`, patch `64e537fa2459d3540522bf934a32ee2e8e089ea7`. Explicit checkout on PYTHONPATH; no paid/provider endpoint. The loopback service returns controlled vectors, not trained-model inference. - Exact command / steps: Save the script below as `proof.py`. From each checkout run `python proof.py sync <fresh-output-directory> before` on main, or `python proof.py sync <fresh-output-directory> after` on the patch. Identical scripts ran outside pytest. The script initializes a real LocalBackend and SQLite store, seeds 501 chronologically ordered Alice rows plus a Bob row, and runs sync into a real CodexAdapter AGENTS.md file. It then presents the oldest existing fact from a second agent file and calls sync_import. The unpatched duplicate path makes an actual Ollama HTTP request to a loopback embedding service; the patched deduplication path makes none. <details> <summary>Standalone reproduction script</summary> ```python """Standalone real SQLite/vector/FTS/HTTP and agent-file boundary reproductions. Usage: python memory_http_proof.py {list,vectors,sync} FRESH_OUTPUT {before,after} The loopback embedding service returns controlled vectors, not model inference. """ import asyncio from datetime import datetime, timedelta, timezone import hashlib from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import json import os from pathlib import Path import platform import sqlite3 import sys import threading import headroom import headroom._core from headroom.memory.backends.local import LocalBackend, LocalBackendConfig from headroom.memory.config import EmbedderBackend, MemoryConfig, VectorBackend from headroom.memory.core import HierarchicalMemory from headroom.memory.models import Memory from headroom.memory.sync import sync, sync_import from headroom.memory.sync_adapters.codex_agent import CodexAdapter from headroom.memory.system import MemorySystem sys.stdout.reconfigure(encoding="utf-8") case, directory, expected = sys.argv[1:] assert case == "sync" assert expected in ("before", "after") out = Path(directory).resolve() out.mkdir(parents=True, exist_ok=False) os.environ["HEADROOM_STATELESS"] = "0" os.environ["HEADROOM_WORKSPACE_DIR"] = str(out / "workspace") os.environ["HEADROOM_CCR_BACKEND"] = "memory" os.environ["HEADROOM_OFFLINE"] = "0" # Only the ephemeral loopback service is called. class Endpoint(BaseHTTPRequestHandler): dimension = 384 requests = [] def log_message(self, *_): pass def do_POST(self): payload = json.loads(self.rfile.read(int(self.headers["content-length"]))) Endpoint.requests.append({"path": self.path, "body": payload, "dimension": self.dimension}) assert self.path == "/api/embeddings", self.path body = json.dumps({"embedding": [1.0] + [0.0] * (self.dimension - 1)}).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) server = ThreadingHTTPServer(("127.0.0.1", 0), Endpoint) worker = threading.Thread(target=server.serve_forever, daemon=True) worker.start() url = f"http://127.0.0.1:{server.server_port}" def backend_config(name): return LocalBackendConfig( db_path=str(out / (name + ".db")), embedder_backend="ollama", embedder_model="probe-fixed", ollama_base_url=url, vector_dimension=384, ) async def sync_case(): backend = LocalBackend(backend_config("sync")) try: await backend._ensure_initialized() now = datetime.now(timezone.utc) memories = [Memory( id=f"alice-{i:04}", content=f"Distinct memory {i}", user_id="alice", created_at=now - timedelta(seconds=i), ) for i in range(501)] memories.append(Memory(content="Bob's private fact", user_id="bob")) await backend._hierarchical_memory._store.save_batch(memories) adapter = CodexAdapter(out / "AGENTS.md") first = await sync(backend, adapter, "alice", state_path=out / "sync-state.json", force=True) exported = {m.content for m in await adapter.read_memories()} older_adapter = CodexAdapter(out / "older-agent.md") await older_adapter.write_memories([{"content": "Distinct memory 500"}]) imported = await sync_import(backend, older_adapter, "alice") result = { "first_exported": first.exported, "oldest_exported": "Distinct memory 500" in exported, "other_user_exported": "Bob's private fact" in exported, "reimported_existing_oldest": imported, "agent_file_sha256": hashlib.sha256((out / "AGENTS.md").read_bytes()).hexdigest(), } with sqlite3.connect(out / "sync.db") as db: result["alice_primary_rows"] = db.execute("SELECT COUNT(*) FROM memories WHERE user_id = 'alice'").fetchone()[0] assert result["first_exported"] == (501 if expected == "after" else 500), result assert result["oldest_exported"] == (expected == "after"), result assert result["reimported_existing_oldest"] == (0 if expected == "after" else 1), result assert result["alice_primary_rows"] == (501 if expected == "after" else 502), result assert result["other_user_exported"] is False return result finally: await backend.close() try: result = asyncio.run(sync_case()) result.update({ "case": case, "expected": expected, "python": platform.python_version(), "source": str(Path(headroom.__file__).resolve()), "native": str(Path(headroom._core.__file__).resolve()), "http_requests": Endpoint.requests, "not_tested": "Trained model inference, paid APIs, provider answer quality, non-Windows runtime, non-local memory backends.", }) (out / "result.json").write_text(json.dumps(result, indent=2), encoding="utf-8", newline="\n") print(json.dumps(result, indent=2)) finally: server.shutdown() server.server_close() worker.join(timeout=5) ``` </details> - Observed result: Main exports 500 memories, omits the oldest and imports that existing fact as a new row (502 Alice rows). The patch exports all 501, includes the oldest, imports zero duplicates and keeps 501 Alice rows. Bob's row is excluded in both runs. - Not tested: Very large-store throughput or peak memory, concurrent sync snapshot consistency, changes to the existing fingerprint sampling policy, remote memory backends, trained model inference, paid APIs, non-Windows runtime, or the complete Python suite. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing memory behavior; no new feature flag or channel expansion. - Minimum rollout channel: Normal release after maintainer review and CI qualification. - Stable/default behavior changed: Agent sync covers all user rows; ordinary get_user_memories calls keep the default 100-row limit and explicit limits still apply. - Kill switch / disable path: Use the existing memory-disable path, or revert this commit. - Unsafe override required: No. - Qualification impact: Related tests and original local gates passed; hosted CI is tracked separately. - Rollback path: Revert this commit; no dependency or storage-schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove the fix - [x] Relevant new and existing tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Synchronization already materializes a snapshot and hash set. Covering the full store makes their size scale with the user's retained memories; this patch does not introduce a paging protocol or alter the existing no-op fingerprint algorithm. Developed and validated with Codex assistance. The author reviewed the changes and before/after evidence. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4832" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
99021af20a |
fix(proxy): make the Rust front's native Bedrock and Vertex routes opt-in (#4851)
## Description
Two routes in the Rust front (`crates/headroom-proxy`) attach the host's
own cloud credentials to whatever request reaches them, without checking
the caller:
- **Bedrock native** (`/model/{id}/invoke`, `/converse`,
`/invoke-with-response-stream`, `/converse-stream`) is on by default and
signs every request with SigV4 using the AWS credentials found on the
host.
- **Vertex** (`…/publishers/anthropic/models/{m}:rawPredict` /
`:streamRawPredict`) is mounted unconditionally, replaces the caller's
`Authorization` with a bearer from the host's Google ADC, and forwards
to the generic `--upstream`, whatever host that is. So with a non-Google
upstream (the Python proxy, another provider) the host's Google token is
sent there. `--vertex-adc-scope` was parsed but never used.
This PR makes both routes explicit opt-ins, only sends the ADC bearer to
a Google API host, and uses the configured scope.
**Dependency:** these routes still do not authenticate callers
themselves. Callers are expected to be authenticated by the inbound
`HEADROOM_PROXY_TOKEN` gate added in #4840 (which also moves the default
listen address to loopback). This PR is based on `main` and does not
require #4840 to merge first, but operators who enable either route on a
non-loopback address need #4840 to restrict who can use them.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `config.rs`: `--enable-bedrock-native` /
`HEADROOM_PROXY_ENABLE_BEDROCK_NATIVE` now defaults to `false`. New
`--enable-vertex-native` / `HEADROOM_PROXY_ENABLE_VERTEX_NATIVE`
(default `false`) and `--vertex-allow-non-google-upstream` /
`HEADROOM_PROXY_VERTEX_ALLOW_NON_GOOGLE_UPSTREAM` (default `false`).
`Config::for_test` keeps both routes on so the existing route tests run
unchanged.
- `proxy.rs`: the Vertex route is mounted only when enabled; with it
off, Vertex paths go through the catch-all with the caller's own headers
(as the Python proxy's Vertex passthrough expects). The ADC token source
is built with `--vertex-adc-scope`. The "Bedrock route not mounted"
startup line is now `info`, since off is the default.
- `vertex/raw_predict.rs`: before fetching the ADC bearer, the handler
refuses (502 `vertex upstream is not a Google API host`,
`vertex_upstream_not_google` error log) unless `--upstream` is
`https://` on a `*.googleapis.com` host (regional, global and
`*.p.googleapis.com` endpoints), or the operator set
`--vertex-allow-non-google-upstream=true` (local emulators, gateways).
No token is fetched on that path.
- `RUST_DEV.md`: the three flags in the configuration table, with what
enabling them means.
**Behaviour change for operators:** deployments that used the Rust
front's native Bedrock or Vertex signing must now pass
`--enable-bedrock-native true` / `--enable-vertex-native true` (or the
env vars). Vertex with a non-Google `--upstream` additionally needs
`--vertex-allow-non-google-upstream true`. Without the flags, those
paths are forwarded to `--upstream` unsigned with the client's own
credentials (Claude Code in Vertex mode keeps sending its own ADC token,
which is what `docs/content/docs/claude-code-vertex.mdx` describes).
`scripts/qualify_cargo_dependencies.py` already passes
`--enable-bedrock-native true`.
## Testing
- [x] `cargo test -p headroom-proxy` (45 test binaries, 596 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes
New tests in `tests/integration_cloud_routes_opt_in.rs` build the config
from the real CLI defaults (`CliArgs` → `Config::from_cli`), give the
app host AWS credentials and a host ADC bearer, and record what a
`wiremock` upstream receives:
- `default_config_does_not_sign_bedrock_requests`: no `AWS4-HMAC-SHA256`
Authorization on `invoke`, `invoke-with-response-stream`, `converse`.
- `default_config_never_attaches_host_adc_bearer`: `rawPredict` and
`streamRawPredict` reach the upstream without the host bearer.
- `vertex_bearer_is_not_sent_to_a_non_google_upstream`: with the route
enabled and an `http://127.0.0.1` upstream, 502 and the upstream sees
nothing.
- `vertex_opt_in_still_signs_when_non_google_upstream_is_allowed`: the
explicit escape hatch still works.
- `configured_adc_scope_reaches_token_source`.
- Unit test `vertex::raw_predict::tests::google_api_host_check`
(`http://`, `googleapis.com.example.net`, `example-googleapis.com`,
loopback are rejected).
### Test Output
Before (the three tests that compile against `main`, run with this PR's
`src/` changes stashed):
```text
test configured_adc_scope_reaches_token_source ... FAILED
test default_config_never_attaches_host_adc_bearer ... FAILED
test default_config_does_not_sign_bedrock_requests ... FAILED
test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.77s
```
After:
```text
$ cargo test -p headroom-proxy --test integration_cloud_routes_opt_in
test configured_adc_scope_reaches_token_source ... ok
test vertex_opt_in_still_signs_when_non_google_upstream_is_allowed ... ok
test vertex_bearer_is_not_sent_to_a_non_google_upstream ... ok
test default_config_never_attaches_host_adc_bearer ... ok
test default_config_does_not_sign_bedrock_requests ... ok
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.69s
```
## Real Behavior Proof
- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (
|
||
|
|
d077263012 |
fix(openclaw): preserve assistant text blocks when tool arguments change (#4829)
## Description
OpenClaw history can contain text on both sides of an assistant tool
call. If the compression response changes the call arguments while
leaving the wire text intact, restoring that message currently moves
both pieces of text ahead of the call and joins them into a single
block.
## Type of Change
- [x] Bug fix
## Changes Made
- Compare the returned assistant text with the text sent for that
message. Keep the original blocks when the text matches, then apply the
changed arguments to the matching tool call.
- Extend the assemble regression to check the order and content of both
text blocks, the signed thinking block, and the updated tool call.
## Testing
- [x] Regression and existing OpenClaw tests pass.
- [x] Manual testing performed.
### Test Output
```text
cd plugins/openclaw
npm test # 11 files, 162 tests passed
npm run typecheck # passed
npm run build # passed
git diff --check # passed
```
The repository commit hooks and Rust formatting check passed. The full
local pre-push suite stopped at Rust dependency resolution after the
registry fetch stalled. I used the hook's documented push override for
this TypeScript-only change. The verification above covers the changed
plugin.
## Real Behavior Proof
- Environment: macOS arm64, Node 22.22.2, built OpenClaw plugin with the
pinned JavaScript SDK, and a Node HTTP server listening on loopback. The
payload was synthetic and did not need a model.
- Exact command / steps: Run `npm run build` in `plugins/openclaw` on
base `9f2cc01` and the patched tree. In Node, import
`HeadroomContextEngine` from the built `dist/index.js`. Start a loopback
server that answers `GET /v1/retrieve/stats` with `{"store":{}}`. For
`POST /v1/compress`, echo the submitted messages and set the first tool
call's argument string to `{"id":7}`. Return `tokens_before: 100`,
`tokens_after: 25`, and `tokens_saved: 75`. Create the engine with that
`proxyUrl`, `autoStart: false`, and `minContextChars: 0`. Await
`ensureProxyUrl()`, then call `assemble()` with one assistant message
whose blocks are signed thinking, text `Checking.`, a `lookup` call with
ID `lookup-1` and arguments `{id: 7, notes: "fixture ".repeat(80)}`, and
text `Awaiting result.`.
- Observed result: the SDK sent one actual compression request in both
runs. Base returned `thinking, text("Checking.Awaiting result."),
toolCall({id:7})`. The patch returned `thinking, text("Checking."),
toolCall({id:7}), text("Awaiting result.")`. The signature and call ID
were preserved, and the engine reported 25 tokens in both runs.
- Not tested: a remote proxy, model calls, or an OpenClaw host process.
The runtime proof exercised the built plugin and SDK over HTTP with a
deterministic compression response.
## Runtime Rollout Safety
- Rollout-managed feature(s): OpenClaw context restoration.
- Minimum rollout channel: existing plugin channel.
- Stable/default behavior changed: text block placement is preserved
when only assistant tool arguments change.
- Kill switch / disable path: existing OpenClaw plugin enablement.
- Unsafe override required: none.
- Qualification impact: OpenClaw plugin suite and loopback verification
above.
- Rollback path: revert the commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
This is a small correction to the restoration path introduced by #3826.
AI assistance was used to investigate, implement, and verify the change.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4829"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: 123XieC <323077714+123XieC@users.noreply.github.com>
|
||
|
|
d58b3ef19b |
fix(cache): clear older skips when storing successful results (#4819)
## Description
Concurrent compression requests can finish with different verdicts for
the same content key. If a no-op marks the key as skipped before another
request stores a successful result, the old put leaves that skip in
place. The next request returns the original even though the successful
compression is present in memory and SQLite.
Remove the older skip under the existing memory lock when admitting a
successful result. First-sight mark_skip and explicit move_to_skip
retain their original behavior. The initial PR also routed mark_skip
through disk invalidation; review identified the unnecessary DELETEs
that added to new non-compressible blocks, so that part has been
removed.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Clear an older skip when a successful result is admitted.
- Cover both existing skip APIs with memory-only and SQLite-backed
caches.
- Trace real SQLite statements to preserve the memory-only first-sight
skip path.
## Testing
- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook in WSL)
- [x] Regression tests added
- [x] Standalone runtime validation completed
### Test Output
```text
Four successful-result regressions failed on original main.
The added first-sight SQL regression failed on the initial PR and passes after review revision.
pytest tests/test_router_result_cache.py tests/test_router_cache_policy.py tests/test_transforms/test_content_router.py tests/test_content_router_concurrency.py tests/test_compression_determinism.py tests/test_cache tests/test_cold_prefix.py:
561 passed, 3 skipped
Original pre-commit and commit-msg hooks: passed
Whole-tree Ruff check and format --check: passed
Original pre-push make ci-precheck: passed
Rust workspace: 1749 passed, 3 ignored
Native extension: rebuilt, installed and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```
Optional-model skips and the existing token-count approximation warning
remain visible in the gate logs.
## Real Behavior Proof
- Environment: Windows 11, CPython 3.13.9, existing dev venv; clean
baseline `c05fbdddc873078dd97cb9ede841e75ef7ed0bef`, patch
`b7f8872ac59c41abd12e2274364bcfecba67ceb7`. Explicit checkout on
PYTHONPATH; no provider credentials. The proof enables loopback HTTP;
model downloads remain disabled.
- Exact command / steps: Save the following script as `proof.py` and run
`python proof.py` from either checkout with `PYTHONPATH=.`. Identical
before/after scripts ran outside pytest.
```python
"""Two actual router requests with real remote HTTP, then SQLite replay."""
from concurrent.futures import ThreadPoolExecutor
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
from pathlib import Path
import tempfile
import threading
from headroom.cache.result_store import ResultStore
from headroom.tokenizers.estimator import CharacterCounter
from headroom.transforms.content_router import CompressionCache, ContentRouter, ContentRouterConfig
os.environ['HEADROOM_CCR_BACKEND']='memory'
os.environ['HEADROOM_OFFLINE']='0' # This script binds and calls loopback only.
os.environ['HEADROOM_FREEZE_BLOCK_DECISION']='0'
first_seen, release_first = threading.Event(), threading.Event()
lock=threading.Lock();calls=[]
short='The deployment is healthy and the requested records are present.'
class Handler(BaseHTTPRequestHandler):
def do_POST(self):
payload=json.loads(self.rfile.read(int(self.headers['Content-Length'])))
with lock:
index=len(calls);calls.append(payload)
if index==0:
first_seen.set()
if not release_first.wait(15): raise RuntimeError('controller did not release response')
text=short
else:
text=payload['content']
data=json.dumps({'compressed':text,'original_tokens':len(payload['content'].split()),'compressed_tokens':len(text.split()),'compression_ratio':0.1 if index==0 else 1.0}).encode()
self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
def log_message(self,*args):pass
server=ThreadingHTTPServer(('127.0.0.1',0),Handler)
thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()
os.environ['HEADROOM_KOMPRESS_ENDPOINT']=f'http://127.0.0.1:{server.server_port}'
payload=[{'role':'tool','tool_call_id':'t1','content':'The service contains useful deployment observations for the current investigation. '*80}]
tokenizer=CharacterCounter()
try:
with tempfile.TemporaryDirectory(prefix='headroom-tier-proof-') as d:
router=ContentRouter(ContentRouterConfig(ccr_inject_marker=False, result_cache_disk=False))
disk=ResultStore(Path(d)/'cache.db',1800)
router._cache=CompressionCache(store=disk,namespace='proof:')
def apply():return router.apply(payload,tokenizer,force_kompress=True).messages[0]['content']
# The real HTTP client is built once before sharing the router.
router._get_kompress()
with ThreadPoolExecutor(max_workers=2) as pool:
earlier=pool.submit(apply)
assert first_seen.wait(10),'first router call did not reach HTTP'
later=pool.submit(apply)
noop=later.result(timeout=10)
release_first.set()
compressed=earlier.result(timeout=10)
replay=apply()
reopened=ContentRouter(ContentRouterConfig(ccr_inject_marker=False,result_cache_disk=False))
reopened._cache=CompressionCache(store=ResultStore(Path(d)/'cache.db',1800),namespace='proof:')
disk_replay=reopened.apply(payload,tokenizer,force_kompress=True).messages[0]['content']
print(json.dumps({'noop_is_original':noop==payload[0]['content'],'successful_request_is_compressed':compressed==short,'next_request_reuses_success':replay==short,'reopened_sqlite_reuses_success':disk_replay==short,'skip_entries':router._cache.skip_size,'http_calls':len(calls)},indent=2))
reopened._cache._store._db.close()
disk._db.close()
router._get_kompress()._client.close()
finally:
release_first.set();server.shutdown();server.server_close();thread.join(5)
```
The first-sight SQLite trace was also run outside pytest with this
script:
```python
"""Trace actual SQLite statements for first-sight negative cache entries."""
import json
from pathlib import Path
import tempfile
from headroom.cache.result_store import ResultStore
from headroom.transforms.content_router import CompressionCache
with tempfile.TemporaryDirectory(prefix='headroom-skip-io-') as directory:
store=ResultStore(Path(directory)/'cache.db',1800)
cache=CompressionCache(store=store)
statements=[]
store._db.set_trace_callback(statements.append)
for i in range(100):cache.mark_skip(f'new-{i}')
print(json.dumps({'new_skip_entries':cache.skip_size,'sqlite_statements':len(statements),'deletes':sum(s.startswith('DELETE ') for s in statements)}))
store._db.close()
```
- Observed result: Two real ContentRouter.apply calls reach a bound
loopback RemoteKompress endpoint. The controller lets the no-op finish
first, then releases the successful compression. On clean main,
next_request_reuses_success=false and skip_entries=1, while reopening
SQLite reads the successful result. With the patch both the next request
and reopened cache reuse that result, skip_entries=0, and the endpoint
still receives only the original two calls. The trace script records 100
DELETEs for 100 new skips on the initial PR, versus zero SQL statements
after this revision.
- Not tested: Trained-model quality, paid endpoints, production
throughput, cross-process in-memory coherence, network filesystems, or
the complete Python suite. HTTP responses are deterministic synthetic
fixtures; actual router, HTTP client and SQLite operations are used.
## Runtime Rollout Safety
- Rollout-managed feature(s): Existing cache behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: A successful store supersedes an
older skip. First-sight negative classification keeps its existing
memory-only path.
- Kill switch / disable path: Revert this commit to restore the prior
successful-store behavior; no new configuration is introduced.
- Unsafe override required: No.
- Qualification impact: Related tests and original local gates passed;
hosted CI is tracked separately.
- Rollback path: Revert this commit; no dependency or storage-schema
change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Developed and validated with Codex assistance. I reviewed the initial
changes and runtime evidence; the review follow-up was checked with
Codex.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4819"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
48f650ad04 |
test: make the trust_remote_code scan see every truthy value and run without git (#4785)
## Description
`tests/test_no_trust_remote_code.py` scans the tracked tree so that
nothing re-enables HuggingFace remote code execution. Its regex captured
the assigned value only when it started with a letter
(`([A-Za-z_][\w.]*)`), but the library tests the flag for truthiness, so
these also enable it and were not seen: `trust_remote_code=1`,
`trust_remote_code=(True)`, `trust_remote_code="yes"`, `setattr(cfg,
"trust_remote_code", True)` and `kwargs.setdefault("trust_remote_code",
True)`. The test also called `pytest.skip` when `git ls-files` failed
(for example in an exported tarball), so the guard silently stopped
running.
This PR widens the value rule (a quoted string, or a name or number,
with an optional leading `(`, treating anything except the exact name
`False` as enabling the flag), adds a second pattern for the `"name",
value` form, matches across a line break, and falls back to scanning
`headroom/**/*.py` when git is unavailable instead of skipping. The scan
of the current tree finds no offender with the wider rules.
Part of #4783
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `tests/test_no_trust_remote_code.py`: `_VALUE`, a wider `ASSIGNMENT`,
a new `NAMED_ARGUMENT` limited to
`setattr`/`getattr`/`setdefault`/`get`/`pop` calls, a `_values()` helper
that reports `(line, value)`, a git-less fallback in `_tracked_files()`,
and 17 new cases: calls that do not set the flag (`log.info("...",
True)`) must be ignored, every truthy shape must be flagged, a value on
the next line must be flagged, and an explicit `False` (including
`(False)`, `kwargs["..."] = False` and `setattr(..., False)`) must be
accepted.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
`mypy headroom` was not run (no change under `headroom/`). `ruff check`
and `ruff format` (0.17.3) are clean.
### Test Output
The old regex on the shapes the old test never saw (`re.findall` with
the old pattern):
```text
old regex matches=True trust_remote_code=True
old regex matches=False trust_remote_code=1
old regex matches=False trust_remote_code=(True)
old regex matches=False trust_remote_code="yes"
old regex matches=False setattr(cfg, "trust_remote_code", True)
old regex matches=False kwargs.setdefault("trust_remote_code", True)
```
The new tests with the old scan rule (value must start with a letter, no
`"name", value` pattern):
```text
6 failed, 10 passed (the =1, ="yes", mapping =1, setattr, setdefault and setattr(..., False) cases fail)
```
The new tests with this branch's scan:
```text
tests/test_no_trust_remote_code.py 19 passed (the tree scan finds no offender)
same file run from a copy of the tree without .git 19 passed, 0 skipped
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, pytest 9.2.0, `main` at
`3c66c91e`.
- Exact command / steps: `python -m pytest
tests/test_no_trust_remote_code.py`; then the same file from a copy of
the repository without `.git`; then the new tests with the scan logic
swapped for the old rule.
- Observed result: as above. The tree scan passes on the current tree
with the wider rules, so no existing code needs to change.
- Not tested: CI, and a deliberately injected `trust_remote_code=1` in a
real module (the shapes are covered by the unit cases instead).
## Runtime Rollout Safety
- Rollout-managed feature(s): none; test only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for the library and proxy. The
scan is stricter and no longer skips without git.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (this
file; the whole suite was not run)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- A value such as `"False"` (a string) is truthy in Python, so the scan
flags it; I think that is right for this guard.
- The git-less fallback scans only `headroom/`: an exported tarball has
no other tracked-file list.
AI assistance: this change, its tests and the runs above were done with
Claude (Sonnet 5.5), operated by me (HetCreep).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4785"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
dc94c46a1f |
test: remove the developer's GitHub and Copilot tokens from every test's environment (#4790)
## Description Any test that enters the proxy lifespan starts the Copilot quota poller (`headroom/subscription/copilot_quota.py`), which reads `GITHUB_COPILOT_GITHUB_TOKEN`, `GITHUB_TOKEN`, `COPILOT_GITHUB_TOKEN` or `GITHUB_COPILOT_API_TOKEN` from the environment and sends it as a `Bearer` header to `api.github.com`. `tests/conftest.py` scrubs only `HEADROOM_*`, so a developer with `GITHUB_TOKEN` in the shell has that token used by the test run. `tests/test_subscription_poll_credential_policy.py:1-8` already treats using a credential nobody meant for this purpose as a defect. This PR adds an autouse fixture that removes the nine Copilot/GitHub token variables that `copilot_quota.py` and `copilot_auth.py` read. Tests that need one still set it with `monkeypatch.setenv`, which runs after the fixture. Part of #4789 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `tests/conftest.py`: `_COPILOT_TOKEN_ENV_VARS` and the autouse fixture `_scrub_developer_copilot_tokens`, placed before `_scrub_developer_headroom_env`. - `tests/test_conftest_scrubs_copilot_tokens.py`: five names must be absent from `os.environ`, `discover_github_token()` finds nothing, and a test can still set its own token. No product code changes. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `mypy headroom` was not run (no change under `headroom/`). `ruff check` and `ruff format --check` (0.17.3) are clean on the two files. ### Test Output With `GITHUB_TOKEN` and `GH_TOKEN` set to dummy values in the shell, the new file: ```text this branch: 7 passed conftest from main: 3 failed, 4 passed (GITHUB_TOKEN and GH_TOKEN still in the environment, and discover_github_token() returns the dummy token) ``` The Copilot and GitHub-token related test files, with the same dummy variables set: ```text test_cli/test_wrap_copilot.py, test_cli/test_wrap_opencode.py, test_copilot_auth.py, test_copilot_quota.py, test_offline_egress_chokepoint.py this branch: 1 failed, 435 passed, 4 skipped conftest from main: 1 failed, 435 passed, 4 skipped test_release_workflows.py, test_subscription_poll_credential_policy.py, test_copilot_subscription_smoke.py this branch: 86 passed ``` The one failure is `tests/test_offline_egress_chokepoint.py::TestUpdateCheckOffline::test_explicit_update_check_never_constructs_request` (`AttributeError: module 'headroom.cli.main' has no attribute 'name'`). It fails the same way with `main`'s `conftest.py` and passes when run alone with either, so it depends on test order and is not caused by this change; I did not investigate it. ## Real Behavior Proof - Environment: Windows 11, Python 3.12, pytest 9.2.0, the compiled `headroom._core`, `main` at `3c66c91e`. - Exact command / steps: `GITHUB_TOKEN=ghp_dummy_not_real GH_TOKEN=dummy2 python -m pytest tests/test_conftest_scrubs_copilot_tokens.py`, once with this branch's `conftest.py` and once with `main`'s; then the two groups of files above with the same variables. - Observed result: as in Test Output. Only dummy values were used, never a real token (the runs of the other files with the dummy variables may have sent them to GitHub, where they are rejected). - Not tested: CI, Linux and macOS, the whole suite, and a network capture of the poller (the 53 lookups in #4789 come from an instrumented run, not from this PR's proof). ## Runtime Rollout Safety - Rollout-managed feature(s): none; test only. - Minimum rollout channel: n/a - Stable/default behavior changed: no. - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes (the files listed above; the whole suite was not run, and one order-dependent failure exists on `main`) - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Not in this PR: the other items of #4789 (credential readers in `test_copilot_auth.py`, writes to `~/.headroom`, tests that reach the network). - If a CI job relies on a token reaching a test, that test would now need `monkeypatch.setenv`; I found none in the files above. AI assistance: this change, its tests and the runs above were done with Claude (Sonnet 5.5), operated by me (HetCreep). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4790" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
327d92414f |
fix(cache): keep literal cache_control fields in the response-cache key (#4799)
## Description The proxy response-cache key policy strips `cache_control` recursively at every dict depth, with a single carve-out for JSON Schema `properties` names (#327). But `cache_control` is overloaded: on a content block, a `system` entry or a tool definition it is a transport prompt-cache directive, while inside tool arguments and JSON Schema literals it is ordinary application data the model actually sees. So two requests that differ only in that data get the same key. With `cache_enabled` on by default, the second caller is replayed the first caller's response for the full TTL. Reproduced on `main` (`e995eb9f`) against the real `SemanticCache` API, no mock: ```python msgs = [{"role": "assistant", "content": [{"type": "tool_use", "id": "c1", "name": "configure", "input": {"cache_control": "first"}}]}] changed = copy.deepcopy(msgs); changed[0]["content"][0]["input"]["cache_control"] = "second" c = SemanticCache() await c.set(msgs, "m", b'{"id":"first"}', {}, partition="caller") print("replayed:", (await c.get(changed, "m", partition="caller")) is not None) ``` Before (this branch's parent, `e995eb9f`): ```text changed tool payload replayed: True changed schema replayed: True ``` Closes #4272 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/semantic_cache_key_policy.py`: `strip_cache_control` no longer recurses into fields whose value is opaque payload — `input`, `arguments`, `input_schema`, `parameters`, `schema`, `properties`, `const`, `default`, `enum`, `example`, `examples`, `text` — so a `cache_control`-named field there stays in the key. Everywhere else (content blocks, `system`, tool definitions) it is still dropped, so a moved transport breakpoint still does not fragment the key. The narrower `properties`-only carve-out falls out of the general rule. - `tests/test_proxy_semantic_cache_key_policy.py`: covers the mandatory-partition precondition (`ValueError` when no partition is passed), which was the one uncovered line Codecov flagged in this change. - `tests/test_semantic_cache_literal_payloads.py`: new regression suite covering tool `input`, tool `input_schema` `default`, and `response_format` schema `const`, each through the real `SemanticCache` (changed request misses, original still hits), plus a guard for the decoded-dict `function.arguments` shape Headroom accepts (`handlers/openai.py` parses either), a guard that OpenAI's JSON-*string* `arguments` is passed through verbatim, and guards that transport `cache_control` is still ignored and nested transport annotations are still dropped. One pure module changed; no provider, no native rebuild. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `mypy` is unchecked because the pre-commit hook fails on 11 pre-existing Windows-only `attr-defined` errors in files this PR does not touch (`headroom/fileperms.py`, `headroom/ccr/mcp_server.py`, `headroom/proxy/interceptors/astgrep.py`: `os.fchmod`, `os.O_DIRECTORY`, `fcntl`); neither changed file produces a mypy error. ### Test Output Regression tests fail before the fix. With `headroom/proxy/semantic_cache_key_policy.py` reverted and the new tests kept: ```text $ git show HEAD~1:headroom/proxy/semantic_cache_key_policy.py > headroom/proxy/semantic_cache_key_policy.py $ uv run pytest tests/test_semantic_cache_literal_payloads.py -q FAILED tests/test_semantic_cache_literal_payloads.py::test_literal_cache_control_does_not_alias_cached_responses[tool_input] FAILED ... [tool_arguments_dict] FAILED ... [tool_schema] FAILED ... [response_schema] FAILED tests/test_semantic_cache_literal_payloads.py::test_literal_cache_control_changes_the_key[tool_input] FAILED ... [tool_arguments_dict] FAILED ... [tool_schema] FAILED ... [response_schema] FAILED tests/test_semantic_cache_literal_payloads.py::test_strip_cache_control_still_drops_nested_transport_annotations 9 failed, 3 passed in 0.64s ``` The three that pass against the old policy are the guards: the OpenAI string-`arguments` case (a string is never rewritten), the schema-property-name case (#327), and the moved-transport-breakpoint case. After the fix: ```text $ uv run pytest tests/test_semantic_cache_literal_payloads.py tests/test_proxy_semantic_cache_key_policy.py \ tests/test_semantic_cache_key_policy.py tests/test_proxy_response_cache_partition.py \ tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_integration.py \ tests/test_openai_response_cache_key.py tests/test_proxy_openai_cache_key_integration.py -q 94 passed, 1 warning in 28.39s $ uv run pytest <the four policy/partition files> -q \ --cov=headroom.proxy.semantic_cache_key_policy --cov-report=term-missing --cov-branch Name Stmts Miss Branch BrPart Cover headroom\proxy\semantic_cache_key_policy.py 54 0 16 0 100% ``` $ uv run pytest tests/test_cache/test_semantic.py tests/test_openai_response_cache_key.py \ tests/test_proxy_openai_cache_key_integration.py tests/test_proxy_response_cache_store_guard.py \ tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_integration.py \ tests/test_shared_context.py -q 133 passed, 1 warning in 34.11s $ uv run pytest tests/test_cache_control_move_bust.py tests/test_anthropic_fork_lineage_cache.py \ tests/test_cross_turn_cache_safety.py tests/test_cache/test_prefix_tracker.py -q 113 passed, 1 warning in 6.21s $ uv run ruff check headroom/proxy/semantic_cache_key_policy.py tests/test_semantic_cache_literal_payloads.py All checks passed! $ uv run ruff format --check <both files> 1 file already formatted ``` ## Real Behavior Proof - Environment: Windows 11, `uv run` against the project venv (Python 3.13.11), branch `fix/4272-literal-payload-cache-keys` cut from upstream `main` `e995eb9f`. No mocked cache and no provider: the reproducer calls the real `SemanticCache.set`/`get` in `headroom/proxy/semantic_cache.py`. - Exact command / steps: run the reproducer above as a script under `uv run python` — store a response under a tool input containing `{"cache_control": "first"}`, deep-copy the messages, change that value to `"second"`, then `get` with the changed messages; repeat for a tool `input_schema` `default` and a `response_format` schema `const`. - Observed result: before the fix the changed request hit the cache in both cases (`changed tool payload replayed: True`, `changed schema replayed: True`). After the fix both print `False`, and the original request still hits (asserted in each regression test), so the key was not merely widened. - Not tested: the real OpenAI HTTP path with a decoded-dict `arguments` field, which is a nonstandard client encoding (`arguments` is normally a JSON string, covered by `test_openai_arguments_string_is_not_rewritten`); the full `make ci-precheck` (Rust/native rebuild) and the end-to-end proxy HTTP path; the change is confined to the pure key policy, which the tests above exercise directly through the real `SemanticCache`. `headroom/cache/prefix_tracker.py` has the same blunt recursive strip, but it serves prefix-content equality rather than response replay and is left as a separate follow-up. ## Runtime Rollout Safety - Rollout-managed feature(s): none; the proxy response cache (`cache_enabled`) is not a rollout-gated feature. - Minimum rollout channel: n/a - Stable/default behavior changed: yes, deliberately — for requests whose tool data or JSON Schema values contain a `cache_control`-named field, two distinct requests that used to share one key now get distinct keys. Requests without such a field produce the same key as before, so hit rate there is unchanged. - Kill switch / disable path: `cache_enabled = false`, or the existing proxy config for the response cache; no new switch. - Unsafe override required: no - Qualification impact: none; no rollout-managed surface touched. - Rollback path: revert this commit. Cache keys are process-local (`SemanticCache` is in-memory), so a revert needs no cache migration or invalidation. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — not needed: the key policy's docstrings describe the new rule, and no user-facing docs claim `cache_control` is stripped from payloads. - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes `headroom/cache/prefix_tracker.py:165` has the same blunt recursive strip, used for prefix-content equality rather than response replay. Worth a follow-up, deliberately out of scope here so the response-replay fix stays a one-module change. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
973c12d2af |
fix(cache): replay every parallel tool result (#4818)
## Description
An Anthropic user message can hold several parallel `tool_result`
blocks. The session compression cache recorded and replayed only the
first one, but a hit for that result could freeze the whole message.
Later results then reached the provider in their raw form instead of the
previously compressed form.
Visit every result when recording, replaying and marking stable content.
Freeze a message only if every result is replayable or textless. Reuse
the existing single-result shape guard, so images, tool references,
multiblock boundaries and text-block metadata retain their established
protection. Cache seeding checks the paired tool-call IDs before
associating original and compressed text.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Cache and replay each parallel result independently without mutating
caller input.
- Stop the frozen prefix on a later miss or unsupported text layout.
- Cover a textless first result, later text-block metadata, changed IDs
and partial replay. Update the existing later-result limitation test to
assert actual replay.
## Testing
- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook)
- [x] Regression tests added
- [x] Standalone runtime validation completed
### Test Output
```text
New regressions against original production code: 5 failed, 1 passed
pytest tests/test_compression_cache.py tests/test_session_engine.py tests/test_cache_replay_shapes.py tests/test_token_headroom_mode.py tests/test_cache_prefix_safety.py tests/test_proxy_anthropic_cache_stability.py tests/test_prefix_tracker_restart.py:
189 passed
Original pre-commit and commit-msg hooks: passed
Whole-tree Ruff check and format --check: passed
Original pre-push make ci-precheck: passed
Rust workspace: 1749 passed, 3 ignored
Native extension: rebuilt, installed and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```
Optional-model skips and the existing token-count approximation warning
remain visible in the gate logs. The restart tests also report the
environment's existing Starlette/httpx deprecation warning.
## Real Behavior Proof
- Environment: Windows 11, CPython 3.13.9, existing dev venv; clean
baseline `c05fbdddc873078dd97cb9ede841e75ef7ed0bef`, patch
`a09ad47dc8ceb04b42858a67eecb83a7ee246485`. Explicit checkout on
PYTHONPATH; no provider credentials. Headroom runs offline; the parallel
proof uses a real SDK to send only to its bound loopback receiver.
- Exact command / steps: Save the following script as `proof.py` and run
`python proof.py` from either checkout with `PYTHONPATH=.`. Identical
before/after scripts ran outside pytest.
```python
"""Production cache/session APIs followed by real Anthropic SDK loopback HTTP."""
import copy
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import threading
from anthropic import Anthropic
from headroom.cache.compression_cache import CompressionCache
from headroom.proxy.session_engine import FREEZE_POLICY_REPLAYABLE, prepare_turn
captured = []
class Handler(BaseHTTPRequestHandler):
def do_POST(self):
captured.append(json.loads(self.rfile.read(int(self.headers['Content-Length']))))
data = json.dumps({'id':'local','type':'message','role':'assistant','model':'local-fixture','content':[{'type':'text','text':'received'}],'stop_reason':'end_turn','stop_sequence':None,'usage':{'input_tokens':1,'output_tokens':1}}).encode()
self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
def log_message(self,*args): pass
def message(a,b):
return {'role':'user','content':[{'type':'tool_result','tool_use_id':'t1','content':a},{'type':'tool_result','tool_use_id':'t2','content':b}]}
server = ThreadingHTTPServer(('127.0.0.1',0),Handler)
thread = threading.Thread(target=server.serve_forever,daemon=True);thread.start()
raw = message('first original '*20,[{'type':'text','text':'second original '*20,'cache_control':{'type':'ephemeral'}}])
short = message('first compressed',[{'type':'text','text':'second compressed','cache_control':{'type':'ephemeral'}}])
messages=[{'role':'assistant','content':[{'type':'tool_use','id':'t1','name':'inspect','input':{}},{'type':'tool_use','id':'t2','name':'inspect','input':{}}]},raw,{'role':'user','content':'continue'}]
before = copy.deepcopy(messages)
cache = CompressionCache();cache.update_from_result([raw],[short])
prep = prepare_turn(cache,messages,policy=FREEZE_POLICY_REPLAYABLE)
try:
with Anthropic(api_key='local-fixture',base_url=f'http://127.0.0.1:{server.server_port}',max_retries=0) as client:
client.messages.create(model='local-fixture',max_tokens=16,messages=prep.pipeline_input)
wire = captured[-1]['messages'][1]
print(json.dumps({'all_results_replayed':wire==short,'second_result':wire['content'][1]['content'],'frozen_count':prep.frozen_message_count,'caller_unchanged':messages==before,'requests':len(captured)},indent=2))
finally:
server.shutdown();server.server_close();thread.join(5)
```
- Observed result: With the same original/compressed message pair, clean
main freezes the two-message prefix but the real Anthropic SDK sends the
second result as its original text. With the patch, the HTTP receiver
gets both compressed results, including the second text block's
cache_control; frozen_count remains 2 and caller_unchanged remains true.
- Not tested: Live Anthropic service acceptance, model inference,
billing, task accuracy, or the complete Python suite. The receiver
records actual SDK JSON and returns a synthetic response.
## Runtime Rollout Safety
- Rollout-managed feature(s): Existing cache behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Cached replay covers every tool
result in a message; a later cache miss or unsupported shape keeps that
message mutable.
- Kill switch / disable path: Revert this commit to restore the prior
replay behavior; no new configuration is introduced.
- Unsafe override required: No.
- Qualification impact: Related tests and original local gates passed;
hosted CI is tracked separately.
- Rollback path: Revert this commit; no dependency or storage-schema
change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Developed and validated with Codex assistance. I reviewed the changes
and runtime evidence.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4818"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
d7a2fcf092 |
test: make the loopback-gating tests able to tell the guard from a missing key (#4784)
## Description
`tests/test_proxy_loopback_gating.py` says non-loopback callers get a
404 on the gated routes, but `test_non_loopback_caller_gets_404` used a
default `TestClient`, whose peer is the string `"testclient"` (not an
IP), and asserted only the 404 status. Two of the twelve gated routes,
`/v1/telemetry/tools/{hash}` and `/v1/toin/pattern/{prefix}`, answer 404
on their own when the key is absent, and `test_loopback_caller_allowed`
accepts `{200, 404, 422}`. So removing the guard from those routes, or
making it refuse every parseable IP, did not turn the test red.
The guard's denial body is `{"detail": "Not Found"}` (what FastAPI
answers for an unknown path); a detail route answers with its own
message. This PR (1) uses a routable remote peer (`203.0.113.9`,
TEST-NET-3) with a loopback `Host`, so only the client-IP check can
refuse it, (2) asserts the guard's exact denial body, and (3) asserts
that a loopback caller never receives that body.
Part of #4783
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `tests/test_proxy_loopback_gating.py`: a `GUARD_DENIAL` constant with
a comment saying why; `test_non_loopback_caller_gets_404` uses
`client=("203.0.113.9", 4242)` and `base_url="http://127.0.0.1"` and
asserts `resp.json() == GUARD_DENIAL`; `test_loopback_caller_allowed`
also asserts `resp.json() != GUARD_DENIAL`. No product code changes.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
`mypy headroom` was not run (no change under `headroom/`). `ruff check`
and `ruff format --check` (0.17.3) are clean.
### Test Output
Whole file, product unchanged:
```text
pytest tests/test_proxy_loopback_gating.py 147 passed
```
The same file run against two mutations of the product (each restored
afterwards), with the old tests (`origin/main`) and the new ones:
```text
Mutation A: loopback_guard.py `return address.is_loopback` -> `return True`
old tests: 20 failed, 127 passed
new tests: 32 failed, 115 passed (the 12 non-loopback cases now fail too)
Mutation B: remove `dependencies=[Depends(_require_loopback)]` from
/v1/telemetry/tools/{signature_hash} and /v1/toin/pattern/{hash_prefix:path}
old tests: 147 passed (nothing noticed)
new tests: 2 failed, 145 passed (the two routes)
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, pytest 9.2.0, the compiled
`headroom._core`, `main` at `3c66c91e`.
- Exact command / steps: `create_app(ProxyConfig(...))` and three
`TestClient`s (default peer, `client=("203.0.113.9", 4242)`,
`client=("127.0.0.1", 12345)`, the last two with
`base_url="http://127.0.0.1"`); `GET` each of
`/v1/telemetry/tools/example`, `/v1/toin/pattern/example`,
`/v1/feedback/example`; then the two mutations above.
- Observed result: both non-loopback peers got `404 {"detail":"Not
Found"}` on all three; the loopback peer got `404 {"detail":"No
telemetry found for signature: example"}`, `404 {"detail":"No TOIN
pattern found with hash starting with: example"}` and `200`
respectively. The mutation results are in Test Output.
- Not tested: CI, other platforms, and the two non-GET gated routes
under mutation B (they are covered by the new body assertion but I did
not mutate them).
## Runtime Rollout Safety
- Rollout-managed feature(s): none; test only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (this
file; the whole suite was not run)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- The vanilla `TestClient` (peer `"testclient"`) is also refused by the
guard; I did not keep a test for it because it is not a
loopback-versus-remote distinction.
AI assistance: this change, its tests and the runs above were done with
Claude (Sonnet 5.5), operated by me (HetCreep).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4784"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
854a696544 |
fix(proxy): retry a TLS error only while the request is being written (#4776)
## Description A corrupt TLS record after the handshake surfaces as a raw `ssl.SSLError` (for example `[SSL: RECORD_LAYER_FAILURE] record layer failure`, or the `SSLV3_ALERT_BAD_RECORD_MAC` bursts in #1615). httpcore maps `ssl.SSLError` to `ConnectError` only in `start_tls`; its `read`/`write` maps cover only anyio's resource errors and timeouts, and anyio's `TLSStream` re-raises a non-EOF `SSLError` as it is. So the error is not an `httpx.TransportError`, and the #1639 retry in `_retry_request` and `_stream_response` lets it through. That error alone does not say whether upstream received the request. This PR retries it only when httpcore reports that writing the request failed: its documented `trace` request extension emits `send_request_headers.failed` or `send_request_body.failed`. Upstream then never had a complete request to run. A TLS error raised after the request was written propagates, as on main, so a POST upstream may have accepted is never sent twice. In the streaming path it still ends as the SSE 502, now with `upstream_tls_error` instead of a generic `api_error`. What this no longer fixes: the case the first version of this PR targeted, a corrupt record in the reply on a reused HTTP/1.1 keep-alive connection, happens after the request was written. It is ambiguous, so it still fails the request (502), as on main. What it does fix: a TLS failure while the request is still being written, for example on a shared HTTP/2 connection where a large body waits for flow-control window and the read for that window hits the corrupt record. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `helpers.py`: `RequestWriteTrace`, an httpcore `trace` callback that records whether `send_request_headers` or `send_request_body` failed. `forbids_retry(exc)` is true for a raw `ssl.SSLError` raised after a complete write. - `server.py` (`_retry_request`) and `handlers/streaming.py` (`_stream_response`): each attempt attaches a fresh trace, and a raw `ssl.SSLError` is retried only when that trace shows the write failed. Every other retry is unchanged. The streaming exhaustion handler still catches `ssl.SSLError`, for the 502 `upstream_tls_error` SSE body. - `tests/test_h2_stream_reset_retry.py`: the SSL cases now run through real httpcore over a fake socket. An accepted POST sends one POST and propagates, in the buffered and the streaming path. A failed write is retried, in both paths. The review's harness (an `httpx.MockTransport` that records the POST, then raises `ssl.SSLError`) sends one POST. - Three test fakes of `http_client.post` accept the new `extensions` kwarg. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_h2_stream_reset_retry.py -q 11 passed in 0.39s # source reverted to the previous head's retry ( |
||
|
|
998d71080c |
fix(proxy): keep Copilot requests on Copilot when an extension advises another provider (#4894)
## Description A proxy in front of GitHub Copilot (started by `headroom wrap vscode` / `wrap copilot`) holds only a Copilot token. When a proxy extension published cross-provider route advice (`request.state.headroom_route`), `BackendResolver.for_request` built a translating LiteLLM backend for it without looking at the upstream. A GPT-family Copilot call advised onto a Claude model was sent to Anthropic directly and failed: ``` LiteLLM backend initialized (provider=anthropic, region=None) route advice: claude-sonnet-5-5 via anthropic (...) /stats -> failed_by_provider: litellm-anthropic ``` The resolver now takes the request's upstream and ignores cross-provider advice when that upstream is Copilot. Companion change: headroomlabs-ai/routemegood#49 stops the routing extension publishing such advice in shadow mode and for holdouts. This PR is the safety property for apply mode. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/route_advice.py`: `for_request` takes `upstream_url`. When `is_copilot_upstream_url(upstream_url)` it returns the default backend and leaves `body["model"]` alone, with one info log line (model id and provider only). - `headroom/proxy/handlers/openai.py`, `handlers/anthropic.py`: all five call sites pass the request's upstream. The Anthropic sites are included because `wrap vscode` points the Anthropic target at Copilot as well. - `handlers/openai.py`: chat-stream CCR eligibility asks the resolver whether a backend will serve the request, instead of treating any advice as "routed to backend". Ignored advice no longer disables CCR tool injection. - `tests/test_route_advice_native_providers.py`: a resolver-level test (GHE Copilot host) and an end-to-end test through `create_app`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text CI on this PR: 34 checks passed, 12 skipped, 0 failed (lint, commitlint, test shards 1-4, test-macos). Locally: ruff 0.17.4 check + format --check clean on the 4 changed files; mypy clean on the 3 source files. tests/test_route_advice_native_providers.py: 8 passed. With the Copilot check disabled, test_copilot_chat_stays_on_copilot_despite_cross_provider_advice fails with: assert ['anthropic'] == [] ``` ## Real Behavior Proof - Environment: macOS 27.2.1, Python 3.12.13, core run from source (this branch vs `origin/main` |
||
|
|
ac932eb7c4 |
feat(telemetry): split beacon sessions by harness, model and day (#4881)
## Description
After backfilling the Beacon corpus, six questions about Oct 19 – Nov 14
could not be answered from what the client sends:
1. **Which harness, for a quarter of sessions.** 228,441 of 933,044
schema-v2 sessions (36.2M turns) carry no client. 30.2M of those turns
came through `/v1/compress`.
2. **Tokens and savings per harness.** `clients` has turn counts only;
34,553 sessions mix harnesses and cannot be divided.
3. **Harness version.** Only the Headroom version is reported.
4. **Tokens and savings per model.** `models` is a list, so savings
cannot be priced.
5. **Real installs versus churned ids.** Linux proxies show 19,916
installs for about the traffic of 4,258 on macOS.
6. **Per-day figures.** Always-on proxies report one session that grows
for weeks (34% of savings sat in multi-day sessions).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
All client changes are in `headroom/telemetry/session.py`, from data the
outcome funnel and the HTTP middleware already have. No handler changes.
`schema_version` stays 2; the fields are additive.
| Field | Content |
|---|---|
| `client_usage` | Per-harness rows of `n, original, saved, tool_saved,
output, cache_read, cache_write, cache_write_inferred, cache_write_5m,
cache_write_1h, uncached`. Covers every turn: unclaimed turns are
`via_mcp`, `via_compress_api` or `unidentified`. |
| `model_usage` | The same columns per public model id; anything else
pools into `other`. |
| `client_versions` | `(client, version, n)` from the User-Agent, e.g.
`claude_code 2.1.311`. Request counts, not turns. |
| `install` | `age_days` (id file mtime), `container`, `ci`. |
| rollover | A session at least 12h old is closed by the first request
on a later UTC day (`ended: "rollover"`). Sessions that merely run past
midnight are untouched. |
- `clients` and `models` are unchanged.
- `deploy/beacon/worker.js` learns the new keys and their nested shapes.
- `headroom/proxy/auth_policy.py` gains
`client_version_from_user_agent`; `headroom/proxy/server.py` calls it
for every POST.
**Pricing a usage row (review feedback).** Every column is the raw
per-turn counter and sums to its session total. Where a provider reports
no write count, Headroom infers one as input minus cached; those are the
same tokens as `uncached`. `cache_write_inferred` is that part, so
billed input is `uncached + cache_read + (cache_write -
cache_write_inferred)`, the rule `CostTracker.record` applies, and
`cache_write_5m` / `cache_write_1h` give the TTL split.
`test_a_usage_row_prices_inferred_cache_writes_the_way_the_cost_tracker_does`
covers the 1000-not-2000 case.
**Privacy.**
- Harness names in the new fields are checked against a closed list
(`_KNOWN_CLIENTS`), because `classify_client` returns the caller's
`X-Client` header as typed. Unknown names become `other`. The existing
`clients` field has the same gap and is not changed here.
- A version is sent only if it directly follows a known harness prefix
and is one to three numeric parts.
- The Worker holds the new string leaves to a slug or version pattern;
its test asserts injected text, hostnames and wrong types are dropped.
## Testing
- [x] Unit tests pass (`pytest`) — the suites listed below, not the
whole repository
- [x] Linting passes (`ruff check .`) — on the changed files
- [ ] Type checking passes (`mypy headroom`) — 8 errors in
`headroom/cache/prefix_tracker.py` and `headroom/ccr/mcp_server.py`,
identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_beacon_schema_v2.py tests/test_telemetry_session_selfcheck.py \
tests/test_telemetry.py tests/test_beacon_gzip.py tests/test_auth_policy.py tests/test_header_isolation.py -q
============================= 165 passed in 11.75s =============================
$ node deploy/beacon/test-rollup.mjs
ok
$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
```
Not run: the full Python suite. 3 tests in
`tests/test_proxy_telemetry_env.py` fail the same way on `main` without
this change.
## Real Behavior Proof
- Environment: macOS (darwin arm64), Python 3.12.13, this branch's proxy
(`service.version 0.45.0-dev`) with an isolated
`HEADROOM_WORKSPACE_DIR`, `HEADROOM_BEACON=on`,
`HEADROOM_TELEMETRY_ENDPOINT=http://127.0.0.1:8791/v1/logs`. Receiver:
this branch's `deploy/beacon/worker.js` `fetch()` served by a 30-line
Node HTTP listener with the R2 binding replaced by a directory.
Providers: Anthropic `claude-haiku-4-5-20251001` and OpenAI
`gpt-4o-mini`, live.
- Exact command / steps:
1. `node receiver.mjs deploy/beacon/worker.js ./r2 8791`
2. `headroom proxy --port 8799` (from this branch)
3. Four POSTs through the proxy: two `/v1/messages` with `User-Agent:
claude-cli/2.1.311 (external, cli)`, one `/v1/messages` with
`User-Agent: curl/8.4.0`, one `/v1/chat/completions` with `User-Agent:
codex-cli/0.45.0`.
4. `kill -INT` the proxy so it sends its closing report.
- Observed result: the three Anthropic calls returned 200 (the second
and third were served by Headroom's response cache); the OpenAI call
returned 429 from the provider (account quota). The receiver logged
`POST /v1/logs -> 204` and stored one object,
`sessions/dt=2026-11-15/hh=15/<uuid>.json`. Its new fields, as stored
after the allowlist:
```json
"session": {"seq": 0, "duration_s": 5, "turns": 4, "ended": "shutdown", "final": true}
"tokens": {"original": 28, "input": 28, "output": 4, "saved": 0, "cache_read": 0, "cache_write": 14, "uncached": 28}
"clients": [{"client": "claude", "n": 1}, {"client": "claude_code", "n": 2}, {"client": "codex", "n": 1}]
"client_usage": [
{"client": "claude", "n": 1, "original": 0, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 0},
{"client": "claude_code", "n": 2, "original": 14, "saved": 0, "tool_saved": 0, "output": 4, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14},
{"client": "codex", "n": 1, "original": 14, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 14, "cache_write_inferred": 14, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14}]
"model_usage": [
{"model": "claude-haiku-4-5-20251001", "n": 3, "original": 14, "output": 4, "cache_write": 0, "cache_write_inferred": 0, "uncached": 14, "...": "other columns 0"},
{"model": "gpt-4o-mini", "n": 1, "original": 14, "output": 0, "cache_write": 14, "cache_write_inferred": 14, "uncached": 14, "...": "other columns 0"}]
"client_versions": [{"client": "claude_code", "version": "2.1.311", "n": 2}, {"client": "codex", "version": "0.45.0", "n": 1}]
"install": {"age_days": 0, "container": false, "ci": false}
```
Both tables' `n` sum to `session.turns` (4) and `original` to
`tokens.original` (28). The OpenAI row is the reviewed case:
`cache_write` 14 is entirely inferred, and `uncached + cache_read +
(cache_write - cache_write_inferred)` is 14, not 28. Nothing from the
User-Agent beyond harness and version reached the stored object.
- Not tested:
- The Worker on Cloudflare itself (Workers runtime, real R2).
`worker.js` ran under Node.
- A successful OpenAI turn, a Responses API turn with a reported write,
and Anthropic turns with cache reads or 5m/1h writes (covered by unit
tests only).
- `via_compress_api`, `via_mcp` and `unidentified` labels, a session
mixing harnesses on one model, and the 12-hour rollover (unit tests
only).
- Codex over WebSocket, which never passes the HTTP middleware and so
reports no version.
- gzip transport (`HEADROOM_BEACON_GZIP=1`) and a heartbeat (`ended:
"active"`); only the closing report was observed.
- Linux, Windows, containers and CI.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; the beacon is not behind a rollout
channel.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, for installs with the beacon on
(the default). The session report carries four more keys (about 5 KB
more per report for a typical session with gzip off), and a session at
least 12 hours old now ends at the first request on a later UTC day, so
always-on installs report about one session per day instead of one that
grows for weeks. No request-path behaviour changes.
- Kill switch / disable path: `HEADROOM_BEACON=off`,
`HEADROOM_TELEMETRY=off`, `DO_NOT_TRACK=1` or offline mode disable all
of it, as before. Server side, removing a key from `ALLOWED_KEYS` in
`deploy/beacon/worker.js` drops that field for every install without a
client release.
- Unsafe override required: no
- Qualification impact: none known.
- Rollback path: revert this PR. Rows already stored keep the extra
keys; readers use `union_by_name`.
- Deploy order: the Worker must be deployed before a client release that
sends these fields, or they are dropped on arrival. Production is still
the 2026-10-08 Worker build.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — none
made; `docs/content/docs/security-model.mdx` and `api-reference.mdx`
describe beacon fields and have not been updated
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — for the
suites listed under Testing
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- `client_versions` misses WebSocket sessions.
- `/v1/compress` traffic is now visible but still not attributed to the
integration that called it.
- Tool-call kinds, tool-definition size, compaction and subagent turns
are not included: they need the request body in each handler and a
privacy design for tool names.
- An earlier review suggested splitting rollover into its own PR since
it changes what a session row means; it is kept here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
00436a8b11 |
feat(savings): book modelled output savings per request, as a token and dollar range (#4878)
## Description
Benchmark factors registered with `register_modelled_factors` reached
`/stats` only as a percentage. They never reached the per-request
rollup, so modelled output savings had no tokens or dollars in the
lifetime totals that the local dashboard and the control-plane heartbeat
read. This PR books each shaped request's output saving as a low–high
range of tokens and dollars, from the same tier the dashboard reports.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue) — an unshaped
treatment request was booked a baseline saving
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `SavingsRecorder.request_savings_range(labels, output_tokens) -> (low,
high)` replaces `estimate_request_savings` in the outcome funnel.
- It counts only requests that were actually shaped, marked by the
`output_shaper:verbosity:L<n>` label. The treatment stratum label is
attached before shaping is attempted, so it proves nothing.
- A holdout (control) request books nothing.
- A response with no reported output usage (output 0) books nothing.
- The tier comes from `SavingsLedger.select_tier(level)`, the function
`best_estimate` now delegates to. Lifetime dollars therefore always come
from the tier that the dashboard's percentage comes from:
- **measured:** the request's own stratum's control mean minus its
output, with low = high. It applies only if that stratum qualifies at
the cluster gate the measurement used. A stratum outside the measurement
books 0, because the measured headline excludes it too.
- **estimated:** the request's stratum baseline, with low = high.
- **modelled:** the factor band, `observed × r/(1−r)` at each end.
- `SavingsTracker`:
- Books the low end everywhere it booked before (history, rollups,
OTel).
- Adds `lifetime.output_tokens_saved_high` and
`output_savings_usd_high`, priced per model at output rates.
- Both state builders write the new keys, so a fresh tracker has them.
- Dashboard: the Output Tokens Saved tile adds "≈ $low–$high saved on
output · low–high tokens, all time", read through a component getter.
OSS is unchanged. It ships an empty factor table, so the modelled tier
never appears.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): clean on the touched files
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ git log --oneline -1
|
||
|
|
1f4c66f50a |
fix(beacon): roll up each hour in sixteen parallel queue slices (#4880)
## Description
The hourly rollup (`deploy/beacon/worker.js`) read every heartbeat
object of an hour inside one cron invocation.
- A Worker may have 6 requests in flight, so reads ran at about 12
objects/s regardless of `FANOUT = 100`.
- A cron invocation is stopped at 15 minutes, so one run could read
about 11k objects.
- Hours passed that size on 2026-08-19. A stopped run wrote nothing.
Result: 1,388 of 2,490 clock hours (23.8M of 33.3M heartbeat objects)
never got a rollup. Raw `sessions/` is complete; only `rollup/` had
gaps. The stats dashboard reads only rollups. The missing hours have
since been backfilled outside the Worker, so this PR is about not
falling behind again.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `scheduled()` no longer reads heartbeats. It lists hours with no
rollup and enqueues 16 messages per hour to a Cloudflare Queue, one per
first hex character of the object name (names are `crypto.randomUUID()`,
so slices are even and this works on objects already stored).
- `queue()` handles one slice per invocation: `rollupSlice()` reads it
(6 in flight) and writes `rollup-parts/<hour>/<slice>`; `finalizeHour()`
merges the 16 parts into `rollup/<hour>/data.ndjson` once all are
present, then deletes the parts.
- A failed slice is retried alone (60 s delay, 5 tries, then dead-letter
queue). An hour still unfinished after 3 hours is enqueued again.
`scheduled()` sweeps anything a finished hour left under
`rollup-parts/`.
- Rollup output is unchanged: same key, same winner per (install,
session).
- `wrangler.toml` gains the queue producer and consumer bindings.
## Testing
- [ ] Unit tests pass (`pytest`) — not applicable, no Python changed
- [ ] Linting passes (`ruff check .`) — not applicable
- [ ] Type checking passes (`mypy headroom`) — not applicable
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cd deploy/beacon && node test-rollup.mjs
rollup dt=2026-08-06/hh=14: skipped 1 unparseable record(s)
gzip: 4079 B -> 104 B, bomb refused
rate limit: 429 over, fail-open on outage and when unbound
nested allowlist: client fields kept, unknown nested keys and free text dropped
ok
```
New cases: cross-slice dedupe, a failed slice leaving the hour
unpublished, enqueue / skip / re-enqueue in `scheduled()`, retry with
delay, reuse of an existing part, and the sweep.
## Real Behavior Proof
- Environment: macOS 27.2.1 (arm64), Node v26.7.0, DuckDB v1.5.5 with
`httpfs`, this branch at `811e6cf1`. Input is production data read from
the R2 bucket `headroom-telemetry` with a read-only key. The Worker
module ran under Node, not on Cloudflare; the R2 binding was an
in-memory map that pages `list()` at 1000 keys like R2.
- Exact command / steps (both scripts are in full below):
1. `./fetch_hour.sh 2026-08-10 03 h0810` and `./fetch_hour.sh 2026-10-14
12 h1014` — download every raw object of the hour, and the rollup object
stored in R2 for it.
2. `node compare.mjs deploy/beacon/worker.js h0810 dt=2026-08-10/hh=03`
(and likewise for `h1014`) — deliver 16 slice messages concurrently to
this branch's `queue()` handler, then compare what it published with the
stored rollup, row by row.
3. `node deploy/beacon/test-rollup.mjs h0810/raw` — the repo's own
real-corpus check.
- Observed result:
```text
$ ./fetch_hour.sh 2026-08-10 03 h0810
3666 raw objects written to h0810/raw
$ ./fetch_hour.sh 2026-10-14 12 h1014
24850 raw objects written to h1014/raw
$ node compare.mjs deploy/beacon/worker.js h0810 dt=2026-08-10/hh=03
{ "part": "dt=2026-08-10/hh=03", "raw_objects": 3666, "raw_gets": 3666, "acked": 16, "retried": 0,
"sessions_published": 945, "sessions_in_reference": 945, "same_session_and_seq": 945,
"identical_rows": 945, "parts_left_behind": 0, "wall_ms": 39, "peak_rss_mb": 103 }
$ node compare.mjs deploy/beacon/worker.js h1014 dt=2026-10-14/hh=12
{ "part": "dt=2026-10-14/hh=12", "raw_objects": 24850, "raw_gets": 24850, "acked": 16, "retried": 0,
"sessions_published": 6903, "sessions_in_reference": 6903, "same_session_and_seq": 6903,
"identical_rows": 6903, "parts_left_behind": 0, "wall_ms": 706, "peak_rss_mb": 432 }
$ node deploy/beacon/test-rollup.mjs h0810/raw
real corpus: 3666 objects -> 945 sessions in 1 object
ok
```
- **`dt=2026-08-10/hh=03`** (3,666 objects): the reference is the rollup
the current production Worker wrote. All 945 rows are identical after
key-order normalisation.
- **`dt=2026-10-14/hh=12`** (24,850 objects, among the busiest hours):
the old Worker never completed this hour. The reference is the rollup
built by the offline backfill, a separate implementation, and uploaded
to R2 on 2026-11-15. All 6,903 rows are identical. The published object
is 15.7 MB.
- In both runs every raw object was read exactly once, all 16 messages
were acknowledged, none retried, and no parts were left behind.
- Not tested:
- **Anything on Cloudflare.** Not deployed, not run under `wrangler
dev`. Queue delivery, consumer concurrency and ramp-up, the 6-connection
limit, the 15-minute limit, retries, the dead-letter queue and real R2
latency are unobserved.
- **Memory on the Workers runtime.** `peak_rss_mb` above is the Node
process including all raw objects held by the stub, so it says nothing
about the 128 MB isolate. `finalizeHour` holds the merged hour in memory
and builds the output as one string; for the 15.7 MB hour that is a few
multiples of 15.7 MB, by estimate only.
- **`scheduled()` against a real bucket**, including the listing of
`rollup/` and `rollup-parts/` at production size and enqueueing a
backlog.
- Throughput per slice. The measured ~12 objects/s per invocation comes
from the old Worker's write timestamps, not from this code.
<details><summary><code>fetch_hour.sh</code></summary>
```bash
#!/usr/bin/env bash
# Download one hour of raw beacon objects from R2 into <out>/<object name>, plus
# the rollup object stored for that hour. ./fetch_hour.sh 2026-08-10 03 <out>
# Needs R2_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY (read-only is enough).
set -euo pipefail
DAY="$1"; HH="$2"; OUT="$3"; mkdir -p "$OUT/raw"
duckdb >/dev/null <<SQL
LOAD httpfs; SET threads=48; SET http_retries=8; SET http_timeout=120;
CREATE SECRET (TYPE r2, KEY_ID '${R2_ACCESS_KEY_ID}', SECRET '${R2_SECRET_ACCESS_KEY}', ACCOUNT_ID '${R2_ACCOUNT_ID}');
COPY (SELECT filename, json::VARCHAR line FROM read_ndjson_objects('r2://headroom-telemetry/sessions/dt=${DAY}/hh=${HH}/*.json', filename=true))
TO '${OUT}/raw.jsonl' (FORMAT json);
COPY (SELECT json::VARCHAR line FROM read_ndjson_objects('r2://headroom-telemetry/rollup/dt=${DAY}/hh=${HH}/data.ndjson', maximum_object_size=40000000))
TO '${OUT}/stored_rollup.ndjson' (FORMAT csv, HEADER false, QUOTE '', ESCAPE '', DELIMITER '\x1f');
SQL
python3 - "$OUT" <<'PY'
import json, os, sys, collections
out = sys.argv[1]; files = collections.defaultdict(list)
for l in open(f"{out}/raw.jsonl"):
r = json.loads(l); files[os.path.basename(r["filename"])].append(r["line"])
for name, lines in files.items():
open(f"{out}/raw/{name}", "w").write("\n".join(lines))
print(f"{len(files)} raw objects written to {out}/raw")
PY
```
</details>
<details><summary><code>compare.mjs</code></summary>
```js
// Run this branch's queue consumer over one real hour of raw objects and compare
// what it publishes with a reference rollup, row by row.
// node compare.mjs <worker.js> <dir with raw/ and stored_rollup.ndjson> <dt=…/hh=…>
import { readFileSync, readdirSync } from 'node:fs';
import { resolve } from 'node:path';
import { pathToFileURL } from 'node:url';
const [workerPath, dir, part] = process.argv.slice(2);
const worker = (await import(pathToFileURL(resolve(workerPath)).href)).default;
// The slice of the R2 binding the Worker uses, over a Map. list() pages at 1000
// like R2 does.
const files = new Map();
for (const name of readdirSync(`${dir}/raw`)) {
files.set(`sessions/${part}/${name}`, readFileSync(`${dir}/raw/${name}`, 'utf8'));
}
const raw = files.size;
let gets = 0;
const CORPUS = {
list: async ({ prefix, cursor, limit = 1000 }) => {
const keys = [...files.keys()].filter((k) => k.startsWith(prefix)).sort();
const start = cursor ? keys.indexOf(cursor) : 0;
const page = keys.slice(start, start + limit);
const next = keys[start + limit];
return { objects: page.map((key) => ({ key, uploaded: new Date(0) })), truncated: !!next, cursor: next };
},
get: async (key) => { gets++; return files.has(key) ? { text: async () => files.get(key) } : null; },
head: async (key) => (files.has(key) ? { key } : null),
put: async (key, body) => { files.set(key, body); },
delete: async (keys) => { for (const key of keys) files.delete(key); },
};
const started = Date.now();
const outcome = { ack: 0, retry: 0 };
// Sixteen messages, one per slice, delivered concurrently as the queue would.
await Promise.all([...'0123456789abcdef'].map((slice) =>
worker.queue({ messages: [{ body: { part, slice }, ack: () => outcome.ack++, retry: () => outcome.retry++ }] }, { CORPUS })));
const ms = Date.now() - started;
const canon = (v) => Array.isArray(v) ? v.map(canon)
: v && typeof v === 'object' ? Object.fromEntries(Object.keys(v).sort().map((k) => [k, canon(v[k])])) : v;
const index = (text) => new Map(text.split('\n').filter(Boolean).map((line) => {
const rec = JSON.parse(line);
return [`${rec.resource?.['headroom.install_id']} ${rec.session?.id}`, { seq: rec.session?.seq, row: JSON.stringify(canon(rec)) }];
}));
const mine = index(files.get(`rollup/${part}/data.ndjson`) ?? '');
const reference = index(readFileSync(`${dir}/stored_rollup.ndjson`, 'utf8'));
let sameSeq = 0, sameRow = 0;
for (const [key, value] of mine) {
const other = reference.get(key);
if (other?.seq === value.seq) sameSeq++;
if (other?.row === value.row) sameRow++;
}
console.log(JSON.stringify({
part, raw_objects: raw, raw_gets: gets - 16, acked: outcome.ack, retried: outcome.retry,
sessions_published: mine.size, sessions_in_reference: reference.size,
same_session_and_seq: sameSeq, identical_rows: sameRow,
parts_left_behind: [...files.keys()].filter((k) => k.startsWith('rollup-parts/')).length,
wall_ms: ms, peak_rss_mb: Math.round(process.resourceUsage().maxRSS / 1024),
}, null, 1));
```
</details>
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is the telemetry receiver's
cron, not the proxy; no client or request-path behaviour changes.
- Minimum rollout channel: n/a
- Stable/default behavior changed: only on the Worker. Rollups are built
by queue consumers instead of inside the cron run; the rollup objects
readers consume are unchanged. A new `rollup-parts/` prefix appears in
the bucket while an hour is in flight.
- Kill switch / disable path: remove the cron trigger or the queue
consumer in `wrangler.toml` and redeploy. Raw `sessions/` is never
modified, so any rollup can be rebuilt.
- Unsafe override required: no
- Qualification impact: none known.
- Rollback path: redeploy the previous Worker build. Hours it cannot
finish can be filled by the offline backfill, as was done for the
existing gap.
- Deploy steps (not done by this PR):
```
npx wrangler queues create beacon-rollup
npx wrangler queues create beacon-rollup-dlq
npx wrangler deploy
```
Queues need the Workers Paid plan, which this Worker already requires.
Deploying also ships the two merged-but-undeployed Worker changes (#4008
rate limit, #4691 nested allowlist); production is still the 2026-10-08
build.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — none
needed that I know of
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — `node
deploy/beacon/test-rollup.mjs`
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- Nothing alerts on the dead-letter queue or on hours that stay
unrolled.
- The first run on Cloudflare should be watched: check that `rollup/`
gains the previous hour within a few minutes of :05 and that
`rollup-parts/` empties.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
d9ae7ad721 |
fix(proxy): stop a "*" origin setting from waiving the WebSocket Origin check (#4846)
## Description The WebSocket routes (`/v1/responses`, `/v1/live`) check the browser `Origin` against `HEADROOM_WS_ORIGINS`, falling back to `HEADROOM_CORS_ORIGINS`. A `*` entry in either made every Origin acceptable. `HEADROOM_CORS_ORIGINS=*` is a documented value for CORS (which pages may *read* a response), so loosening CORS for a remote dashboard also let any page open in the user's browser open these sockets, and the proxy then adds its own credentials upstream. `*` now matches nothing in the Origin check. A list that holds only `*` keeps the loopback default; explicit origins in either variable keep working. CORS itself is unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: `_allowed_ws_origins_from_env` drops `*` entries (empty result means the loopback default); `_is_allowed_websocket_origin` no longer short-circuits to `True` on `*`. `providers/codex/live.py` uses the same helper, so `/v1/live` is covered too. - `tests/test_openai_codex_routing.py`: policy test for `*` via either variable and mixed with an explicit origin, plus a handler test that a foreign Origin is closed with 1008 under `HEADROOM_CORS_ORIGINS=*`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # new tests on origin/main (before the fix) $ pytest tests/test_openai_codex_routing.py -k wildcard FAILED tests/test_openai_codex_routing.py::test_websocket_origin_policy_ignores_wildcard[None-*] FAILED tests/test_openai_codex_routing.py::test_websocket_origin_policy_ignores_wildcard[*-None] FAILED tests/test_openai_codex_routing.py::test_websocket_origin_policy_ignores_wildcard[None-*, https://dash.example.com] FAILED tests/test_openai_codex_routing.py::test_handle_openai_responses_ws_closes_foreign_origin_under_wildcard_cors ======================= 4 failed, 43 deselected in 1.33s ======================= # with the fix $ pytest tests/test_openai_codex_routing.py tests/test_codex_live.py ======================= 57 passed, 7 warnings in 16.31s ======================== $ ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_routing.py All checks passed! $ mypy headroom/proxy/handlers/openai.py --ignore-missing-imports Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, `headroom proxy --host 127.0.0.1 --no-optimize` started with `HEADROOM_CORS_ORIGINS='*'`, one run on origin/main and one on this branch. - Exact command / steps: open `ws://127.0.0.1:<port>/v1/responses` with the `websockets` client, once with `Origin: https://page.example` and once with `Origin: http://127.0.0.1:<port>`. - Observed result: ```text == origin/main (HEADROOM_CORS_ORIGINS=*) Origin=https://page.example -> handshake accepted, socket open Origin=http://127.0.0.1:18941 -> handshake accepted, socket open == this branch (HEADROOM_CORS_ORIGINS=*) Origin=https://page.example -> handshake refused HTTP 403 Origin=http://127.0.0.1:18942 -> handshake accepted, socket open ``` - Not tested: `/v1/live` end to end (same helper, covered by the unit tests' shared code path). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: only for operators who set `*` in `HEADROOM_WS_ORIGINS` or `HEADROOM_CORS_ORIGINS`. Native clients (Codex, SDKs) send no Origin and are unaffected; loopback pages still connect. - Kill switch / disable path: list the browser origins that need the WebSocket routes explicitly in `HEADROOM_WS_ORIGINS`. - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes `HEADROOM_WS_ORIGINS` is not documented anywhere, so there was no doc to update. A companion PR applies the same Origin rule to the HTTP routes; once both land, `*` is ignored there too because they share `_is_allowed_websocket_origin`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4846" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Reported by @JoaoMarcos44. Thank you for your contribution! 🩵 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
bae62bc87b |
fix(compression): bound the JSON-block and truncation-banner scans (#4845)
## Description
Two scans over tool output ran in quadratic time:
1. **Mixed-content JSON detection** (`_has_valid_json_block_with_text`,
default-on through `ContentRouter` → `is_mixed_content`) and the section
splitter try every line that starts with `{` or `[` as a JSON block. A
candidate that never balances walks to the end of the content, so N such
lines cost N²/2 line steps (20,000 lines of `{` took about a minute).
The existing per-line memo removes the character work but not the line
walk.
2. **ast-grep truncation-banner detection** (`_detect_truncation`): the
banner regex has unbounded runs on both sides of the `showing lines A-B
of T total` phrase, so an unclosed banner was rescanned once per phrase.
Fixes:
1. `_JsonBlockScanner` answers every candidate of one pass with exactly
the result of walking it. Candidates are walked directly until they have
walked as many lines as the content has. After that, all candidates are
answered together:
- Every walk is a chain of (line, string-state) nodes with one successor
each, so all walks form one forest, and each node is scanned once.
- With `D(v)` the bracket and brace sums from node `v` to the end of its
chain, a walk from `u` balances at `w` exactly when `D(next(w)) >= D(u)`
in both sums. So each candidate ends just before its nearest ancestor
that satisfies this.
- One depth-first pass with an undoable tree of (brace sum, depth)
stacks, keyed by bracket sum, finds that ancestor in O(log² n). There is
no budget.
2. The banner is found in two linear steps: locate `[truncated: partial
view`, take the span to the next bracket (it must be `]`), then match
the phrase inside that span. Same matches as the old regex (checked
against it on 3,000 random layouts).
Fix adapted from a proposal by HetCreep.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/mixed_content.py`: `_JsonBlockScanner` (direct
walks, then a forest solve over all candidates), used by
`_has_valid_json_block_with_text` and `split_into_sections`.
`_extract_json_block` is unchanged from main.
- `headroom/proxy/interceptors/astgrep.py`: `_TRUNCATION_SIGNATURES` now
holds only the phrase pattern; `_iter_banners` finds the `[truncated:
partial view … ]` span and searches the phrase inside it.
- `tests/test_tool_output_scan_bounds.py`: timing tests for both shapes,
an oracle test against the previous banner regex, and checks that a real
banner and a 5,000-line JSON block inside text are still found. A JSON
object that follows 150 or 20,000 unclosed `{` lines, or a prefix whose
bracket and brace counts each reach zero but never on the same line
(once, and repeated 130 times), still gets its own JSON section, as does
one after 100 or 10,000 pairs of lines ending inside an escaped string.
The solve is checked against `_extract_json_block` on 3,000 random
inputs.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
New tests on `origin/main` code:
```text
FAILED tests/test_tool_output_scan_bounds.py::test_truncation_scan_is_linear_on_unclosed_banner
FAILED tests/test_tool_output_scan_bounds.py::test_json_block_scans_are_bounded_on_unclosed_braces
assert time.perf_counter() - start < 10.0
E assert (63611.161382791 - 63549.279589625) < 10.0
==================== 2 failed, 3 passed in 75.96s (0:01:15) ====================
```
With the fix:
```text
$ pytest -q tests/test_tool_output_scan_bounds.py --durations=3 # (first version; now 10 tests, 10 passed in 0.88s)
0.34s call tests/test_tool_output_scan_bounds.py::test_json_block_scans_are_bounded_on_unclosed_braces
0.02s call tests/test_tool_output_scan_bounds.py::test_truncation_banner_matches_previous_regex
============================== 5 passed in 1.09s ===============================
$ pytest -q tests/test_tool_output_scan_bounds.py tests/test_mixed_content_scan_cache.py tests/test_mixed_content_sections.py tests/test_content_router_mixed_fence_placeholder.py
============================= 300 passed in 5.90s ==============================
$ pytest -q tests/test_tool_result_interceptors.py tests/test_proxy_compression_executor.py tests/test_subprocess_encoding.py tests/test_runtime_env.py tests/test_content_router*.py
============================= 247 passed in 32.58s =============================
$ ruff check / ruff format --check (0.17.3) on changed files: All checks passed
$ mypy headroom/proxy/interceptors/astgrep.py headroom/transforms/mixed_content.py --ignore-missing-imports
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: macOS (Apple silicon), Python 3.12 venv, this branch vs
`origin/main`.
- Exact command / steps: called `_has_valid_json_block_with_text` +
`split_into_sections` on 20,000 lines of `{`, and `_detect_truncation`
on `"[truncated: partial view " + "showing lines 1-10 of 50 total " *
12000`.
- Observed result: JSON scans about 62 s on main, 0.34 s here; banner
scan about 13 s on main, milliseconds here. The existing scan-cache
oracle tests (which compare against the original extractor) still pass.
- Not tested: end-to-end through the proxy; both functions are pure and
covered directly.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no change in results (exact); only
time and memory bounds change
- Kill switch / disable path: none needed
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Timings with the fix: 20,000 unclosed `{` lines take 0.13 s, the
crossed-count prefix repeated 130 times takes 0.23 s, 20,000 alternating
`{` / `}[` lines take 0.29 s, and 10,000 escaped-string line pairs take
0.14 s. With direct walking, the first takes about 62 s. Separately, on
main a balanced block nested about 10,000 deep makes `json.loads` raise
`RecursionError` in the gate; that is unchanged here.
Note: `tests/test_transforms_content_router.py` has 3 failures
(`test_json_in_html_preserves_records_and_markup`,
`test_embedded_json_mixed_tool_output_preserves_every_record[minify-*]`)
that also fail on unmodified `origin/main` in my environment, so they
are not caused by this change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reported by @JoaoMarcos44. Thank you for your contribution! 🩵
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
f5c4217648 |
fix(proxy): enforce HEADROOM_PROXY_TOKEN in the Rust front and listen on loopback by default (#4840)
## Description
The Rust front (`crates/headroom-proxy`) relays every request to the
Python proxy from `127.0.0.1`. The Python proxy treats a loopback caller
as local: it skips the `HEADROOM_PROXY_TOKEN` check and serves `/admin`,
`/debug` and `/settings` to it. The front had no inbound check of its
own, listened on `0.0.0.0:8787` by default, and dropped
`x-headroom-proxy-token` (as an `x-headroom-*` header), so a token set
on the Python side neither protected the stack nor could be presented
through the front.
This PR makes the Rust front apply the same token model as the Python
proxy (`read_proxy_token` / `scrub_proxy_token_headers` in
`headroom/proxy/server.py`, `headroom/proxy/bind_policy.py`) before
anything is forwarded, and makes its default listen address loopback.
Fix adapted from a proposal by HetCreep.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `crates/headroom-proxy/src/inbound_auth.rs`, installed as a
router-wide layer (all routes, the catch-all and WebSocket upgrades)
when `HEADROOM_PROXY_TOKEN` / `--proxy-token` is set:
- a non-loopback peer (or one with no peer address) must send the token
as `X-Headroom-Proxy-Token` or `Authorization: Bearer`; the explicit
header wins, as in Python. Constant-time compare. Otherwise `401
{"error":"unauthorized"}` and no upstream contact; a
`proxy_auth_rejected` warning is logged (no credential values).
- loopback peers are exempt (same as Python); `GET /healthz`,
`/healthz/upstream`, `/health`, `/livez`, `/readyz` stay open;
`GET`/`HEAD /api/hello` (also under `/p/<project>`) is answered locally
with an empty 200, as Python does, instead of being relayed.
- the token is removed before forwarding: `x-headroom-proxy-token`
always, `Authorization` only when it carries the token (a provider
credential is left alone).
- the token is redacted from `Debug` output of `Config`.
- `main.rs`: startup bind policy, same rule and variable as Python: a
non-loopback `--listen` with no token exits with code 2 and a message
naming both remedies, unless `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`
(then a warning is logged).
- `config.rs`: `--listen` / `HEADROOM_PROXY_LISTEN` default changes from
`0.0.0.0:8787` to `127.0.0.1:8787`; new `--proxy-token` (env
`HEADROOM_PROXY_TOKEN`, value hidden from `--help`).
- `RUST_DEV.md`: runbook exports a token for the `0.0.0.0` cutover, flag
table updated, new "Inbound authentication" section.
**Behaviour change for operators:** a Rust front started with `--listen
0.0.0.0:...` (or relying on the old default to be reachable from other
hosts) now needs `HEADROOM_PROXY_TOKEN` set, and clients on other hosts
must send it. Setups that front the port with something else that
restricts access can set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`.
Loopback clients (Claude Code, Codex, SDKs on the same host) are
unaffected. The Docker image's entrypoint is the Python proxy, so it is
unaffected.
## Testing
- [x] `cargo test -p headroom-proxy` (46 test binaries, 608 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes
New tests:
- `tests/integration_inbound_auth.rs` (11 tests, `oneshot` with an
injected `ConnectInfo` so a non-loopback peer can be simulated,
`wiremock` upstream): missing/wrong token is 401 and the upstream sees
nothing; correct token via header or Bearer is forwarded without the
token; provider `Authorization` kept when the header carries the token;
loopback and IPv4-mapped loopback peers pass; missing peer address fails
closed; health probes open but `POST /readyz` is 401; `/api/hello`
answered locally; no token configured keeps passthrough.
- `tests/integration_bind_policy.rs` (3 tests, real binary): `0.0.0.0`
with no token refuses to start; loopback starts; default `--listen` is
loopback.
- unit tests in `inbound_auth.rs` for token parsing, constant-time
compare, bind policy and exemptions.
### Test Output
Before (the bind-policy test file copied onto a clean `origin/main`
worktree; `integration_inbound_auth.rs` does not compile there since
main has no token setting at all):
```text
test default_listen_address_is_loopback ... FAILED
test loopback_listen_without_token_still_starts ... ok
test non_loopback_listen_without_token_refuses_to_start ... FAILED
test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20.01s
```
After:
```text
$ cargo test -p headroom-proxy --test integration_inbound_auth --test integration_bind_policy
test default_listen_address_is_loopback ... ok
test non_loopback_listen_without_token_refuses_to_start ... ok
test loopback_listen_without_token_still_starts ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.01s
test missing_peer_address_fails_closed ... ok
test remote_peer_with_bearer_token_is_forwarded_without_it ... ok
test remote_peer_without_token_gets_401_and_upstream_is_not_contacted ... ok
test token_header_wins_over_a_provider_bearer ... ok
test no_token_configured_keeps_todays_passthrough ... ok
test loopback_peer_without_token_is_still_forwarded ... ok
test ipv4_mapped_loopback_peer_counts_as_loopback ... ok
test remote_peer_with_token_header_is_forwarded_without_it ... ok
test hello_probe_is_answered_locally_without_token ... ok
test remote_peer_with_wrong_token_gets_401 ... ok
test health_probes_stay_reachable_without_token ... ok
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.74s
```
## Real Behavior Proof
- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (
|
||
|
|
b0a74a5540 | Merge branch 'main' into feat/https-compose-caddy | ||
|
|
cfdeb1c466 |
fix(learn): render non-string Bash commands without raising (#4826)
## Description
Fixes a crash in `headroom learn` when a scanned session stores a
non-string Bash `command`.
`ToolCall.input_summary` sliced `input_data["command"]` directly,
assuming a string. A real opencode v2 database contained eight Bash
calls with `{"command": true}`, which raised:
```
TypeError: object of type 'bool' has no len()
```
Both the digest formatter (`analyzer._format_tool_call`) and the loop
sampler (`loops.detect_loops`) read `input_summary`, so a single
malformed call aborted the entire `headroom learn --all --apply` run.
No linked issue: this was found while validating the opencode v2 scanner
change in #4199 (now merged). The crash is in shared `ToolCall` code, so
the fix is submitted separately.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `ToolCall.input_summary` now coerces a non-string `command` with
`str()` instead of slicing the raw value; `None` renders as an empty
string.
- `input_summary` treats a non-mapping `input_data` as `{}` before
calling `.get()`, matching the defensive shape already used in
`loops._identity_input`.
- Added `TestInputSummaryToleratesMalformedInput` with six cases:
bool/int/`None`/list commands, a non-mapping input, and an end-to-end
`detect_loops` + `_build_digest` case.
- No behavior change for valid string commands; the other summary
branches already use `str()`/f-strings.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] Type checking passes (`mypy headroom`) — not run; the touched
module was not type-checked as part of this change.
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_learn/test_loop_weighting.py -k TestInputSummaryToleratesMalformedInput -q
6 passed, 89 deselected, 1 warning in 0.78s
$ .venv/bin/python -m pytest tests/test_learn/ tests/test_cli_learn.py -q
455 passed, 7 skipped, 2 warnings in 32.88s
$ uvx ruff@0.16.8 check headroom/learn/models.py tests/test_learn/test_loop_weighting.py
All checks passed!
$ uvx ruff@0.16.8 format --check headroom/learn/models.py tests/test_learn/test_loop_weighting.py
2 files already formatted
```
All six new tests fail against the unpatched `models.py`.
## Real Behavior Proof
- Environment: Linux x86_64; Python 3.14.8; `uv sync --frozen`; PR head
`22e6a6f8fb05f79dcc82ea57a24bfbd6615305b8`. No provider credentials,
network, or private data required.
- Exact command / steps: save the script below as
`proof_malformed_opencode.py` at the repo root, then run
`.venv/bin/python proof_malformed_opencode.py`. It creates a temporary
opencode v2 SQLite database (`project` / `session_v2` /
`session_message`) containing three Bash calls whose input is
`{"command": true}`, then runs `OpenCodePlugin.scan_project` →
`detect_loops` → `_build_digest`.
- Observed result: with this PR, `sessions: 1 calls: 3`, `summaries:
['True', 'True', 'True']`, `loops: [('Bash', 3, 'True')]`, `digest
chars: 684`, `digest contains malformed call: True`. With
`headroom/learn/models.py` reverted to `origin/main`, the same script
exits with `TypeError: object of type 'bool' has no len()` at
`models.py:67` (`return cmd[:100] + "..." if len(cmd) > 100 else cmd`).
The original private database scan (28 sessions / 7,882 calls, 8
malformed commands, 29 loops, 319,885-char digest) is additional
evidence but is not reproducible outside that machine.
- Not tested: a real LLM analysis call (requires provider credentials),
Windows path handling, other agents' databases, and `mypy`. The scanner
and analyzer logic are unchanged; only the shared display property is
guarded.
<details>
<summary>proof_malformed_opencode.py</summary>
```python
"""Reproduce the malformed-command crash with a synthetic opencode v2 DB."""
import json
import sqlite3
import tempfile
from pathlib import Path
from headroom.learn.analyzer import _build_digest
from headroom.learn.loops import detect_loops
from headroom.learn.plugins.opencode import OpenCodePlugin
with tempfile.TemporaryDirectory() as tmp:
tmp_path = Path(tmp)
project_path = tmp_path / "project"
project_path.mkdir()
db_path = tmp_path / "opencode.db"
conn = sqlite3.connect(db_path)
conn.executescript(
"""
CREATE TABLE project (id TEXT PRIMARY KEY, worktree TEXT NOT NULL, name TEXT);
CREATE TABLE session_v2 (
id TEXT PRIMARY KEY, project_id TEXT NOT NULL, time_created INTEGER NOT NULL
);
CREATE TABLE session_message (
id TEXT PRIMARY KEY, session_id TEXT NOT NULL, type TEXT NOT NULL,
seq INTEGER NOT NULL, time_created INTEGER NOT NULL,
time_updated INTEGER NOT NULL, data TEXT NOT NULL
);
"""
)
conn.execute("INSERT INTO project VALUES (?, ?, ?)", ("p", str(project_path), "proof"))
conn.execute("INSERT INTO session_v2 VALUES (?, ?, ?)", ("s", "p", 1))
content = [
{
"type": "tool",
"id": f"call-{i}",
"name": "bash",
"state": {
"status": "completed",
"input": {"command": True},
"content": [{"type": "text", "text": "exit code 0"}],
},
}
for i in range(3)
]
conn.execute(
"INSERT INTO session_message VALUES (?, ?, ?, ?, ?, ?, ?)",
(
"m",
"s",
"assistant",
1,
1,
1,
json.dumps({"content": content}, separators=(",", ":")),
),
)
conn.commit()
conn.close()
plugin = OpenCodePlugin(db_path=db_path)
project = plugin.discover_projects()[0]
sessions = plugin.scan_project(project)
print("sessions:", len(sessions), "calls:", sum(len(s.tool_calls) for s in sessions))
print("summaries:", [c.input_summary for c in sessions[0].tool_calls])
loops = detect_loops(sessions)
print("loops:", [(lp.tool, lp.count, lp.sample_input) for lp in loops])
digest = _build_digest(project, sessions, loops=loops)
print("digest chars:", len(digest))
print("digest contains malformed call:", "Bash: True" in digest)
```
</details>
## Runtime Rollout Safety
- Rollout-managed feature(s): `headroom learn` local session scanning
and digest formatting only; no proxy, provider, or runtime request path.
- Minimum rollout channel: N/A — library bug fix, not gated by a rollout
channel.
- Stable/default behavior changed: No. Valid string commands render
exactly as before; only malformed inputs that previously raised now
render.
- Kill switch / disable path: N/A — revert the commit to restore the
prior behavior.
- Unsafe override required: No.
- Qualification impact: None; no network, model, or rollout behavior is
touched.
- Rollback path: revert this commit/PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation — N/A, no
public API or docs change
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — no UI change.
## Additional Notes
The malformed value is real session data, not a synthetic edge case:
session `ses_e6bcaf904fd4blosS7bPwIvQ1f` in a local opencode v2 DB
stores `{"command": true}` for 8 Bash calls. The fix only affects
display/summary rendering, so loop identity, failure classification, and
recommendations are unchanged.
Co-authored-by: JavierLeon9966 <javier@lumineproxy.org>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
93331f2874 |
fix(proxy): retain final backend streaming usage (#4814)
## Description
Translated OpenAI streams (`/v1/chat/completions` through a backend such
as `anyllm` or LiteLLM) recorded the **first nonzero** usage snapshot
instead of the last one.
`_absorb` in `_stream_openai_via_backend` wrote a counter only when the
accumulator was still falsy:
```python
if key in usage and not stream_state.get(key):
stream_state[key] = usage[key]
```
A provider that reports climbing cumulative snapshots on successive
content chunks therefore had every later frame thrown away. Replaying
the two frames from the issue through the real proxy path, `/stats`
recorded 10 input / 1 output tokens and `/metrics` recorded 2 cache
reads — while the client received both frames intact. The bug is in
accounting after delivery, not a dropped upstream event.
Closes #4263
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `_parse_sse_usage_from_buffer` (openai branch): absent counters no
longer default to `0`. A frame that omits a field was previously
indistinguishable from one reporting an explicit zero, which is a
prerequisite for "absent keeps the last value".
- `_absorb`: assigns unconditionally, so the latest cumulative snapshot
wins. An absent field keeps its earlier value; an explicit zero still
replaces it.
- Removed `_parse_completion_tokens_from_sse_chunk`. It existed to catch
completion-only frames, but the buffered SSE parser already handles
those, and its `not stream_state["output_tokens"]` guard latched the
output count a second time.
- Replaced that helper's unit tests with eight regressions driving the
real `_stream_openai_via_backend` path through stream finalization, in
Chat and Responses shapes.
- `docs/content/docs/proxy.mdx`: one paragraph on the `POST
/v1/chat/completions` page documenting the accounting rule, in the
published docs site.
Net: -100/+113 lines, of which most deletions are the removed helper and
its implementation-only tests.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_streaming_usage_parser.py -q
........ [100%]
8 passed in 1.01s
$ ruff check headroom/proxy/handlers/streaming.py tests/test_streaming_usage_parser.py
All checks passed!
$ ruff format --check headroom/proxy/handlers/streaming.py tests/test_streaming_usage_parser.py
2 files already formatted
$ mypy headroom/proxy/handlers/streaming.py
Success: no issues found in 1 source file
```
Regression set — the new tests against the unfixed handler (`git stash`
of `streaming.py` only):
```text
$ python -m pytest tests/test_streaming_usage_parser.py -q
FAILED ...[latest-snapshot-chat] assert (10, 1, 2, 3) == (13, 5, 9, 1)
FAILED ...[latest-snapshot-responses]
FAILED ...[partial-chat] assert (10, 1, 2, 3) == (10, 5, 2, 3)
FAILED ...[partial-responses]
FAILED ...[explicit-zero-chat] assert (10, 1, 2, 3) == (0, 0, 0, 0)
FAILED ...[explicit-zero-responses]
6 failed, 2 passed in 1.07s
```
Six fail before the fix, all eight pass after — matching the issue
exactly.
Wider streaming and parser suites:
```text
$ python -m pytest tests/test_streaming_usage_parser.py tests/test_streaming_sse_malformed_events.py \
tests/test_provider_billed_input.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_handler_helpers.py \
tests/test_proxy_streaming_request_logger.py tests/test_route_advice.py tests/test_h2_stream_reset_retry.py \
tests/test_proxy_streaming_resilience.py tests/test_proxy_streaming_ratelimit_headers.py -q
3 failed, 607 passed in 10.21s
```
The three failures are
`tests/test_proxy_cache_ttl_metrics.py::test_stats_endpoint_reports_*`,
which raise `SystemExit: 78` from `_check_rust_core()` in `server.py`.
They are environmental (unbuilt Rust core in this checkout) and fail
identically on unmodified `upstream/main` with this branch stashed —
unrelated to this change.
## Real Behavior Proof
- Environment: Windows, Python 3.13.11, this branch, `PYTHONPATH` set to
the checkout.
- Exact command / steps: replayed the issue's two usage snapshots as a
Chat stream through `_stream_openai_via_backend` with a stub backend,
then read the resulting `RequestOutcome`.
- Observed result: the corrected counts from the issue; previously 10 /
1 / 2.
```text
provider_input_tokens=13 (expect 13)
output_tokens =5 (expect 5)
cache_read_tokens =9 (expect 9)
```
- Not tested: a live `anyllm`/LiteLLM upstream over real HTTP (no
credentials in this environment). The replay drives the identical
handler and parser path the proxy uses; the frames are byte-for-byte the
issue's.
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: token accounting for translated
OpenAI streams now records the final snapshot. This is the fix; counts
move up from truncated to complete.
- Kill switch / disable path: none beyond reverting this commit.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: `git revert`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Authorship is credited to @oxura, who diagnosed this in #4263 and wrote
the original patch on their fork; they were holding it back on the
ten-open-PR limit. This PR carries their change rebased onto current
`upstream/main` (their branch was cut from `fbf7bc32`, and
`streaming.py` has since drifted ~200 lines), with the commit message
and tests adapted to the current tree. Their `Authored-by` trailer is on
the commit.
One hunk from their patch was redirected: they documented the rule in
`wiki/proxy.md`, which `AGENTS.md:38` says is unpublished and must not
receive new content, so the paragraph now lives on the `POST
/v1/chat/completions` page in `docs/content/docs/proxy.mdx`. Thanks to
Devin Review for flagging it.
Co-authored-by: oxura <grozafroza44@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
199912e020 |
test(cli): assert the --learn/--no-learn warning text (#4800)
## Description `TestLearnNoLearnConflict::test_both_flags_warns_and_exits_zero` ends with `assert "both --learn and --no-learn" in result.output or (result.output is not None), ...`. The right side of the `or` is true for any run (`result.output` is a string), so the assertion cannot fail and the test passes without the warning. This PR drops the `or` so the test checks the warning text it is named for. Part of #4783 (found in a second pass; listed there as a test that cannot fail). It is a one-line test change, no product code. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) - [x] Test-only change ## Changes Made - `tests/test_cli_proxy_improvements.py`: the warning assertion is now `assert "both --learn and --no-learn" in result.output, result.output`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `ruff check` and `ruff format --check` (0.17.3) are clean on the file. ### Test Output ```text tests/test_cli_proxy_improvements.py -k TestLearnNoLearnConflict this branch: 4 passed mutation: headroom/cli/proxy.py `if learn and no_learn:` -> `if False and learn and no_learn:` test_both_flags_warns_and_exits_zero, this branch: FAILED test_both_flags_warns_and_exits_zero, main: passed ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, click 8.5.1, pytest 9.2.0, `main` at `3c66c91e`. - Exact command / steps: ran the four tests of `TestLearnNoLearnConflict`; then disabled the warning branch in `headroom/cli/proxy.py` locally and ran `test_both_flags_warns_and_exits_zero` with this change and with `main`'s test (the product file was restored afterwards). - Observed result: as in Test Output. `main`'s assertion passes with the warning removed; this change fails. - Not tested: other click versions (the assertion relies on `result.output` containing stderr, which is how click 8.5.1's `CliRunner` behaves), the whole suite, CI. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a - Stable/default behavior changed: no (test only). - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes (the four tests of the class; the whole suite was not run) - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) AI assistance: this change and the runs above were done with Claude (Sonnet 5.5), operated by me (HetCreep). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4800" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
d379efe90c |
test: skip tests marked live by default (#4786)
## Description
`pyproject.toml` declares a `live` marker ("opt-in multi-turn tests that
hit real upstream APIs; require provider keys"), and two test files
carry it: `tests/test_thinking_signature_scope_live.py` ("Opt-in:
requires a real key and is gated behind `pytest.mark.live`. Run with
`pytest -m live ...`") and `tests/test_realignment_live_multi_turn.py`
(which describes a "default `pytest -m "not live"` run that never
touches the network"). `addopts` was `-v --tb=short` with no `-m`
filter, so a plain `pytest` run on a machine with `ANTHROPIC_API_KEY` in
the environment, or in the project's `.env` (loaded by
`tests/_dotenv.py`), collected these tests and called the real API.
This PR adds `-m 'not live'` to `addopts`, so the default run deselects
them. `pytest -m live <file>` still selects them (a later `-m` replaces
this one).
Part of #4783
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `pyproject.toml`: `addopts = "-v --tb=short -m 'not live'"`, with a
three-line comment saying why and how to opt in.
- `tests/test_realignment_live_multi_turn.py`: the module-wide `live`
marker is replaced by `@pytest.mark.live` on the eight tests that need a
provider key, so the offline
`test_classify_auth_mode_routes_payg_vs_oauth` keeps running by default.
- `tests/test_live_marker_default_deselect.py`: collection-only test
that the default run deselects live tests, keeps the offline one, and
that `-m live` selects them again (fails against the old `addopts`).
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
This changes pytest configuration only, so no new test is added; the
proof is the collection result below (`--collect-only`, with a dummy
`ANTHROPIC_API_KEY`, nothing contacted).
### Test Output
```text
Before (origin/main pyproject.toml):
pytest tests/test_thinking_signature_scope_live.py tests/test_realignment_live_multi_turn.py --collect-only -q
15 tests collected
After (this branch):
same command -> 1 test collected (the offline test_classify_auth_mode_routes_payg_vs_oauth), 14 deselected
same command with -m live -> 14 tests collected
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, pytest 9.2.0,
`ANTHROPIC_API_KEY=sk-ant-dummy-collect-only` in the environment.
- Exact command / steps: `python -m pytest
tests/test_thinking_signature_scope_live.py
tests/test_realignment_live_multi_turn.py --collect-only -q`, with the
pyproject from `origin/main` and with this branch's, and once more with
`-m live`.
- Observed result: before, 15 tests collected (they would run and call
the API when a key is present); after, the 14 provider-calling tests are
deselected and the offline one is still collected; with `-m live`, 14
collected.
- Not tested: an actual live run (no provider key was used), and CI (the
CI workflows I read do not export a provider key to pytest, so this PR
is aimed at developer machines).
## Runtime Rollout Safety
- Rollout-managed feature(s): none; test configuration only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for the library and proxy. The
default `pytest` run no longer collects tests marked `live`.
- Kill switch / disable path: run `pytest -m live ...` (or pass any `-m`
expression).
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes (the
whole suite was not run; see Testing)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- If a CI job runs `pytest -m live` deliberately it keeps working,
because the later `-m` wins. I did not find such a job.
AI assistance: this change and the runs above were done with Claude
(Sonnet 5.5), operated by me (HetCreep).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4786"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
6665940656 |
fix(memory): return stored id from a background DirectMem0 save (#4759)
## Description `DirectMem0Adapter.save_memory(background=True)` returned an id from a second `_generate_id` call. That helper salts with the current time, so the returned id never matched the Qdrant point the background task wrote, and a later `memory_update` / `memory_delete` with it found nothing. The direct-write ids are now generated once, passed to the background task, and returned. On the Mem0 extraction path (no pre-extracted data), Mem0 assigns ids itself, so no id is known up front. The save is marked `_id_pending`, and `memory_save` reports `memory_id: None` instead of an invented id. The real id is available from `get_task_status(task_id)`. Refs #4362 (item 6 only). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/memory/backends/direct_mem0.py`: generate fact ids before scheduling the background task, then store and return those same ids. `_write_facts_to_qdrant` accepts optional `memory_ids`. Mem0-path background saves are marked `_id_pending`. Fixed the `_generate_id` docstring, which said the ids were "deterministic". - `headroom/memory/system.py`: `memory_save` returns `memory_id: None` when the id is pending. - Background saves snapshot `facts` before generating ids and schedule that same snapshot, so a caller mutating its list after queueing can't make the task fail. The returned `_all_memory_ids` is a copy for the same reason. - `tests/test_memory/test_direct_mem0_background_ids.py`: 4 regression tests, using a fake Qdrant client and a fake Mem0 client. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_memory/test_direct_mem0_background_ids.py -v test_background_save_returns_stored_fact_id PASSED test_mutating_lists_after_queueing_does_not_break_the_save PASSED test_background_mem0_extraction_does_not_invent_an_id PASSED test_memory_save_tool_withholds_a_pending_id PASSED 4 passed in 0.36s ``` ## Real Behavior Proof - Environment: macOS; Qdrant v1.15.5 server in Docker (`localhost:6333`); `mem0ai` 2.2.1 and `qdrant-client` 1.20.0 (real `Memory.get/update/delete`); embeddings from Ollama 0.40.6 `nomic-embed-text` through its OpenAI-compatible API (`OPENAI_BASE_URL=http://localhost:11434/v1`). `DirectMem0Adapter` ran unmodified, with `enable_graph=False`. - Exact command / steps: a script on a fresh 768-dim collection: `save_memory(facts=[...], background=True)` → append to the caller's `facts` list → `flush_pending()` → `get_memory` / `update_memory` / `get_memory` / `delete_memory` / `get_memory`, all using the **returned** id. Run once on this branch and once on `main` (`a385ea8e`) as a control. - Observed result: Returned IDs resolve to the stored points; before/after results are below. | | this branch | `main` | |---|---|---| | task status | completed | completed | | returned id is a stored Qdrant point | **True** | False | | `get_memory(returned id)` | `'Prefers Python over JavaScript'` | `None` | | `update_memory` → `get_memory` | `'Prefers Rust over Python'` | `Memory with id 0ad08f… not found` | | `delete_memory(returned id)` | `True`, 0 points left | `False`, 2 points left | | fact appended after queueing | not stored (snapshot) | stored under an id never returned to the caller | - Not tested: the Mem0 LLM-extraction background path (`_id_pending`; it needs an OpenAI LLM and none was available); Neo4j graph writes; hosted Qdrant; OpenAI's own embedding API. ## Runtime Rollout Safety - Rollout-managed feature(s): None - Minimum rollout channel: NA - Stable/default behavior changed: Only background saves on `DirectMem0Adapter` (opt-in via `background=true` or `async_writes=True`). The proxy doesn't use background saves. Synchronous saves are unchanged. - Kill switch / disable path: Don't use background saves. - Unsafe override required: NA - Qualification impact: None - Rollback path: Revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4759" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
177fab3782 |
docs: add guidance for scripted PR/issue comment posting (#4760)
## Description Documents safe patterns for agent/script-authored PR and issue comments, so replies like `System.Collections.Hashtable` (seen on #4726) don't ship unnoticed. Adds a "Posting PR/issue comments from scripts or agents" section to `CONTRIBUTING.md` with shell-agnostic rules: file-based bodies and a read-back requirement. Refs #4757. Only the documentation checkbox of the issue is covered here, so this does not close it; the issue's decision record is below. ## Type of Change - [x] Documentation update ## Changes Made - `CONTRIBUTING.md`: new shell-neutral section: single-string file-based bodies, no collections in `-f`/`-F`, read-back verification, delete-and-repost for bad comments. **Decision on the optional follow-ups (acceptance criterion 2):** deferred. A governance-workflow regex check and a `gh-reply` helper script would add new CI/script surface to maintain for a one-off failure mode. The documented read-back step is the main guardrail. Happy to follow up if maintainers want either. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text Docs-only change; no code touched. Reviewed the rendered Markdown diff. ``` ## Real Behavior Proof - Environment: any OS/shell with `gh` - Steps: reviewed `git diff origin/main -- CONTRIBUTING.md`; confirmed the section renders before "Development setup" and contains no shell-specific syntax. - Not tested: not executed against a live PR (docs only). ## Runtime Rollout Safety - Rollout-managed feature(s): N/A (docs only) - Minimum rollout channel: N/A - Stable/default behavior changed: No - Kill switch / disable path: N/A - Unsafe override required: No - Qualification impact: None - Rollback path: revert the commit ## Review Readiness - [x] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [x] I have made corresponding changes to the documentation - [x] I did **not** edit `CHANGELOG.md` --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
6b41b6e451 |
feat(proxy): add cli and settings knob for ccr_inject_system_instructions (#4642)
<!-- maintainer-current-main --> ## Maintainer latest-main checkpoint Published `d13ba17dff65a5b31253ef3cf0775ffc2504d8da` contains current upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 150 CLI/settings/Gemini tests passed. Earlier live behavior proofs below retain their commit/platform scope; contributor runtime logic is unchanged. The common shard-2 failure was independently confirmed on main's hosted job 127069149772 and locally (four failures, six passes): the Gemini memory fixture mocked an unused body reader and provided no request stream. The maintainer test-only follow-up now constructs a real FastAPI/ASGI Request carrying JSON bytes and exercises the actual body reader. All six test-function ASTs/assertions are unchanged, and all 10 parameterized Gemini cases pass. Touched-file Ruff/format and whitespace checks pass. This fixes the inherited fixture instead of asking contributors to run CI or weakening the memory/content assertions. Fresh exact-head hosted qualification is requested. Nothing merged. <!-- /maintainer-current-main --> ## Description Closes #4636. `ccr_inject_system_instructions` (which instructs models how to retrieve compressed context when CCR markers are present) previously existed as a field on `ProxyConfig` but could only be toggled programmatically via the Python API. Operators deploying `headroom proxy` as a container or standalone service had no CLI flag, environment variable, or settings store entry to enable it without wrapping the process in custom Python code. This PR exposes `ccr_inject_system_instructions` as: 1. CLI flag: `--ccr-inject-system-instructions` 2. Environment variable: `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS` 3. Curated settings store field: `ccr_inject_system_instructions` in `settings_store.SETTINGS` 4. Server environment configuration: loaded in `_proxy_config_from_env()` The default remains `False` preserving exact existing behavior. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `headroom/cli/proxy.py`: Add `@click.option("--ccr-inject-system-instructions", is_flag=True, envvar="HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS", ...)`, add parameter to `proxy()`, and thread it into `ProxyConfig`. - `headroom/settings_store.py`: Add `SettingField("HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS", "ccr_inject_system_instructions", ...)` to `SETTINGS` registry so it is validated, saved to `settings.json`, and surfaced on `/settings`. - `headroom/proxy/server.py`: Read `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS` in `_proxy_config_from_env()` for multi-worker and container deployments. - `docs/content/docs/proxy.mdx`: Document `--ccr-inject-system-instructions` and `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS` in the CCR configuration table. - `tests/test_cli_proxy_env.py`: Add test cases verifying `--ccr-inject-system-instructions`, `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS`, `settings.json`, and `_proxy_config_from_env()` correctly populate `ProxyConfig.ccr_inject_system_instructions`. ## Testing - [x] Unit tests pass (`pytest tests/test_proxy/test_settings_store.py tests/test_cli_proxy_env.py`) - [x] Linting passes (`ruff check`) - [x] Formatting passes (`ruff format --check`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy/test_settings_store.py -q 40 passed, 1 warning in 2.18s $ pytest tests/test_cli_proxy_env.py -k "test_ccr_inject_system_instructions or test_settings_file_reaches_ccr_inject_system_instructions or test_proxy_config_from_env_reads_ccr_inject_system_instructions" -q 5 passed, 91 deselected in 3.42s $ ruff check headroom/cli/proxy.py headroom/settings_store.py headroom/proxy/server.py tests/test_cli_proxy_env.py All checks passed! $ ruff format --check headroom/cli/proxy.py headroom/settings_store.py headroom/proxy/server.py tests/test_cli_proxy_env.py 4 files already formatted ``` ## Real Behavior Proof - Environment: Windows 11 x86_64, Python 3.14.5, Click CliRunner test harness and settings store test harness. - Exact command / steps: Executed `runner.invoke(main, ["proxy", "--ccr-inject-system-instructions"])`, verified captured ProxyConfig; tested env var `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS=1`, verified ProxyConfig; tested `settings.json` with `{"ccr_inject_system_instructions": true}`, verified ProxyConfig; tested `_proxy_config_from_env()`. - Observed result: In all test cases, ProxyConfig.ccr_inject_system_instructions resolved to True when enabled and False by default, matching the behavior of sibling knobs like HEADROOM_CCR_INLINE_RESOLVE and HEADROOM_CCR_BUFFERED_CHAT_STREAMING. - Not tested: Live Kubernetes deployment or multi-node container cluster. ## Runtime Rollout Safety - Rollout-managed feature(s): none, opt-in configuration knob - Minimum rollout channel: n/a - Stable/default behavior changed: no, default remains False - Kill switch / disable path: omit the flag or unset HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS - Unsafe override required: no - Qualification impact: none while disabled - Rollback path: git revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Follows the exact pattern established by `--ccr-inline-resolve` and `--ccr-buffered-chat-streaming` (#4631). <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4642" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ## Maintainer runtime and integration qualification Published `9c0b0a10d63227554350a3ba2cb936e09cea1286` includes upstream main `ee975d49c0b742e34b09df50a01c7b5680e85cdc` with a clean merge and no contributor logic changes. Main is actively advancing, so final qualification must check the then-current main again. Windows/Python 3.12.13: `python -m pytest tests/test_cli_proxy_env.py tests/test_proxy/test_settings_store.py -q` reports **140 passed** with the existing development native extension. Touched-file Ruff, format and whitespace checks pass; no fresh native build claimed. Real CLI proof (no replaced run_server or provider transport): launch the actual Click entry point `from headroom.cli import main; main()` in six isolated subprocesses with `proxy --mode token --no-optimize --no-cache --no-rate-limit --no-ccr-proactive-expansion --no-memory-tools --no-memory-context --disable-kompress --anthropic-api-url http://127.0.0.1:<fixture-port>`. Each process has an isolated workspace/settings file, offline/beacon opt-out and a real CompressionStore entry created before launch; GET `/v1/retrieve/{hash}` confirms ownership. Send `/v1/messages` with that stored hash in a historical tool-result marker, through the actual TCP proxy to a local HTTP Anthropic-shaped provider fixture, and inspect the received upstream JSON. Observed: all six calls return HTTP 200. Default: no `Compressed Context Available` instructions. CLI flag, environment `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS=1`, and a settings file with `{"ccr_inject_system_instructions":true}` each produce the instructions upstream. `--no-ccr` and `--lossless`, both requested alongside the flag, produce neither instructions nor a retrieval tool despite the owned historical marker. This verifies the disabled-CCR finding is addressed on the real request path. The documented warm-prefix/cache-mode skip remains an intentional limitation; this run uses token mode. Not tested: live cloud models, Kubernetes/multiple nodes, every provider wire shape, warm-cache replay or a fresh native build. The local HTTP fixture is synthetic; request/configuration/ownership paths and TCP transport are real. Hosted CI must finish on the final head; nothing merged. ### Maintainer current-main requalification Exact head `b38e5a5937cf09c3b5127f3f066e0384db235955`, upstream main `ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`. Clean integration with upstream/main |
||
|
|
e9301f34c1 |
feat(tray): opt-in system tray icon with savings counter, dashboard and update alert (#4726)
## Description
Adds an **opt-in system tray icon** for Headroom while it runs (Windows
first; macOS/Linux planned). The icon is the Headroom logomark and its
right-click menu shows a compact savings counter (tokens and estimated $
saved), opens the dashboard, alerts when an update is available, and
lets you quit.
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `headroom/tray/` package: `core.py` (GUI-free stats
polling/formatting), `icon.py` (logomark drawn with Pillow, no binary
assets in the wheel), `app.py` (pystray app), `launcher.py` (spawns the
tray as a child of the proxy).
- `headroom proxy --tray` starts the tray with the proxy; `headroom tray
--port N` attaches to a running proxy.
- Menu: live `▼ tokens · ~$ saved` counter (from `/stats` →
`persistent_savings.lifetime`, polled every 5s), Open Dashboard, "Update
available" (only when newer; reuses `headroom.update_check`, honours
`HEADROOM_UPDATE_CHECK=off`; one-time notification), Quit.
- Icon states: normal, orange dot when an update exists, grey when the
proxy is offline.
- New optional extra `headroom-ai[tray]` (`pystray`, `pillow`).
Unsupported platform / missing extra: `--tray` prints why and the proxy
starts normally.
- Docs: `docs/content/docs/system-tray.mdx` + nav entry; screenshots in
`docs/public/assets/tray/`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
> python -m ruff check headroom\tray headroom\cli tests\test_tray_core.py
All checks passed!
> python -m mypy headroom\tray headroom\cli\tray.py
Success: no issues found in 7 source files
> python -m pytest tests\test_tray_core.py tests\cli tests\test_offline_egress_chokepoint.py tests\test_proxy\test_settings_store.py -q
293 passed, 2 skipped
```
(Scoped runs; I did not run the full suite or `mypy headroom` over the
whole package.)
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.13, 3840x2160 display, `pystray`
+ `pillow` 12.3.0, branch based on `upstream/main` @
|
||
|
|
ba26fd7d92 |
docs(install): add Windows auto-start and warm-join happy path (#4725)
## Description Documents the Windows happy path for persistent auto-start and warm-join. Behavior already exists upstream (boot + 5-minute health tasks, S4U-to-session fallback from #4627, per-port `wrap` reuse/recovery); the docs had no end-to-end Windows walkthrough. Closes # ## Type of Change - [x] Documentation update ## Changes Made - Add "Windows happy path: auto-start and warm-join" to `docs/content/docs/persistent-installs.mdx`. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text Docs-only; no automated tests apply. ``` ## Real Behavior Proof - Environment: Windows 11, non-elevated user, headroom-ai 0.43.1 runtime; persistent-task profiles `local-agents` (9010), `local-copilot` (9012). - Exact command / steps: `headroom install status --profile local-agents`; `headroom wrap claude --port 9010 -- --version` (joined the running proxy, no second proxy); 3 concurrent `headroom wrap codex --port 9555 -- --version` on a cold port (one "Starting Headroom proxy", others joined); confirmed S4U `schtasks /Create` returns "Access is denied" for this user, matching the documented fallback. - Observed result: proxy healthy and reused; Codex traffic observed in proxy log; one proxy per port. - Not tested: S4U/logged-off recovery (denied on this account); `headroom install apply` against 0.44 from a clean machine. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: no - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert docs commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4725" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
66e0086952 |
feat(opencode): add repeatable --extra-model flag for custom wrap-time models (#2364)
## Description
`headroom wrap opencode` injects a hardcoded model list
(`HEADROOM_OPENCODE_MODELS`) into the `headroom` provider block of
`opencode.json`. There's no way to add custom models (for example a
free-tier DeepSeek) at wrap time. Hand-patching `opencode.json` is the
only workaround, and the next wrap overwrites it.
This PR adds a repeatable `--extra-model` flag and a
`HEADROOM_OPENCODE_EXTRA_MODELS` env var. Both take
`id[=name[=context_window[=output_limit]]]` specs. Extra models are
merged into the injected provider block alongside the built-ins, and
flag specs win over env specs on id collision. `name` defaults to the
id, `context_window` to 200000, and `output_limit` to `min(16384,
context_window)`.
Fields are separated by `=` rather than `:` because model ids routinely
contain `:`. Ollama ids are `name:tag` (`qwen2.5-coder:7b`), and
OpenRouter free-tier ids end in `:free`. With a `:` separator those were
silently cut down to a different id (Devin review, 2026-10-03).
The PR also fixes the wrap/unwrap lifecycle the flag depends on. A wrap
that starts without an `opencode.json` used to write the file with no
backup and no record of what Headroom put in it, so `headroom unwrap
opencode` left the injected models and the proxy base URL behind while
reporting that OpenCode was no longer routed through Headroom, and a
re-wrap under `--openai-api-url` kept an extra model Headroom had
injected but no longer advertised. Wrap now records what it owns in
`opencode.json.headroom-managed.json`, and unwrap uses that record: it
removes the model ids and the base URL Headroom wrote, keeps everything
else in the file, drops an extra model that is no longer requested, and
deletes the config file when Headroom created it and nothing else is
left in it.
Closes #2359
## Type of Change
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [x] Documentation update
- [ ] Refactoring (no functional changes)
## Changes Made
- `headroom/providers/opencode/config.py`:
- New `parse_extra_model_spec()`, which splits on `=` with an optional
fourth `output_limit` field and raises `ValueError` on an empty id or a
non-integer / non-positive limit.
- New `extra_models_from_env()`, which reads comma-separated specs from
`HEADROOM_OPENCODE_EXTRA_MODELS`.
- An `extra_models` parameter threaded through
`headroom_provider_entry()`, `_render_provider_block()` and
`inject_opencode_provider_config()`.
- With `keep_user_entries` (#3125's `--openai-api-url` gate),
`--extra-model` entries win over a kept user entry with the same id.
- New managed-state sidecar (`opencode.json.headroom-managed.json`)
recording the model ids and base URL Headroom injected and whether
Headroom created the config file. `inject_opencode_provider_config()`
writes it on every injection; `record_opencode_config_origin()` marks a
config that Headroom creates because none existed.
- `remove_managed_opencode_provider()` drops the recorded model ids, the
recorded base URL, and the provider block once nothing else is left in
it. Entries the user added or edited are preserved.
- `_drop_stale_managed_models()` prunes an extra model Headroom injected
but no longer advertises, including an extra that overrode a built-in id
and is dropped again, so a re-wrap without the flag restores the
built-in defaults rather than keeping the old override.
- `snapshot_opencode_config_if_unwrapped()` no longer snapshots a config
Headroom wrote itself as pre-wrap content.
- `headroom/providers/opencode/runtime.py`: extras also reach the
launch-time `OPENCODE_CONFIG_CONTENT`.
- `headroom/cli/wrap.py`:
- A repeatable `--extra-model` option on `wrap opencode`. Env and flag
specs are parsed up front, and a malformed spec raises
`click.BadParameter`. The merged dict goes to both
`inject_opencode_provider_config` call sites (prepare-only and launch)
and to the launch env. Also adds a help text and docstring example.
- `wrap opencode` records the config origin before the first config
mutation.
- `unwrap opencode` undoes the recorded provider entries when there is
no pre-wrap backup, keeps the rest of the file, and removes a config
Headroom created once nothing Headroom-owned is left in it.
- `docs/content/docs/opencode.mdx`: new "Adding models" subsection
(format, defaults, Ollama/OpenRouter example, built-in id override, the
env var's no-comma limitation) and an env-var table row. The subsection
and the "How It Works Under The Hood" unwrap step now describe the
managed-state record and the automatic cleanup it enables.
- Tests:
- Spec parsing: full spec with output, defaults, the output cap at small
contexts, colon-bearing ids (`qwen2.5-coder:7b`,
`deepseek/deepseek-chat:free`, a name containing `:`), and invalid specs
including a bad output limit.
- Env parsing, provider-entry merge, and end-to-end inject. Also
@JerrettDavis's collision test across both `keep_user_entries` values.
- CLI: the flag reaches `opencode.json`, the env var persists, both
reach `OPENCODE_CONFIG_CONTENT` on launch, the flag wins over a kept
user entry under `--openai-api-url`, and a malformed spec exits with a
usage error.
- Managed state: the sidecar records the injected ids and base URL, a
config the user already had is not marked as Headroom-created, a re-wrap
does not snapshot Headroom's own config, a dropped extra model is pruned
(including an override of a built-in id, which returns to the built-in
defaults), a user model and a user-edited model survive, and the
provider removal keeps unrelated user entries.
- Lifecycle: a fresh wrap followed by an unwrap removes the config file
again (with and without the MCP entry), user content added after a fresh
wrap survives the unwrap, a re-wrap without the extra model drops the
stale one, and a directly injected provider entry is undone by the
unwrap callback as well.
- A user setting that is false, zero or an empty string is content, not
an empty section: such a setting survives the unwrap instead of costing
the user the whole file.
## Testing
- [x] Unit tests pass locally
- [x] Lint/format/type checks pass locally
- [x] New tests added for new functionality
The new parsing tests fail against the previous `:`-based parser:
```text
FAILED tests/test_providers_opencode_config.py::test_parse_extra_model_spec_keeps_colons_in_ids[qwen2.5-coder:7b-...]
FAILED tests/test_providers_opencode_config.py::test_parse_extra_model_spec_keeps_colons_in_ids[deepseek/deepseek-chat:free=DeepSeek Free-...]
FAILED tests/test_providers_opencode_config.py::test_parse_extra_model_spec_full
...
================ 21 failed, 2 passed, 103 deselected in 0.86s =================
```
With this branch (head
|
||
|
|
863b3c7cc7 |
fix(opencode): percent-encode the project name in x-headroom-project (#4780)
## Description The OpenCode plugin set `x-headroom-project` to the raw project name in `plugins/opencode/src/transport.ts` (two sites) and `plugins/opencode/src/plugin.ts`. A header value must be a ByteString, so a project name with any character above U+00FF (Thai, Chinese, Cyrillic, Arabic, `€`, emoji and so on) made `Headers.set` throw a `TypeError` inside the patched `fetch` and `https.request`, and every routed request failed. `headroom/cli/wrap.py` already percent-encodes this header for the same reason, and the proxy decodes it (`sanitize_project_name` calls `unquote`). This PR does the same in the plugin through one helper, `projectHeaderValue`, used at the three sites. Header values that contain only characters from the URL-unreserved set are sent as before. Closes #4779 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `plugins/opencode/src/transport.ts`: new exported `projectHeaderValue(project)` (`encodeURIComponent`), used for `x-headroom-project` in the fetch and Node request paths. - `plugins/opencode/src/plugin.ts`: the model headers set by `routeModelsThroughProxy` use `projectHeaderValue(project)`. - `plugins/opencode/src/transport.test.ts`: two tests with a Thai, Chinese, Cyrillic and Polish project name, one through `fetch` and one through Node `https.request` against a loopback server. - `headroom/providers/opencode/_dist/index.js` and `hook-shim/handler.js`: regenerated with `npm run build:standalone` (the CI step compares both with a fresh build; the staged blobs equal the build output). ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed This is a TypeScript-only change, so the Python checks above do not apply; the plugin equivalents were run (Node v24.22.2, vitest, `npm ci --ignore-scripts` from the committed lockfile, Ubuntu on WSL2). ### Test Output Fail before (`transport.ts` from `3c66c91e`, new tests kept): ```text vitest run src/transport.test.ts -t "non-Latin-1" × percent-encodes a non-Latin-1 project name instead of throwing on routed fetch calls × percent-encodes a non-Latin-1 project name on routed Node https.request calls TypeError: Cannot convert argument to a ByteString because the character at index 8 has a value of 3650 which is greater than 255. TypeError: Cannot convert argument to a ByteString because the character at index 0 has a value of 3650 which is greater than 255. Tests 2 failed | 47 skipped (49) ``` Pass after (this branch): ```text vitest run src/transport.test.ts -t "non-Latin-1" Tests 2 passed | 47 skipped (49) vitest run (whole plugins/opencode suite) Test Files 4 passed (4) Tests 81 passed (81) npm run typecheck -> exit 0 npm run build:standalone -> success; the committed bundle equals this build ``` ## Real Behavior Proof - Environment: Ubuntu on WSL2, Node v24.22.2, the committed wheel bundle `headroom/providers/opencode/_dist/index.js` (not vitest), and a loopback `node:http` server standing in for the proxy. - Exact command / steps: load the bundle (`origin/main` for "before", this branch for "after"), call `default.server({ project: { id: "p" }, directory: "/work" }, { proxyUrl: "http://127.0.0.1:<port>", project })`, then `fetch("https://api.anthropic.com/v1/messages", { method: "POST", body: "{}" })`, once with `project = "my-project"` and once with a Thai, Chinese and Cyrillic string. - Observed result: before, `my-project` was sent as `my-project` and the Thai/Chinese/Cyrillic name made `fetch` throw `TypeError: Cannot convert argument to a ByteString ... 3650 ... greater than 255` with 0 requests recorded. After, `my-project` is still sent as `my-project`, and the multilingual name reaches the recorder as `%E0%B9%82...%2F%E4%B8%AD%E6%96%87%2F%D0%9F...` which `decodeURIComponent` turns back into the original string. - Not tested: a real Headroom proxy (the recorder only shows what is sent; the decode step is from reading `headroom/proxy/project_name_policy.py`), OpenCode 1.x or 2.x hosts, and anything outside the proxy that reads the header. ## Runtime Rollout Safety - Rollout-managed feature(s): none; OpenCode plugin transport only. - Minimum rollout channel: n/a - Stable/default behavior changed: only for a project name with characters outside the URL-unreserved set. Those are now percent-encoded (a path such as `/home/u/app` is sent as `%2Fhome%2Fu%2Fapp`) and the proxy decodes them; plain names like `my-project` are unchanged. - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - The plugin test for `routeModelsThroughProxy` was not extended; the `plugin.ts` site is the same one-line call to the helper, covered by the bundle proof above. - The shell env variable `HEADROOM_PROJECT` is unchanged: it is an environment value, not a header. AI assistance: this change, its tests and the runs above were done with Claude (Sonnet 5.5), operated by me (HetCreep). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4780" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
87c12931fb |
fix(opencode): stop routing a host's bare /messages or /responses to the proxy (#4781) (#4791)
## Description A `POST` whose path ends in a wire endpoint but that sits at a host's origin root (`/messages`, `/responses`, `/chat/completions`) is no longer routed to the proxy unless that origin is a configured provider base URL. Closes #4781. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `plugins/opencode/src/transport.ts` — `hasWireRootBefore` no longer treats "no segment before the endpoint" as a wire root, so a bare `/messages` at an unknown host goes direct. - `plugins/opencode/src/transport.test.ts` — the bare-endpoint assertions from #4758 move into a #4781 case that pins both directions: direct by default, routed once the base is registered; the #4758 case keeps the prefixes that still route. - `plugins/opencode/README.md` — states the actual routing rule, and the two shapes that still route without a configured base. - `headroom/providers/opencode/_dist/index.js`, `headroom/providers/opencode/hook-shim/handler.js` — rebuilt, since the `opencode-plugin` gate compares bytes against `dist-standalone`. - `plugins/opencode/scripts/live-proof-4781.mjs` — the live proof below, committed so it can be rerun. ## Why #4758 stopped a suffix match from routing non-LLM hosts, but kept bare endpoints routable "because compatible gateways use those paths". That left exactly one non-LLM shape still leaking: `POST https://rest.messagebird.com/messages` with `Authorization: AccessKey ...` went to the proxy, carrying a non-LLM credential (#4781). Becoming the bare path is not evidence of a provider — it is equally the shape of a messaging API. A provider mounted at an origin root is still routed, because its base URL *is* a configured wire root (`isConfiguredWireRoot` matches every path on that origin when the base path is empty), which is how OpenCode reports `provider.<id>.options.baseURL`. ```ts return ( last !== undefined && (isVersionSegment(last) || IMMEDIATE_WIRE_ROOTS.has(last) || segments.some((s) => NAMED_WIRE_ROOTS.has(s))) ); ``` Everything else is untouched: version prefixes (`/v1`, `/v1beta`, `/v4`), immediate `api`, named roots (`backend-api`, `codex`, `openai`, `anthropic`, `zen`), Gemini colon actions, configured bases, `excludeHosts`, and the method gate. ## Testing - [x] New tests added for new functionality - [x] Manual testing performed `plugins/opencode` is a TypeScript package, so its gates are npm scripts rather than `pytest` / `ruff` / `mypy`; `npm audit` and the rest run in the repo's own CI. ### Test Output ```text > vitest run Test Files 4 passed (4) Tests 80 passed (80) > node node_modules/@typescript/native/bin/tsc --noEmit (clean, no output) > node scripts/live-proof-4781.mjs === decisions (real sockets, real fetch, real built plugin) === PASS MessageBird-shaped POST /messages -> left alone PASS MessageBird batches POST /messages/batches -> left alone PASS LLM control POST /v1/messages -> REROUTED to proxy PASS LLM control POST /v1/chat/completions -> REROUTED to proxy === what reached the proxy === POST /v1/messages x-headroom-base-url=http://localtest.me:8899 original-path=null POST /v1/chat/completions x-headroom-base-url=http://localtest.me:8899 original-path=/v1/chat/completions === MessageBird-shaped requests that reached the proxy: 0 === === what reached the upstream directly === POST /messages auth=AccessKey FAKE-TEST-VALUE (no proxy headers: true) POST /messages/batches auth=AccessKey FAKE-TEST-VALUE (no proxy headers: true) ALL CASES PASS ``` Before the fix, same script, source reverted to `HEAD~1` and the bundle rebuilt: ```text === decisions (real sockets, real fetch, real built plugin) === FAIL MessageBird-shaped POST /messages -> REROUTED to proxy PASS MessageBird batches POST /messages/batches -> left alone PASS LLM control POST /v1/messages -> REROUTED to proxy PASS LLM control POST /v1/chat/completions -> REROUTED to proxy === what reached the proxy === POST /messages x-headroom-base-url=http://localtest.me:8899 original-path=null ``` Reverting only `src/transport.ts` also fails the unit test (`1 failed | 79 passed`), satisfying "a test that fails before your fix and passes after". ## Real Behavior Proof - Environment: Windows 11 (win32), Node v26.10.0, this branch of `plugins/opencode`, committed wheel bundle `headroom/providers/opencode/_dist/index.js` loaded via `plugin.server(...)` as `headroom wrap opencode` does; upstream and proxy are local HTTP fixtures on real TCP sockets and `globalThis.fetch` is Node's undici, not a stub. - Exact command / steps: `cd plugins/opencode && npm ci && npm run build:standalone && node scripts/live-proof-4781.mjs`, which POSTs a MessageBird-shaped SMS payload and two LLM control requests to `http://localtest.me:8899` (resolves to 127.0.0.1 but is not a loopback spelling, so traffic really leaves the process) and records which server received each one. - Observed result: after the patch `POST /messages` and `POST /messages/batches` reach the upstream directly with no `x-headroom-base-url` header and the proxy receives zero MessageBird-shaped requests, while `/v1/messages` and `/v1/chat/completions` still reach the proxy with the upstream origin header; with `src/transport.ts` reverted to `HEAD~1` the same run reroutes `POST /messages` to the proxy, reproducing the reported leak over a real socket (full output in `Testing` → `Test Output`). - Not tested: a real MessageBird account and the real `rest.messagebird.com` host, since the payload and `AccessKey` are fake and routing is decided before DNS; the proxy's handling of an SMS body, which never arrives after this change; and a live OpenCode host process, since the fixture drives the plugin's own `server()` entry rather than an OpenCode session. ## Runtime Rollout Safety - Rollout-managed feature(s): none, no feature flag is involved. - Minimum rollout channel: next released `headroom-opencode` / wheel. - Stable/default behavior changed: yes, deliberately — a `POST` to `/messages`, `/responses` or `/chat/completions` at a host's origin root now goes direct unless that origin is a configured provider base URL; users whose gateway is configured via `provider.<id>.options.baseURL` are unaffected. - Kill switch / disable path: `excludeHosts` / `HEADROOM_OPENCODE_EXCLUDE_HOSTS` forces a host direct, and configuring the base URL or listing it in `wireRoots` forces a bare-path provider through the proxy. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert the commits; the transport and the committed bundle revert together. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title ## Additional Notes Residual, deliberately not solved and now stated in the README: `/api/v2/messages` and `/v2/responses` still route. No path shape separates those from a gateway rooted at `/api` or `/v2`; only the host or the request body could, and reading the body inside a sync `fetch` wrapper buys a new class of failure for a shape `excludeHosts` already covers. I did not add a known-host allowlist either: it can only recognise hosts someone already thought of, and it regresses compatible gateways silently — the failure mode #4747 was filed about. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
5023c84e52 |
fix(proxy): refuse a private x-headroom-base-url on the async chat and responses path (#4879)
## What changes Main currently fails four tests in `test_grok_shared_proxy_call_sites.py` and `test_upstream_guard.py`. Two changes to `handlers/openai.py` merged independently: - One made `_resolve_openai_upstream_base` reject a non-public `x-headroom-base-url` with 400. - The other moved the chat and responses handlers onto a new `_resolve_openai_upstream_base_async`. It was written before the rejection existed, so it still logged "ignoring" and fell back to the configured upstream. Since the handlers call the async resolver, a private override on `/v1/chat/completions` and `/v1/responses` was served from the default upstream instead of being refused. Both resolvers now refuse through one shared `_reject_unsafe_openai_override`, so they can't diverge again. The guard check still runs off the event loop. The change is source-only in one file, with no test changes: the existing tests on main already cover it. ## Verification - `tests/test_grok_shared_proxy_call_sites.py`, `tests/test_upstream_guard.py`, `tests/test_proxy/test_openai_upstream_guard_off_loop.py`, and the two OpenAI cache-key/stability suites: 109 passed. The four that fail on main now pass. - `tests/test_proxy -k "openai or upstream or base_url"`: 90 passed. - `ruff check` and `ruff format --check` are clean. mypy reports no errors in `openai.py`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4879" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d6df127d57 |
feat(compress): accept Gemini and Bedrock Converse bodies on the gateway turn (#4679)
## Description
`/v1/compress` in gateway mode (a request with a `gateway` block) now
accepts two more native request bodies:
- Gemini `generateContent`
- Bedrock Converse
It compresses their text slots and returns `body` in the same native
shape. This lets a gateway like LiteLLM compress native Gemini and
Bedrock passthrough traffic, which it couldn't before.
`config.mode: "ccr"` is rejected for these shapes with a 400. Every
other mode is marker-free for them.
(Was stacked on #4617, which has merged; the branch now carries current
`main`.)
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### How
This works the same way `gateway_responses` does for Codex: it builds a
view instead of translating the body.
- Every compressible text slot becomes one chat-shaped pseudo message.
- Compressed text is written back by its path in the body.
- Everything else stays byte-identical: `thoughtSignature`, thought
parts, `functionCall`, `inlineData`, `toolUse`, `cachePoint`,
`reasoningContent`, `tools`/`toolConfig`.
- Tool calls appear in the view, so tool-name rules still apply. `Read`
stays verbatim, for example.
- A JSON slot is only written back if it still parses to the same JSON
type.
- If the compressed list doesn't line up with the slots, the original
transcript is forwarded and a warning is logged.
### CCR on native bodies (review fix)
A native body can't carry `headroom_retrieve`. Its `tools`/`toolConfig`
are forwarded untouched, and `create_ccr_tool_definition` only builds
chat, Anthropic and Responses shapes. It also never gets a re-drive. So
a CCR marker in a native body points at content the model can never
fetch. Before this fix, `config.mode: "ccr"` on a Gemini or Converse
gateway turn did exactly that: log output came back as a marker with
`ccr_hashes: [<hash>]` and `obligations: []`.
- `handle_compress` now answers `ccr` on a native view with `400
invalid_request`. The message names the shape, explains why, and lists
the supported modes.
- The check runs where the mode is chosen, right after `attach_view` has
recorded the shape. That is before any compression, session work or CCR
store write.
- **The rejection does not depend on `gateway.can_redrive`.** Even a
gateway that can re-drive has no way to put the retrieval tool into a
native request, so there is no capability under which native `ccr`
works.
- **Without a `gateway` block nothing changes.** A Gemini body is
already a 400 (no `messages`). A Converse body goes through the plain
messages path, where its untyped blocks are opaque: 0 tokens saved, no
markers in any mode, `ccr` included (checked). So the rule is
consistent: reject `ccr` wherever Headroom reads a native shape.
- The bypass header still returns the body untouched with no
compression, in any mode.
- I checked every other mode on the same marker-producing input
(default, `lossless`, `lossy_inline`, `lossless_then_lossy`): no marker
and no hash in any of them. These modes build pipelines with
`ccr_inject_marker=False, ccr_enabled=False`. Stale-Read notices are
hashless in marker-free modes since #4678.
- **Second defect, same family:** with `HEADROOM_DEDUPE=1`, cross-turn
dedup wrote `[↑NL same as msg 4]` into a Converse tool result. That `4`
counts view messages (system blocks, tool calls, result slots), not the
caller's `messages`, so it named the wrong message. Native views now set
`cross_turn_dedup_recoverable=False`, as `lossless` mode already did.
- The `not self.native_shape` guards on CCR arming and re-drive in
`GatewayTurn.finish` stay as defence in depth.
### Caller contract
**Gemini**
- Send top-level `contents` as a list, with no `messages` or `input`.
- `systemInstruction`/`system_instruction`, `tools` and
`generationConfig` are optional and pass through.
**Converse**
- Send `messages` where every message has role `user` or `assistant`,
and every content block has no `type` key.
- At least one block must use a Converse key: `text`, `toolUse`,
`toolResult`, `cachePoint`, and so on.
- `system` is optional, as a list of blocks.
**Both shapes**
- `config.mode`: omit it (marker-free default), or use `lossless`,
`lossy_inline` or `lossless_then_lossy`. `ccr` is a 400.
- Only gateway turns get a view. Without a `gateway` block the endpoint
behaves exactly as before. A client-sent `_headroom_native_view` key is
dropped, and `BODY_CONTROL_KEYS` strips the real one before the body is
returned.
- `config.frozen_message_count` and `config.protect_recent` count native
messages: Gemini `contents` entries and Converse `messages`.
`session.frozen_message_count` comes back in native units. The
`/v1/usage` reply's `frozen_message_count` still counts view messages,
and the docs say so.
- No re-drive is ever armed, so `obligations` never contains `redrive`,
and request hooks run stream-safe-only. Tool compaction and output
steering are skipped.
- Relayed usage can be:
- Gemini `usageMetadata`, where `promptTokenCount` already includes
`cachedContentTokenCount`.
- Converse `usage`, where `inputTokens` excludes `cacheReadInputTokens`
and `cacheWriteInputTokens`.
- The whole provider response that carries either.
- Every gateway answer, fail-open ones included, carries `wire_shape`:
`gemini`, `converse`, `openai_responses` or `messages`. A caller that
sent a native body and gets anything else, or no field at all, is
talking to an older Headroom and should forward its original body.
- Top-level `messages` in the answer is the Converse list for Converse
and the chat-shaped view for Gemini.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): no new errors; the 8
reported are in `ccr/mcp_server.py` and `cache/prefix_tracker.py` and
are identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed (local replay, below)
New in `tests/gateway/test_native_shapes.py`:
- `test_ccr_mode_is_rejected_for_native_bodies[gemini|converse]`: 400,
`invalid_request`, message asserted.
- `test_other_modes_forward_native_bodies_without_ccr_markers`: 4 modes
x 2 shapes, on log output large enough to produce a marker. Checks no
hash, no marker, the shape echoed, non-message keys byte-identical and
the native list mapped back.
- `test_ccr_still_works_on_a_chat_gateway_turn`: the same log output in
a chat body with `ccr` still carries a marker. This proves the input
above is marker-producing, and that chat `ccr` didn't regress.
- `test_native_bodies_get_no_dedup_pointer_into_view_messages`.
- `test_native_turns_are_never_asked_to_redrive` now runs in the default
mode, because `ccr` is a 400.
### Test Output
```text
# before the fix (new tests only, rest of the file passing)
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[gemini]
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[converse]
FAILED tests/gateway/test_native_shapes.py::test_native_bodies_get_no_dedup_pointer_into_view_messages
3 failed, 25 passed
# after
tests/gateway/test_native_shapes.py: 28 passed
# tests/gateway tests/test_gateway_turn_unit.py tests/test_compress_api.py
# tests/test_proxy_compress_endpoint.py tests/test_compress_session_mode.py
# tests/test_compress_session_shared.py tests/test_compress_passthrough.py tests/test_compress_failure.py
1 failed, 507 passed, 4 skipped
# the one failure, test_lossy_inline_emits_no_ccr_markers[1-json_minifier], fails identically on main
|
||
|
|
9321a415a6 |
fix(memory): scope TrafficLearner's SQLite reads to its user_id (#4850)
## Description
`TrafficLearner` saves its rows under its configured `user_id`, but its
two direct SQLite reads ignored `user_id`:
- `_hydrate_persisted_state` (startup) loaded every user's
`traffic_learner` rows into the dedup maps, then applied `LIMIT
dedup_window`. Bob's learner could bind a pattern hash to Alice's row
and later bump *her* evidence instead of creating his own row. Other
users' rows also used up his window.
- `_load_persisted_patterns_from_sqlite` (used by
`_collect_all_patterns` when writing learned context files) returned
every user's patterns, so Alice's learned text ended up in Bob's files.
Both queries now filter on `user_id = ?`. In hydration the filter runs
before the `LIMIT`. Only matters when several user IDs share one memory
database; a single-user setup (`user_id="default"`) is unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/memory/traffic_learner.py`: `WHERE user_id = ?` in both
reads. `_load_persisted_patterns_from_sqlite(db_path, user_id)` takes
the user explicitly (private helper; its only caller is updated).
- `tests/test_memory/test_traffic_learner.py`: new
`TestUserScopedPersistence`, which checks that hydration leaves another
user's row alone and collection excludes other users. Hand-built test
schemas gain the `user_id` column that the real `memories` table has,
and direct helper calls pass `"default"`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# new tests against origin/main (
|
||
|
|
491afe98b1 |
fix(proxy): reject a private x-headroom-base-url on chat and responses with 400 (#4857)
## Description
When the upstream guard rejected an `x-headroom-base-url` override (an
address on a private, loopback, link-local or otherwise non-public
network, or a host outside `HEADROOM_ALLOWED_BASE_URLS` in allowlist
mode), `_resolve_openai_upstream_base` logged a warning and returned
`None`. `/v1/chat/completions` and `/v1/responses` then sent the request
— the prompt and the client's `Authorization`, meant for the gateway the
client named — to the configured `OPENAI_API_URL` instead. Every other
route that reads the header (`headroom/providers/proxy_routes.py`:
Anthropic messages, Gemini, models, embeddings, audio, passthrough)
refuses with `400 Rejected unsafe upstream base URL`.
The OpenAI chat and responses handlers now refuse the same way.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
## Changes Made
- `headroom/proxy/handlers/openai.py`: `_resolve_openai_upstream_base`
raises `HTTPException(400, "Rejected unsafe upstream base URL")` instead
of returning `None` when the guard rejects the override (malformed /
non-http(s) values are unchanged: still ignored).
- `docs/content/docs/security-model.mdx`: one behaviour for all routes
(the page previously documented the two different outcomes).
-
`tests/test_upstream_guard.py::test_openai_routes_refuse_a_private_override_instead_of_using_the_default[chat|responses]`:
real app, configured OpenAI target is a local recording server; asserts
400 and that the configured target is never contacted.
**Sibling PRs that touch the same code** (merge order note):
`fix/upstream-guard-off-loop` adds `_resolve_openai_upstream_base_async`
next to this function; whichever merges second must make the async
resolver raise the same 400 (that PR's rejection branch is the one place
to change). `fix/upstream-idna-pin-once` touches the call sites only.
Suggested order: idna-pin-once, off-loop, then this PR rebased.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# New test on origin/main (
|
||
|
|
2f58877732 |
fix(proxy): check x-headroom-base-url off the event loop in chat and responses (#4856)
## Description
`handle_openai_chat` and `handle_openai_responses` validated
`x-headroom-base-url` with the synchronous `is_safe_upstream_url`, which
waits for the DNS lookup (up to `HEADROOM_UPSTREAM_RESOLVE_TIMEOUT_S`, 3
s by default) on the event-loop thread. Chat did this three times per
request and responses twice, so one slow-resolving host held up every
other in-flight request for several seconds. The other routes in
`headroom/providers/proxy_routes.py` already use
`is_safe_upstream_url_async`.
Both handlers now resolve the header once, through a new
`_resolve_openai_upstream_base_async` that uses the async guard (lookup
on a worker thread, pin scope opened on the request's task), and reuse
that result for the destination and the extra-header decision.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement
## Changes Made
- `headroom/proxy/handlers/openai.py`: header parsing split into
`_openai_base_url_candidate`; sync `_resolve_openai_upstream_base` (kept
for `_resolve_openai_upstream`) and new
`_resolve_openai_upstream_base_async` share it;
`_openai_upstream_from(request, custom)` picks the upstream from an
already-resolved override; chat and responses call the async resolver
once.
- `tests/test_proxy/test_openai_upstream_guard_off_loop.py`: asserts the
guard runs once and not on the event-loop thread, for chat and
responses.
- `tests/test_proxy_openai_cache_key_integration.py`: its stub now
patches the async resolver the handler calls.
**Sibling PRs that touch the same code** (merge order note):
`fix/upstream-idna-pin-once` adds the identical `_openai_upstream_from`
and resolve-once hunks (this PR is a superset of that part);
`fix/openai-base-url-reject` changes the rejection branch of
`_resolve_openai_upstream_base`, which after this PR must be mirrored in
`_resolve_openai_upstream_base_async`. Suggested order: idna-pin-once,
then this PR, then reject (rebasing reject means raising the 400 from
the shared rejection path in both resolvers).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# New test on origin/main (
|