mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
fix/plugin-launcher-isolated-python
2438
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
6ffa22cfef | Merge branch 'main' into fix/plugin-launcher-isolated-python | ||
|
|
18067b1d19 |
fix(content-router): protect cache_control markers nested in tool_result content (#4898)
## Description
`ContentRouter` never rewrites a block that carries `cache_control`
(contract 1 in `_process_content_blocks`), but the guard only checks
`"cache_control" in block`. The Messages API also accepts the marker on
a `text` sub-block of a `tool_result`'s list content. That shape is not
seen as marked, so the `tool_result` is compressed and re-wrapped as
`[{"type": "text", "text": compressed}]`: the marker is dropped and the
bytes under it change, even for a tool result earlier in the
conversation. The cross-turn dedup `protected` check has the same gap.
`compress()` and `POST /v1/compress` return the router output as-is, so
callers get the rewritten block back.
This PR treats a `tool_result` whose list content carries a marker as
marked, the same shapes `count_cache_breakpoints` in
`headroom/proxy/helpers.py` already counts.
Fixes #4890
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/content_router.py`: new helper
`_carries_cache_control(block)`, true when the block has `cache_control`
or is a `tool_result` whose list content has a sub-block with
`cache_control`.
- Used in the per-block contract 1 guard of `_process_content_blocks`
and in the `protected` check of the cross-turn dedup. The contract 1
docstring now names the nested shape. The fresh-turn release
(`fresh_turn` and a `tool_result` in the final message) is unchanged.
- `tests/test_transforms_content_router.py`: the existing
`test_tool_result_cache_control_protected` is parametrized over
`block_marker` / `nested_marker`, plus
`test_marked_tool_result_in_history_survives_apply`, which goes through
`apply()` with the marked tool result in history.
- `tests/test_cross_turn_dedup.py`:
`test_dedup_never_rewrites_nested_cache_control_tool_result`, with an
unmarked control that must still be folded.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Source: one commit `916e8395` on `main` `5023c84e`. Runs below were
built from `git archive` of that commit in a `python:3.13-slim-bookworm`
image with `uv sync --frozen --extra proxy --extra dev`
(`headroom._core` built by maturin). ruff ran on the three touched
files, mypy on `headroom`. The new and changed tests were run 5 times in
a row. The module run covers the two touched test files plus
`test_content_router_detection_dedup.py`,
`test_responses_cross_turn_dedup.py` and
`test_openai_chat_dedup_recoverability.py`. For the reverse check,
`content_router.py` was replaced with the copy from `5023c84e` and the
tests kept.
### Test Output
```text
== RUFF CHECK RC 0
== RUFF FORMAT RC 0
== MYPY RC 0
# new and changed tests, run 1 and run 5 (runs 2-4 identical):
====================== 9 passed, 124 deselected in 0.97s =======================
====================== 9 passed, 124 deselected in 0.37s =======================
# related modules:
======================= 149 passed, 1 skipped in 12.32s ========================
# with content_router.py restored to
|
||
|
|
8da7e17e4e |
fix(kompress): honour toin_read_only in the Kompress CCR store (#4895)
## Description
Fixes #4801
The second TOIN write path of #4474. #4493 gated the router →
`SmartCrusher.crush()` path, but `store_kompress_in_ccr` still recorded
a TOIN observation unconditionally, so a Subscription request
(`CompressionPolicy.toin_read_only=True` — "never writes new pattern
observations from this request") that went through Kompress added an
observation to the shared TOIN store anyway. This change threads the
same per-call gate through the Kompress CCR store, both compressors, and
the router seam.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `store_kompress_in_ccr` gains a keyword-only `record_toin: bool =
True`; when `False` the CCR store write still happens (compressed output
stays retrievable, marker unchanged) but the
`get_toin().record_compression(...)` call is skipped.
- `KompressCompressor.compress()` / `compress_batch()` accept and
forward `record_toin` on every internal path (single, batched
reconstruction, sequential fallback); `apply()` derives it from a
`compression_policy` kwarg, the way `SmartCrusher.apply()` and
`CacheAligner` do.
- `RemoteKompressCompressor.compress()` accepts the same keyword
(keeping the drop-in parity contract) and forwards it to the shared
store function.
- `ContentRouter._try_ml_compressor` — the single seam both compressors
are called through — sends `record_toin=False` only when the request's
`_runtime_compression_policy.toin_read_only` is set, mirroring
`_invoke_smart_crusher`; the kwarg is omitted otherwise, so compressors
and test doubles without it keep working and default store calls keep
the exact pre-change 3-argument shape.
- Six regression tests in `tests/test_compression_policy_toin_gate.py`
covering the store function, local `compress()`, `compress_batch()`,
`apply()` under a Subscription policy, the remote client, and router
forwarding.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_compression_policy_toin_gate.py tests/test_remote_kompress_dropin.py tests/test_ccr_marker_policy.py tests/test_compression_policy.py
58 passed in the combined related run (gate file alone: 8 passed, 8 skipped — skips are the suite's own headroom._core-gated tests, no Rust wheel on this machine)
# Same five runnable new tests against unmodified main: 5 failed (they pass with this change)
$ ruff check headroom/transforms/kompress_compressor.py headroom/transforms/kompress_remote.py headroom/transforms/content_router.py tests/test_compression_policy_toin_gate.py
All checks passed!
$ ruff format --check (same four files)
4 files already formatted
# Issue repro, extended with the new gate (TOIN at a temp file):
store_kompress_in_ccr("word " * 400, "word " * 100, 400) -> TOIN total 0 -> 1 (reported bug; kept as the default for policy-less callers)
store_kompress_in_ccr("word " * 400, "word " * 100, 400, record_toin=False) -> TOIN total 1 -> 1, cache key still returned
store_kompress_in_ccr("word " * 400, "word " * 100, 400) -> TOIN total 1 -> 2
```
Type checking was not run locally (no mypy environment on this machine);
CI runs it.
## Real Behavior Proof
- Environment: Linux, Python 3.12, checkout of `main` at
|
||
|
|
f9627eb462 |
fix(compress): verify scalar types in lossless message comparisons (#4884)
## Description
Agent mode can report a message as lossless after a pipeline extension
changes a tool argument from JSON true to 1. Python considers those
values equal even inside nested dicts. The same issue affects integers
and floats.
## Type of Change
- [x] Bug fix
## Changes Made
- Verify aligned message structures before accepting equal-valued
scalars and require their types to match. Identical objects
short-circuit during traversal.
- Exercise changes both with and without simultaneous reversible string
compression. The existing token recount test now simulates a failed
string round trip at the leaf being verified.
## Testing
- [x] Ran the agent-mode, codec, compress API and canonical pipeline
suites. The regression test failed before the fix.
- [x] Ruff lint and format checks passed on changed files.
### Test Output
```text
72 passed, 7 skipped, 1 warning in 1.56s
```
## Real Behavior Proof
- Environment: macOS, Python 3.12.13, native extension from headroom-ai
0.44.0, offline.
- Exact command / steps: Run the following from the checkout with the
Python environment active.
- Observed result: The output changed from `True 0 int 1` to `True 1
bool True`. The changed tool call is now reverted.
- Not tested: Live providers or the full repository suite.
```sh
HEADROOM_OFFLINE=1 python - <<'PYTHON'
from dataclasses import replace
from headroom import compress
from headroom.hooks import CompressionHooks
from headroom.pipeline import PipelineStage
class NumericHook(CompressionHooks):
def on_pipeline_event(self, event):
if event.stage == PipelineStage.INPUT_COMPRESSED:
msg = dict(event.messages[0])
msg["content"] = [dict(msg["content"][0], input={"enabled": 1})]
return replace(event, messages=[msg])
return None
messages = [{"role": "assistant", "content": [
{"type": "tool_use", "id": "call", "name": "run", "input": {"enabled": True}}
]}]
r = compress(messages, mode="agent", hooks=NumericHook())
v = r.messages[0]["content"][0]["input"]["enabled"]
print(r.lossless, r.reverted_messages, type(v).__name__, v)
PYTHON
```
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Lossless verification reverts
messages with scalar type changes.
- Kill switch / disable path: Existing verify_lossless option outside
agent mode.
- Unsafe override required: No.
- Qualification impact: Adds regression coverage for nested booleans,
integers and floats.
- Rollback path: Revert this patch.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
303f521812 |
fix(smart-crusher): keep container-shaped strings as strings in lossless tables (#4875)
## Description
A string cell in a `json` column whose text is a JSON **container**
(`"[]"`, `"{}"`, `"[1,2]"`) is
classified as `StringifiedJson` and un-escaped into
`CellValue::Scalar(parsed)`, so the cell renders
byte-identically to the real value it spells. The csv-schema decoder
cannot tell the two apart
afterwards: `expand_compacted` returns `[]` where the original had
`"[]"`, and with
`verify_lossless=True` (also forced by `mode="agent"`) the round-trip
check fails and the whole
message is reverted, silently dropping its compression.
This is the container-shaped remainder of #4053, which fixed the
scalar-looking case (`"null"`,
`"true"`, `"1"`). The formatter already knows how to render these
strings unambiguously
(`csv_formatter_json_column_quotes_strings_that_parse_as_json` covers
`"[1]"` and `'{"a":1}'`), but
`compact()` never lets them reach it as strings.
Closes #4874
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `compactor::cell_from_value`: for `StringifiedJson` that is not an
array of objects, keep the
original string (`CellValue::Scalar(v.clone())`) instead of the parsed
value. The formatter's
existing ambiguous-text rule JSON-quotes it, so the decoder returns the
string.
- Nested behaviour is unchanged: a stringified **array of objects**
still becomes a nested table, so
`test_nested_table_markers_resolve_to_source_bytes` and its CCR markers
keep working.
- Rust regression test pinning the exact wire cells (`"[]"` for the
array, `"""[1]"""`-style for the
string).
- Python regression test at the caller boundary (`_crush` +
`expand_compacted`).
## Testing
- [x] Unit tests pass (`cargo test -p headroom-core --lib` -> 1044
passed, 1 ignored)
- [x] `cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo test -p headroom-core --lib compaction::
test result: ok. 74 passed; 0 failed; 0 ignored
$ cargo test -p headroom-core --lib
test result: ok. 1044 passed; 0 failed; 1 ignored
# with the fix reverted, the new Rust test fails:
left: "\"[]\""
right: "\"\"\"[]\"\"\""
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, Rust 1.100
`x86_64-pc-windows-gnu`, repo `main` at
`fda75de5`
- Exact command / steps: `cargo test -p headroom-core --lib
compaction::compactor::tests::stringified_container_strings_keep_their_identity`;
the test builds
`[{id:0,v:[]},{id:1,v:"[]"},{id:2,v:[1,2]},{id:3,v:"[1,2]"},{id:4,v:{}},{id:5,v:"{}"}]`,
compacts
it, and asserts the rendered `v` cells.
- Observed result: without the change the array and the string render
the same cell (`"[]"`); with
it the string renders as the JSON string literal (`"""[]"""`), which
`expand_compacted` maps back to `"[]"`.
- Not tested: the Python regression test could not be executed locally
because this machine cannot
build the native extension (Rust build stopped in `icu_segmenter` under
the GNU toolchain); it is
included for CI, where the wheel is built normally.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; the change is in the lossless table
codec only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: a stringified JSON container in a
table cell now stays a string
in the densified text (it was displayed as the parsed value). This is
required for the documented
reversibility of `csv-schema`; the model sees a JSON string literal
instead of nested JSON.
- Kill switch / disable path: n/a (lossless path)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the single `cell_from_value` branch.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4875"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
c8168a0aa0 |
fix(cache): invalidate cached CCR markers after store eviction (#4872)
## Description When `ContentRouter` caches a compressed output that carries a CCR retrieval marker (`Retrieve more: hash=...`, `Retrieve original: hash=...`, or `<<ccr:...>>`), a subsequent `compress()` / `ContentRouter.apply()` call could hit Tier 2 `CompressionCache` and return that marker even after the underlying `CompressionStore` entry was evicted by `max_entries`, cleared, or expired. Any follow-up retrieval for that marker then returned `None`. This change: 1. Adds `CompressionCache.invalidate(key)` to evict a stale cached entry without marking it skipped, and tracks verified CCR marker hashes per cached entry (`_marker_hashes`) cleaned up alongside `_results`. 2. Records only marker hashes that actually existed in `CompressionStore` at `put()` time via `ContentRouter._collect_verified_marker_hashes`, preserving compatibility with synthetic/mocked router tests that do not write to `CompressionStore`. 3. Validates tracked marker hashes via `ContentRouter._cached_markers_retrievable` on both Tier 2 cache hit paths (`ContentRouter.apply` and `ContentRouter._compress_block_content`), invalidating stale entries and falling through to fresh compression so the original content is re-stored and retrievable. Closes #4796 *Disclosure: This fix and PR description were written by an AI agent (沈寻 / Shen Xun, `@xunWhale`) on behalf of `@e0_7`, and verified locally with pre-fix failure / post-fix pass pytest outputs, repo-wide ruff/mypy, and the issue reproduction.* ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/content_router.py`: added `_CCR_MARKER_HASH_RE`, `CompressionCache.invalidate()`, `CompressionCache.get_marker_hashes()`, `_marker_hashes` tracking inside `CompressionCache`, and `ContentRouter._collect_verified_marker_hashes()` / `_cached_markers_retrievable()` checks on both string and content-block Tier 2 cache hit paths. - `tests/test_router_result_cache.py`: added 4 regression tests covering `CompressionCache.invalidate()`, capacity eviction under `compress()` for both `role="tool"` string messages and Anthropic `tool_result` content blocks, and custom `CompressionStore.clear()` invalidation. ## Testing - [x] Unit tests pass (`pytest` on router/cache/compression suites; see scope note below) - [x] Linting passes (`ruff check .` & `ruff format --check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output Pre-fix baseline (`0368a87`, without `content_router.py` fix): ```text $ PYTHONPATH=. pytest tests/test_router_result_cache.py -k "capacity_eviction or store_cleared or invalidate_drops" -v FAILED tests/test_router_result_cache.py::test_invalidate_drops_cached_result_without_marking_skipped - AttributeError: 'CompressionCache' object has no attribute 'invalidate' FAILED tests/test_router_result_cache.py::test_compress_repopulates_ccr_store_after_capacity_eviction - assert None is not None FAILED tests/test_router_result_cache.py::test_compress_block_content_repopulates_ccr_store_after_capacity_eviction - assert None is not None FAILED tests/test_router_result_cache.py::test_router_invalidates_cached_marker_when_store_cleared - TypeError: CompressionCache.put() got an unexpected keyword argument 'marker_hashes' ======================= 4 failed, 28 deselected in 0.39s ======================= ``` Post-fix checks: ```text $ ruff check . All checks passed! $ ruff format --check . 1940 files already formatted $ mypy headroom Success: no issues found in 583 source files $ PYTHONPATH=. pytest tests/test_router_result_cache.py -v ... tests/test_router_result_cache.py::test_invalidate_drops_cached_result_without_marking_skipped PASSED [ 90%] tests/test_router_result_cache.py::test_compress_repopulates_ccr_store_after_capacity_eviction PASSED [ 93%] tests/test_router_result_cache.py::test_compress_block_content_repopulates_ccr_store_after_capacity_eviction PASSED [ 96%] tests/test_router_result_cache.py::test_router_invalidates_cached_marker_when_store_cleared PASSED [100%] ============================== 32 passed in 1.93s ============================== $ PYTHONPATH=. pytest tests/test_router_result_cache.py tests/test_router_cache_policy.py tests/test_transforms_content_router.py tests/test_compression_store.py tests/test_compression_safety_rails.py tests/test_compression_determinism.py tests/test_compression_decision.py tests/test_compression_units.py tests/test_compression_batches.py tests/test_compression_cache.py tests/test_ccr_batch_store.py -q ======================= 423 passed, 2 skipped in 23.02s ======================== ``` *(Scope note: The full repository test suite was not run locally because optional extras such as `[proxy]`/`fastapi` are not installed in this environment; `ruff check .`, `ruff format --check .`, and `mypy headroom` ran across the entire repository.)* ## Real behavior proof - Environment: Debian Linux (Python 3.11.2), Headroom `main` (`0368a87`) with `headroom._core` from `headroom-ai==0.44.0`, `InMemoryBackend(max_entries=2)`, `kompress_model="disabled"`. - Exact command / steps: Ran the issue #4796 reproduction (`compress` on 3 distinct 251-line tool log payloads so entry 0 is capacity-evicted from `CompressionStore`, followed by a repeat `compress(originals[0])` and `store.retrieve(repeat_key)`). - Observed result: - Before patch: `before_repeat_retrieve: None`, `repeat_key == keys[0]: True`, `store.retrieve(repeat_key): None` - After patch: `before_repeat_retrieve: None`, `after_repeat_key_match: True retrieved_present: True exact_match: True` (`store.retrieve(repeat_key).original_content` equals `originals[0][-1]["content"]`). - Not tested: Distributed Redis `CompressionStore` backend, `[proxy]` extra tests (`fastapi` not installed), or live upstream LLM API calls; `kompress_model="disabled"` was used as in the issue reproduction. ## Runtime Rollout Safety - Rollout-managed feature(s): None (bug fix in `ContentRouter` Tier 2 cache hit validation). - Minimum rollout channel: N/A - Stable/default behavior changed: Only invalidates cached CCR marker hits when their store-backed CCR entry is no longer present in `CompressionStore`. - Kill switch / disable path: Standard `HEADROOM_COMPRESSION_CACHE` disable / `invalidate`. - Unsafe override required: No. - Qualification impact: Prevents unrecoverable CCR markers on repeat compression after store capacity eviction. - Rollback path: Revert this single commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A — internal cache bug fix, no public API or doc change) - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4872" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
f4e3ad4b11 |
fix(content-router): keep list-form tool_result text blocks on separate lines (#4739)
## Description
A `tool_result` whose `content` is a list of several `text` blocks gets
flattened with `"".join(...)` in `ContentRouter` before the
read-protection check and before compression. MCP tools that return more
than one `TextContent` and litellm-style clients send this shape. When a
block does not end with a newline, the last line of one block and the
first line of the next one become a single line. The compressed result
is written back as one block, so the model sees a line the tool never
produced.
This PR joins the blocks with a newline unless the block already ends
with one, at both flatten sites.
Fixes #4729
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/content_router.py`: new helper
`_join_tool_result_text_blocks()` that joins list-form `text` blocks
with `\n`, skipping the separator when a block already ends with a
newline. Empty blocks add no extra line.
- Used at both places that flattened with `""`: the read-protection
check and the compression path. The wire shape does not change (the
existing `[{"type": "text", ...}]` re-wrap is kept), and results that
are not compressed keep their original blocks.
- `tests/test_transforms/test_content_router.py`: one `router.apply`
test with a two-block log `tool_result` that asserts that the output was
compressed and that every log line in it is a line the tool actually
returned, plus four parametrized cases for the helper.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Exact source: head `5305d8acb5f505e635d1287cae30ed7f8d9eb595`, base
`c05fbdddc873078dd97cb9ede841e75ef7ed0bef`. The PR has three commits:
`f273fbb9` (the fix, written on `1da8930a`) and two merges of `main`
(`2cba6c5b`, `5305d8ac`), both without conflicts. `git diff
c05fbddd...5305d8ac` still touches only the two files listed above.
Upstream CI on `5305d8ac`, all `completed` / `success`: CI
[43338122484](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122484),
Merge Conflicts
[43338122336](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122336),
Changelog Guard
[43338122370](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122370),
rust
[43338122318](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122318),
Evaluation Suite
[43338122592](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122592),
Security
[43338122460](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122460).
Local runs below were built from `git archive 5305d8ac` in a
`python:3.13-slim-bookworm` image with `uv sync --frozen --extra proxy
--extra dev`. ruff ran on the two touched files and mypy on
`content_router.py`. pytest ran on the content router test modules. For
the reverse check, `content_router.py` was replaced with the copy from
`c05fbddd` and the tests kept.
A wider run over the 122 test files that touch the content router gave 9
failed / 2669 passed on `5305d8ac` and 9 failed / 2664 passed on
`c05fbddd`. The 9 failures are the same set on both (memory_bridge,
offline_egress docs, tabular; they come from the test image, not this
change), and the extra 5 passes are the new tests.
### Test Output
```text
== RUFF CHECK RC 0
== RUFF FORMAT RC 0
== MYPY RC 0
# new tests, 5 consecutive runs, all identical:
======================= 5 passed, 66 deselected in 5.01s =======================
# test_content_router.py + test_content_router_ccr_retrieve_exemption.py + test_content_router_shell_small_results.py:
============================= 125 passed in 5.61s ==============================
# with content_router.py restored to
|
||
|
|
3aafdf2dcc |
fix(router): keep source code out of the prose (Kompress) fallbacks (#4787)
## Description When the code compressor declined a block of source code, the router passed the block to Kompress. Kompress is a prose model. It drops low-salience words, and in code those words are keywords such as `for`, `in`, `if` and `import`. The result no longer parses. #4214 shows a fenced Python script that comes back with `for` and `in` removed. This PR stops that in the Python router and in the Rust live-zone dispatcher. When the code compressor declines, the code is forwarded unchanged. Closes #4214 Credit: the Python change is oxura's work from https://github.com/oxura/headroom/tree/fix/fenced-code-routing-4214-20261018 (277ca1f). Their commit is the first commit here, rebased onto `main`, and oxura is its author. The test, docs and Rust commits were added on top. oxura asked us to open this PR from our fork ([comment](https://github.com/headroomlabs-ai/headroom/issues/4214#issuecomment-6797572655)), and said they have not run this branch, so the model and native results below are ours, not theirs. ## Behaviour change to review With `enable_code_aware=False` (the default), source code now stays unchanged. Before this PR, it was sent to Kompress and lost keywords. With `enable_code_aware=True`, code that the code compressor declines also stays unchanged. Before, Kompress ran on it. A code block the code compressor does compress is unaffected. Prose, logs, JSON and tables are unaffected. Side effect: a code block that Kompress used to shrink (and break) now keeps its full size. This is the intended trade, but it means fewer tokens saved on code when `enable_code_aware=False`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Documentation update - [ ] New feature - [ ] Breaking change (the default-config behaviour change above is intended, and it only stops damage to code) ## Changes Made Five commits: - `596764c3` `fix(router): keep source code out of prose fallbacks`. Author: oxura. Removes the CODE_AWARE to Kompress fallbacks in `headroom/transforms/content_router.py`. This is 277ca1f rebased onto `main`, and it differs from 277ca1f in two ways only: the two test-file conflicts are resolved, and its `wiki/proxy.md` hunk is dropped, because `AGENTS.md` says not to add content under `wiki/`. - `7705a4e6` `test(router): repin the code-to-Kompress tests and add the #4214 fence regression`. Author: Tao Hansen, with a `Co-authored-by: oxura` trailer. Five tests asserted the old fallback. They are rewritten to assert the new behaviour (code unchanged, Kompress never sees it) instead of being deleted. Adds a regression test for the script from #4214, with `enable_code_aware` both off and on. Updates `docs/content/docs/code-compression.mdx` and `docs/content/docs/how-compression-works.mdx`, which said that detected code goes to Kompress. - `6631668a` `fix(live-zone): leave source code unchanged when the code compressor declines`. Author: Tao Hansen. Same fix in `crates/headroom-core/src/transforms/live_zone.rs`: a declined `SourceCode` block is now a NoOp, not a Kompress input. Tests: `tests/live_zone_source_code_decline.rs` (single test, needs no model), and new cases in `tests/live_zone_dispatch.rs`. - `d8776283` `fix(router): turn off the code compressor's internal Kompress fallback`. Author: Tao Hansen. Follows review: the router's code compressor was still built with `fallback_to_kompress=self.config.enable_kompress` (#4719), so unknown languages, Perl and AST failures could build a `KompressCompressor` inside `CodeAwareCompressor`. The router now always sets `fallback_to_kompress=False`. The #4719 test that pinned `True` (`test_enabled_kompress_keeps_code_aware_fallback`) is repinned as `test_enabled_kompress_still_turns_off_code_aware_fallback`, and the unknown-language, Perl (by language hint, since `detect_language()` reports Perl-dominant code as unknown) and AST-failure paths are tested with spies on `is_kompress_available` and `KompressCompressor`, with `enable_kompress` both off and on. `docs/content/docs/code-compression.mdx` notes that the router always turns the option off. - `c1f8464e` `fix(router): keep passthrough code from external compressors and stale caches`. Author: Tao Hansen. Follows review: `_try_external_compressor` skips `PASSTHROUGH` blocks (code with `prefer_code_aware_for_code=False`, whole or as a mixed section), which would otherwise be offered to a prose-only external compressor as `text/plain`. The disk cache namespace now also hashes `_ROUTING_POLICY_REVISION`, so a same-version restart does not replay code that Kompress damaged before this change. Tests: two external-dispatch cases with a `text/plain` external, and one namespace test. All three fail on the previous head. I did not touch `CHANGELOG.md`. ## Testing - [x] Unit tests pass (`pytest`): see below, the failures are environment failures that also fail on `main` - [x] Linting passes (`ruff check .`, `ruff format --check .`) - [x] Type checking passes (`mypy headroom/transforms/content_router.py`) - [x] New tests added for new behaviour - [x] Manual testing performed (real model replay, below) ### Test Output macOS, Python 3.14. Head is now `c1f8464e`. ```text After the second review fix ( |
||
|
|
cf3284675e |
fix(lossless): never grep-fold Go, nginx, klog or access-log timestamps (#4901)
## Description The lossless search fold splits each line as `path:line:content` and factors a repeated `path` into a header. `_TIMESTAMP_ROW_RE` (#3419) keeps timestamped log lines out of it, because a log line has the same colon shape: folding it hoists the date and hour into a fake file header and every row starts mid-clock. The fold round-trips byte-exactly, so the inverse check in `compact_lossless` cannot catch it. The guard knew ISO, `MM/DD/YYYY`, syslog and bare-clock prefixes, but not these four common formats: | Format | Line | Split as | |---|---|---| | Go `log` package | `2026/09/02 14:30:00 ERROR ...` | `2026/09/02 14` / `30` / `00 ERROR ...` | | nginx error log | `2026/09/02 14:30:00 [error] ...` | same | | klog (Kubernetes components) | `E0902 14:30:00.000000 1 reflector.go:100] ...` | `E0902 14` / `30` / `00.000000 ...` | | common/combined access log | `10.0.0.7 - - [02/Sep/2026:14:30:00 +0000] ...` | `10.0.0.7 - - [02/Sep/2026` / `14` / `30:00 ...` | They reach the fold two ways: the bash-search fold (`grep ERROR api.log`) and the router's lossless-first stage, which tries the search fold on any tool output (`cat`, `tail`, `kubectl logs`). This adds those three prefix shapes to `_TIMESTAMP_ROW_RE`, the same place #3419 put the guard. The content detector (#3748) and the mixed-content splitter (#4463) share the regex; neither claimed these lines before, so for them only false positives could change, and there are none on real grep output (below). The access-log alternative differs from the one I suggested in #4900. That one (`\S+ \S+ \S+ \[...`) also matched a real `grep -n` row over an access log (`/var/log/nginx/access.log:12:10.0.0.7 - - [02/Sep/...`), because `\S+` swallows `path:12:10.0.0.7`. The fields now exclude `:`, which a mis-split line cannot contain before its first colon anyway. The host field takes name/IPv4 characters only (`[\w.-]`), capped at 255 (a DNS name is at most 253). With `[^\s:]` the check scanned every grep row's whole path, and fold + detection on 10k grep rows took 10% and 32% longer than main in two interleaved runs; with this class and cap it stops early on a path and on a long unbroken line such as a base64 blob. Closes #4900 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/lossless_compaction.py`: three alternatives in `_TIMESTAMP_ROW_RE`: `YYYY/MM/DD hh:mm` (Go log, nginx error log), `[IWEF]MMDD hh:mm` (klog/glog), and the access-log prefix `host ident user [dd/Mon/yyyy:hh:mm` with a `[\w.-]{1,255}` host. - `tests/test_lossless_timestamp_guard.py`: the four formats, plus an access log with a 253-character host, join `test_timestamped_logs_are_never_folded`. New `test_grep_rows_over_those_logs_still_fold`: real `grep -n` rows over the same logs (including a dated `2026/09/02/api.log` path) still fold under their file header. - `tests/test_bash_search_lossless_fold.py`: `test_go_log_keeps_its_timestamps` runs a Go log through `ContentRouter` as the output of `grep ERROR /var/log/api.log` (bash-search fold) and of `cat /var/log/api.log` (lossless-first stage). ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # this branch's tests against main's lossless_compaction.py (main |
||
|
|
0d203db6b8 |
fix(copilot): keep new VS Code chats on the Local harness so they reach the proxy (#4899)
## Description
VS Code can run a Copilot chat on its "Copilot harness" (Agent Host, the
bundled Copilot SDK runtime) instead of the Local harness. That runtime
ignores `github.copilot.advanced.debug.overrideCapiUrl` /
`overrideProxyUrl`, so the chat goes straight to GitHub. Copilot's small
background calls (title, intent) still reach the proxy, so the proxy
looks connected while the main conversation never arrives. The harness
default (`chat.defaultToCopilotHarness`,
`chat.editor.preferCopilotHarness`) is tagged `experiment: startup` in
VS Code, so it can be flipped per user by rollout.
`headroom wrap vscode` now pins new chats to the Local harness.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/copilot/vscode.py`: the managed block also writes
`"chat.defaultToCopilotHarness": false` and
`"chat.editor.preferCopilotHarness": false`. Because they sit inside the
marker block, `headroom unwrap vscode` removes them. A harness setting
the user already set outside the block is left alone (no duplicate key).
New helper `copilot_harness_settings_enabled(path)` reports a user value
of `true`.
- `headroom/cli/wrap.py`: one line saying new chats stay on the Local
harness and existing chats keep their engine; a warning instead when the
user's own setting is `true`. `--no-configure` prints the two extra
settings.
- `docs/content/docs/vscode-copilot.mdx`: documents the harness settings
and the troubleshooting case.
- Blocks written by older versions (including ones with
`overrideAuthType`) are replaced by marker as before.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
tests/test_provider_copilot_vscode_config.py + tests/test_cli/test_wrap_vscode.py: 32 passed.
Against origin/main source: 7 of those fail (block contents, old-block upgrade, user-set key [True]/[False], wrap output, --no-configure output).
ruff 0.17.4 check + format --check on the 5 changed Python files: clean.
mypy on the changed source files: 8 errors, all in headroom/cache/prefix_tracker.py and headroom/ccr/mcp_server.py, identical on origin/main; none in changed files.
```
## Real Behavior Proof
- Environment: macOS 27.2.1, Python 3.12.13, VS Code 1.145.2 with GitHub
Copilot signed in, core run from this branch, real GitHub Copilot
upstream. Real user `settings.json` (backed up and restored afterwards).
- Exact command / steps:
1. `headroom wrap vscode -p 8795` from this branch.
2. `code chat -n -m agent "Reply with the single word OK and nothing
else."` (opens a new VS Code window and sends one Agent-mode chat).
3. Read the proxy's `/stats`, VS Code's `agenthost.log`, then `headroom
unwrap vscode`.
- Observed result:
```text
VS Code Copilot proxy settings updated: ~/Library/Application
Support/Code/User/settings.json
Keep using Copilot's normal model picker; the selected model is
preserved.
New chats stay on VS Code's Local harness; a chat keeps the engine it
started with, so start a new chat.
settings.json after wrap:
{
// --- Headroom Copilot proxy ---
"github.copilot.advanced.debug.overrideProxyUrl":
"http://127.0.0.1:8795/p/win-local",
"github.copilot.advanced.debug.overrideCapiUrl":
"http://127.0.0.1:8795/p/win-local",
"chat.defaultToCopilotHarness": false,
"chat.editor.preferCopilotHarness": false
// --- end Headroom Copilot proxy ---
}
proxy requests before chat: 0 {}
proxy requests after one Agent-mode chat in a new VS Code window: 4
{'gpt-4o-mini-2024-07-18': 3, 'claude-opus-5.5': 1}
agent host turns during the chat: 0
Removed Headroom Copilot proxy settings from ~/Library/Application
Support/Code/User/settings.json
settings.json after unwrap:
{
}
```
Separately, on the same machine with the settings edited by hand (same
VS Code, same proxy, new window each time):
`"chat.defaultToCopilotHarness": true` -> the Agent-mode chat ran in the
Agent Host (provider `copilotcli`) and 0 requests reached the proxy;
`"chat.defaultToCopilotHarness": false` +
`"chat.editor.preferCopilotHarness": false` -> Local harness, 4 requests
reached the proxy including the main `/v1/messages` call.
`"chat.temporaryDefaultToLocalHarness": true` as a user setting had no
effect, so it is not written.
- Not tested: a VS Code where Microsoft's rollout has flipped the
default (the explicit user value is expected to win, but that could not
be reproduced); an organization that enforces managed sandbox settings,
where VS Code can require the Copilot harness regardless; VS Code
profiles, workspace-level settings, Insiders/VSCodium; Windows and
Linux; the persistent `headroom start --persistent --vscode` path in the
Enterprise client.
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: `headroom wrap vscode` writes two
more VS Code user settings inside its managed block. No proxy behaviour
changes.
- Kill switch / disable path: `headroom unwrap vscode` removes them; a
user who sets either key themselves keeps their value.
- Unsafe override required: no.
- Qualification impact: none expected.
- Rollback path: revert this commit; `unwrap` removes blocks of either
format.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- This keeps users off the Copilot harness; it does not route that
harness through Headroom. Supporting it is separate work.
- Only the configured settings file is checked for a user value. A
`true` in workspace settings, another profile, or policy is not
detected.
- After `unwrap` the keys return to VS Code's default, which a rollout
may move to the Copilot harness.
- Existing chats keep the harness they started with; the wrap output
says to start a new chat.
- `e2e/copilot_live.py` and `e2e/wrap/run.py` were not run.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4899"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
b560cba927 |
fix(memory): parse frontmatter delimiters on complete lines (#4886)
## Description A memory beginning with `Use foo---bar as the separator` writes that text into the generated name/description. The reader takes the embedded `---` as the closing delimiter, mixes frontmatter into the body and changes the content hash. The next sync imports a polluted duplicate. Recognize opening and closing frontmatter delimiters only on complete lines, including LF and CRLF input. Preserve the original text if the header is malformed or has no closing delimiter. Related to #4832's full-store sync work; this fixes a separate contract on the current main snapshot. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Require complete delimiter lines; preserve bodies containing embedded dashes and malformed headers. - Add regressions against the unmodified base `5023c84e527e28281dd73d1ba04184fe738aac0f` and real file/component coverage. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook with httpx installed) - [x] New tests added - [ ] Manual testing performed by the human author ### Test Output 7 new regressions; sync, CLI index and supersession suites: 77 passed. The existing 35 sync cases are included. Existing datetime deprecation warnings remain. Original pre-commit and commit-msg hooks passed. Original full pre-push `make ci-precheck` passed; native extension rebuilt/import-verified. Full gate summaries: ```text cargo fmt / clippy --workspace: passed Rust workspace: 1755 passed, 3 ignored Native extension: rebuilt and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing development environment; base `5023c84e527e28281dd73d1ba04184fe738aac0f`, patch `1fda45a711b4ce38848e8615148c74dc9c1bd6ab`. Real LocalBackend factory, SQLite/FTS/vector store, agent files and Ollama HTTP adapter; the loopback service returns controlled 384D vectors. - Exact command / steps: save the following script as `proof.py`. Run `python proof.py frontmatter <fresh-output-directory> before` from the base checkout, then `python proof.py frontmatter <another-fresh-directory> after` from the patch. The identical script ran outside pytest. - Observed result: first_exported=1 on both; round_trip_equal: main=false, patch=true; repeat_imported/repeat_exported: main=1/1, patch=0/0; primary_count: main=2, patch=1. The patch uses one actual embedding HTTP request; main uses a second request to embed the polluted duplicate. - Not tested: trained-model inference or answer quality, paid/provider APIs, non-local backends, production workloads, other operating systems or the entire Python suite. Local focused, broader and original-gate results are reported separately. <details> <summary>Standalone runtime reproduction</summary> ```python """Real LocalBackend/Ollama HTTP/SQLite/agent-file before-after replay.""" import asyncio import json import os from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path import sys import threading from headroom.memory.backends.local import LocalBackend, LocalBackendConfig from headroom.memory.sync import sync from headroom.memory.sync_adapters.codex_agent import CodexAdapter from headroom.memory.sync_adapters.claude_code import ClaudeCodeAdapter case, folder, expected = sys.argv[1:] out = Path(folder).resolve() out.mkdir(parents=True, exist_ok=False) os.environ.update(HEADROOM_STATELESS="0", HEADROOM_WORKSPACE_DIR=str(out/'workspace'), HEADROOM_OFFLINE="0") class Endpoint(BaseHTTPRequestHandler): requests = [] def log_message(self, *_): pass def do_POST(self): payload=json.loads(self.rfile.read(int(self.headers['content-length']))) self.requests.append({'path':self.path,'body':payload}) assert self.path=='/api/embeddings' body=json.dumps({'embedding':[1.0]+[0.0]*383}).encode() self.send_response(200) self.send_header('Content-Type','application/json') self.send_header('Content-Length',str(len(body))) self.end_headers() self.wfile.write(body) server=ThreadingHTTPServer(('127.0.0.1',0),Endpoint) threading.Thread(target=server.serve_forever,daemon=True).start() async def run(): backend=LocalBackend(LocalBackendConfig(db_path=str(out/'memory.db'),embedder_backend='ollama',embedder_model='controlled-vector',vector_dimension=384,ollama_base_url=f'http://127.0.0.1:{server.server_port}')) state=out/'sync-state.json' try: if case=='fingerprint': saved=await backend.save_memory('Original orchard fact',user_id='alice') adapter=CodexAdapter(out/'AGENTS.md') first=await sync(backend,adapter,'alice',state) unchanged=await sync(backend,adapter,'alice',state) await backend._hierarchical_memory.update(saved.id,content='Corrected orchard fact') result=await sync(backend,adapter,'alice',state) text=adapter._path.read_text(encoding='utf-8') data={'first_exported':first.exported,'unchanged_exported':unchanged.exported,'edited_exported':result.exported,'corrected_in_file':'Corrected orchard fact' in text,'edited_imported':result.imported} assert data['first_exported']==1 and data['unchanged_exported']==0 assert data['corrected_in_file']==(expected=='after'),data assert result.exported==(1 if expected=='after' else 0),data else: content='Use foo---bar as the separator\nKeep the second line intact.' saved=await backend.save_memory(content,user_id='alice') adapter=ClaudeCodeAdapter(out/'claude-memory') first=await sync(backend,adapter,'alice',state,force=True) readback=await adapter.read_memories() repeated=await sync(backend,adapter,'alice',state,force=True) rows=await backend.get_user_memories('alice') data={'first_exported':first.exported,'round_trip_equal':readback[0].content==content,'reread_content':readback[0].content,'repeat_imported':repeated.imported,'repeat_exported':repeated.exported,'primary_count':len(rows)} assert data['round_trip_equal']==(expected=='after'),data assert data['repeat_imported']==(0 if expected=='after' else 1),data data['requests']=Endpoint.requests data['source_file']=__import__('headroom').__file__ (out/'result.json').write_text(json.dumps(data,indent=2),encoding='utf-8',newline='\n') print(json.dumps(data,ensure_ascii=False)) finally: await backend.close() try: asyncio.run(run()) finally: server.shutdown();server.server_close() ``` </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none; existing memory sync. - Minimum rollout channel: existing memory sync users. - Stable/default behavior changed: metadata containing triple dashes remains separate from the memory body. - Kill switch / disable path: stop invoking memory sync; no new toggle. - Unsafe override required: no. - Qualification impact: memory synchronization correctness; no dependency, native or workflow change. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented the non-obvious behavior - [x] I have added regression tests - [x] I did not edit CHANGELOG.md ## Additional Notes Implementation and local test execution were completed with Codex assistance. The author personally reviewed the two patches and the before/after evidence; this does not claim the author personally ran the tests. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4886" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
00b38736d9 |
fix(memory): detect content changes in incremental sync (#4885)
## Description Incremental sync hashes only the row count and the first eight characters of the newest five IDs. An in-place content edit keeps those values unchanged, so sync returns early and never exports the corrected fact. Hash the content, complete IDs and fields used for export/lineage across the already-fetched snapshot. Unchanged snapshots still take the no-op path. The persisted fingerprint is opaque; existing states receive one normal sync after this change. Related to #4832's full-store sync work; this fixes a separate contract on the current main snapshot. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Include full IDs, content and export/lineage fields from every returned row; exclude volatile access counters. - Add regressions against the unmodified base `5023c84e527e28281dd73d1ba04184fe738aac0f` and real file/component coverage. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook with httpx installed) - [x] New tests added - [ ] Manual testing performed by the human author ### Test Output 6 new regressions; focused sync suite: 41 passed. Broader memory run: 675 passed, 82 skipped, 8 failed. The same 8 failures reproduce on the exact unmodified base (275 passed in the two affected files): six Windows SQLite temporary-directory cleanup failures and two POSIX-path assumptions. No old fixture changed. Original pre-commit and commit-msg hooks passed. Original full pre-push `make ci-precheck` passed; native extension rebuilt/import-verified. Full gate summaries: ```text cargo fmt / clippy --workspace: passed Rust workspace: 1755 passed, 3 ignored Native extension: rebuilt and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing development environment; base `5023c84e527e28281dd73d1ba04184fe738aac0f`, patch `c4ffa08665c26491b2379af0f4d953ceb203ee6b`. Real LocalBackend factory, SQLite/FTS/vector store, agent files and Ollama HTTP adapter; the loopback service returns controlled 384D vectors. - Exact command / steps: save the following script as `proof.py`. Run `python proof.py fingerprint <fresh-output-directory> before` from the base checkout, then `python proof.py fingerprint <another-fresh-directory> after` from the patch. The identical script ran outside pytest. - Observed result: first_exported=1; unchanged_exported=0; edited_exported: main=0, patch=1; corrected_in_file: main=false, patch=true. The existing additive bidirectional semantics also reimport the old file fact (edited_imported=1 on the patch); this change does not implement deletion propagation or conflict resolution. - Not tested: trained-model inference or answer quality, paid/provider APIs, non-local backends, production workloads, other operating systems or the entire Python suite. Local focused, broader and original-gate results are reported separately. <details> <summary>Standalone runtime reproduction</summary> ```python """Real LocalBackend/Ollama HTTP/SQLite/agent-file before-after replay.""" import asyncio import json import os from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path import sys import threading from headroom.memory.backends.local import LocalBackend, LocalBackendConfig from headroom.memory.sync import sync from headroom.memory.sync_adapters.codex_agent import CodexAdapter from headroom.memory.sync_adapters.claude_code import ClaudeCodeAdapter case, folder, expected = sys.argv[1:] out = Path(folder).resolve() out.mkdir(parents=True, exist_ok=False) os.environ.update(HEADROOM_STATELESS="0", HEADROOM_WORKSPACE_DIR=str(out/'workspace'), HEADROOM_OFFLINE="0") class Endpoint(BaseHTTPRequestHandler): requests = [] def log_message(self, *_): pass def do_POST(self): payload=json.loads(self.rfile.read(int(self.headers['content-length']))) self.requests.append({'path':self.path,'body':payload}) assert self.path=='/api/embeddings' body=json.dumps({'embedding':[1.0]+[0.0]*383}).encode() self.send_response(200) self.send_header('Content-Type','application/json') self.send_header('Content-Length',str(len(body))) self.end_headers() self.wfile.write(body) server=ThreadingHTTPServer(('127.0.0.1',0),Endpoint) threading.Thread(target=server.serve_forever,daemon=True).start() async def run(): backend=LocalBackend(LocalBackendConfig(db_path=str(out/'memory.db'),embedder_backend='ollama',embedder_model='controlled-vector',vector_dimension=384,ollama_base_url=f'http://127.0.0.1:{server.server_port}')) state=out/'sync-state.json' try: if case=='fingerprint': saved=await backend.save_memory('Original orchard fact',user_id='alice') adapter=CodexAdapter(out/'AGENTS.md') first=await sync(backend,adapter,'alice',state) unchanged=await sync(backend,adapter,'alice',state) await backend._hierarchical_memory.update(saved.id,content='Corrected orchard fact') result=await sync(backend,adapter,'alice',state) text=adapter._path.read_text(encoding='utf-8') data={'first_exported':first.exported,'unchanged_exported':unchanged.exported,'edited_exported':result.exported,'corrected_in_file':'Corrected orchard fact' in text,'edited_imported':result.imported} assert data['first_exported']==1 and data['unchanged_exported']==0 assert data['corrected_in_file']==(expected=='after'),data assert result.exported==(1 if expected=='after' else 0),data else: content='Use foo---bar as the separator\nKeep the second line intact.' saved=await backend.save_memory(content,user_id='alice') adapter=ClaudeCodeAdapter(out/'claude-memory') first=await sync(backend,adapter,'alice',state,force=True) readback=await adapter.read_memories() repeated=await sync(backend,adapter,'alice',state,force=True) rows=await backend.get_user_memories('alice') data={'first_exported':first.exported,'round_trip_equal':readback[0].content==content,'reread_content':readback[0].content,'repeat_imported':repeated.imported,'repeat_exported':repeated.exported,'primary_count':len(rows)} assert data['round_trip_equal']==(expected=='after'),data assert data['repeat_imported']==(0 if expected=='after' else 1),data data['requests']=Endpoint.requests data['source_file']=__import__('headroom').__file__ (out/'result.json').write_text(json.dumps(data,indent=2),encoding='utf-8',newline='\n') print(json.dumps(data,ensure_ascii=False)) finally: await backend.close() try: asyncio.run(run()) finally: server.shutdown();server.server_close() ``` </details> ## Runtime Rollout Safety - Rollout-managed feature(s): none; existing memory sync. - Minimum rollout channel: existing memory sync users. - Stable/default behavior changed: edited sync snapshots trigger comparison/export instead of a false no-op. - Kill switch / disable path: stop invoking memory sync; no new toggle. - Unsafe override required: no. - Qualification impact: memory synchronization correctness; no dependency, native or workflow change. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented the non-obvious behavior - [x] I have added regression tests - [x] I did not edit CHANGELOG.md ## Additional Notes Implementation and local test execution were completed with Codex assistance. The author personally reviewed the two patches and the before/after evidence; this does not claim the author personally ran the tests. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4885" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
0d9155019f |
fix(memory): dispatch the advertised memory_list tool (#4830)
## Description `MemorySystem.get_tools()` advertises `memory_list`, but `process_tool_call()` has no handler for it. An agent can see the tool and still receive `Unknown tool: memory_list` when trying to browse stored memories or obtain an ID for an update. Dispatch the existing tool to the backend's scoped listing API. A session-bound MemorySystem lists only that user's current session; an unscoped one retains user-wide browsing. Return recent memories with their IDs and readable fields, without embeddings or a semantic query. Backends without the requested listing scope return an explicit unsupported-operation error rather than falling back to a wider scope. Refs #4362, item 1 only. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Add the `memory_list` dispatcher and public handler. - Keep the schema's default limit of 10 and bounds of 1–100. Filter by user and, when bound, session before applying the limit, using the existing HierarchicalMemory.get_session_memories API through LocalBackend. - Cover normal and optimized tool inventories, empty results, bounds, unsupported backends and backend failures. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook) - [x] Regression tests added - [x] Standalone runtime validation completed ### Test Output ```text Initial regressions against original production code: 8 failed Session regressions against initial PR head: 4 failed, 9 passed Final list regressions on the corrected head: 13 passed pytest tests/test_memory_list_dispatch.py tests/test_memory_system.py tests/test_memory_tool_project_scope.py -q: 113 passed, 13 skipped Original pre-commit and commit-msg hooks: passed Whole-tree Ruff check and format --check: passed Original pre-push make ci-precheck: passed Rust workspace: 1749 passed, 3 ignored Native extension: rebuilt, installed and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` The owning-suite skips require Hugging Face model files unavailable offline. Existing datetime/optional-model deprecation warnings remain visible in the logs. ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing dev venv; clean production baseline `9f2cc017863f540c61f8747b3c744a97776ebaf2`, patch `42aca035dde6734d1a9d7a0b87f1b99cf8b27e69`. Explicit checkout on PYTHONPATH; no paid/provider endpoint. The loopback service returns controlled vectors, not trained-model inference. - Exact command / steps: Save the script below as `proof.py`. From each checkout run `python proof.py list <fresh-output-directory> before` on main, or `python proof.py list <fresh-output-directory> after` on the patch. Identical scripts ran outside pytest. The script initializes a real LocalBackend, saves three Alice memories and one Bob memory through MemorySystem, then calls the advertised listing tool with limit=2. The Ollama HTTP adapter sends actual requests to a bound loopback receiver returning controlled 384-dimensional vectors; SQLite, FTS5 and the configured local vector index are real. <details> <summary>Standalone reproduction script</summary> ```python """Standalone real SQLite/vector/FTS/HTTP and agent-file boundary reproductions. Usage: python memory_http_proof.py {list,vectors,sync} FRESH_OUTPUT {before,after} The loopback embedding service returns controlled vectors, not model inference. """ import asyncio from datetime import datetime, timedelta, timezone import hashlib from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import json import os from pathlib import Path import platform import sqlite3 import sys import threading import headroom import headroom._core from headroom.memory.backends.local import LocalBackend, LocalBackendConfig from headroom.memory.config import EmbedderBackend, MemoryConfig, VectorBackend from headroom.memory.core import HierarchicalMemory from headroom.memory.models import Memory from headroom.memory.sync import sync, sync_import from headroom.memory.sync_adapters.codex_agent import CodexAdapter from headroom.memory.system import MemorySystem sys.stdout.reconfigure(encoding="utf-8") case, directory, expected = sys.argv[1:] assert case == "list" assert expected in ("before", "after") out = Path(directory).resolve() out.mkdir(parents=True, exist_ok=False) os.environ["HEADROOM_STATELESS"] = "0" os.environ["HEADROOM_WORKSPACE_DIR"] = str(out / "workspace") os.environ["HEADROOM_CCR_BACKEND"] = "memory" os.environ["HEADROOM_OFFLINE"] = "0" # Only the ephemeral loopback service is called. class Endpoint(BaseHTTPRequestHandler): dimension = 384 requests = [] def log_message(self, *_): pass def do_POST(self): payload = json.loads(self.rfile.read(int(self.headers["content-length"]))) Endpoint.requests.append({"path": self.path, "body": payload, "dimension": self.dimension}) assert self.path == "/api/embeddings", self.path body = json.dumps({"embedding": [1.0] + [0.0] * (self.dimension - 1)}).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) server = ThreadingHTTPServer(("127.0.0.1", 0), Endpoint) worker = threading.Thread(target=server.serve_forever, daemon=True) worker.start() url = f"http://127.0.0.1:{server.server_port}" def backend_config(name): return LocalBackendConfig( db_path=str(out / (name + ".db")), embedder_backend="ollama", embedder_model="probe-fixed", ollama_base_url=url, vector_dimension=384, ) async def list_case(): backend = LocalBackend(backend_config("list")) try: system = MemorySystem(backend, user_id="alice") ids = [] for content in ("First scoped fact", "Second scoped fact", "Latest scoped fact"): response = await system.process_tool_call("memory_save", {"content": content, "importance": 0.4}) assert response["success"], response ids.append(response["memory_id"]) response = await MemorySystem(backend, "bob").process_tool_call("memory_save", {"content": "Bob's private fact", "importance": 0.4}) assert response["success"], response before_requests = len(Endpoint.requests) result = await system.process_tool_call("memory_list", {"limit": 2}) result["embedding_requests_for_listing"] = len(Endpoint.requests) - before_requests result["stored_alice_count"] = len(await backend.get_user_memories("alice")) if expected == "after": assert result["success"] and result["count"] == 2, result assert [m["memory_id"] for m in result["memories"]] == [ids[2], ids[1]] assert all("Bob" not in m["content"] for m in result["memories"]) else: assert result["success"] is False and result["error"] == "Unknown tool: memory_list", result assert result["stored_alice_count"] == 3 and result["embedding_requests_for_listing"] == 0 return result finally: await backend.close() try: result = asyncio.run(list_case()) result.update({ "case": case, "expected": expected, "python": platform.python_version(), "source": str(Path(headroom.__file__).resolve()), "native": str(Path(headroom._core.__file__).resolve()), "http_requests": Endpoint.requests, "not_tested": "Trained model inference, paid APIs, provider answer quality, non-Windows runtime, non-local memory backends.", }) (out / "result.json").write_text(json.dumps(result, indent=2), encoding="utf-8", newline="\n") print(json.dumps(result, indent=2)) finally: server.shutdown() server.server_close() worker.join(timeout=5) ``` </details> - Observed result: On main the tool returns Unknown tool: memory_list even though Alice has three stored rows. With the patch it returns Alice's two newest memories, in order, with the saved IDs. Bob's memory is absent, and listing makes zero additional embedding requests. Session-scope follow-up: run the script below as `python session_proof.py <fresh-output-directory> before` on the initial PR head `908762a8705dea9a878f28dca4cb25b3fc5136ee`, or with `after` on the corrected head. It saves real memories for Alice/session-a, Alice/session-b and Bob/session-a through LocalBackend and its actual loopback Ollama HTTP adapter. With limit=1, the initial head returns the newer Alice/session-b fact to session-a; the correction returns session-a's fact. A missing session returns zero results, unscoped browsing still returns both Alice facts, Bob remains excluded, and listing sends no embedding request. <details> <summary>Session isolation reproduction</summary> ```python """Real LocalBackend listing must filter user and session before LIMIT.""" import asyncio from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import json import os from pathlib import Path import sys import threading from headroom.memory.backends.local import LocalBackend, LocalBackendConfig from headroom.memory.system import MemorySystem sys.stdout.reconfigure(encoding="utf-8") directory, expected = sys.argv[1:] assert expected in ("before", "after") out = Path(directory).resolve() out.mkdir(parents=True, exist_ok=False) os.environ.update(HEADROOM_OFFLINE="0", HEADROOM_STATELESS="0", HEADROOM_CCR_BACKEND="memory") class Endpoint(BaseHTTPRequestHandler): calls = [] def log_message(self, *_): pass def do_POST(self): request = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) assert self.path == "/api/embeddings" Endpoint.calls.append(request["prompt"]) response = json.dumps({"embedding": [1.0] + [0.0] * 383}).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(response))) self.end_headers() self.wfile.write(response) server = ThreadingHTTPServer(("127.0.0.1", 0), Endpoint) worker = threading.Thread(target=server.serve_forever, daemon=True) worker.start() async def main(): backend = LocalBackend(LocalBackendConfig( db_path=str(out / "memory.db"), embedder_backend="ollama", embedder_model="probe-fixed", ollama_base_url=f"http://127.0.0.1:{server.server_port}", vector_dimension=384, )) try: current = MemorySystem(backend, "alice", session_id="session-a") other_session = MemorySystem(backend, "alice", session_id="session-b") other_user = MemorySystem(backend, "bob", session_id="session-a") saved = [] for system, content in ((current,"Current-session fact"),(other_session,"Newer other-session fact"),(other_user,"Newest other-user fact")): result = await system.process_tool_call("memory_save", {"content":content,"importance":0.4}) assert result["success"],result saved.append(result["memory_id"]) count = len(Endpoint.calls) listed = await current.process_tool_call("memory_list", {"limit":1}) assert listed["success"] and listed["count"] == 1,listed scope_preserved = listed["memories"][0]["memory_id"] == saved[0] assert scope_preserved == (expected == "after"),listed assert listed["memories"][0]["memory_id"] != saved[2] missing = await MemorySystem(backend,"alice",session_id="missing").process_tool_call("memory_list", {"limit":1}) assert missing["count"] == (0 if expected == "after" else 1),missing unscoped = await MemorySystem(backend,"alice").process_tool_call("memory_list", {"limit":2}) assert [m["memory_id"] for m in unscoped["memories"]] == [saved[1],saved[0]] assert len(Endpoint.calls) == count result = {"session_isolated":scope_preserved,"scoped_content":listed["memories"][0]["content"],"missing_session_count":missing["count"],"unscoped_count":unscoped["count"],"other_user_excluded":True,"embedding_calls_for_listing":len(Endpoint.calls)-count,"save_http_calls":count,"expected":expected,"source":str(Path(__import__('headroom').__file__).resolve())} (out/"result.json").write_text(json.dumps(result,indent=2),encoding="utf-8",newline="\n") print(json.dumps(result,indent=2)) finally: await backend.close() try: asyncio.run(main()) finally: server.shutdown() server.server_close() worker.join(timeout=5) ``` </details> - Not tested: Remote memory backends, trained embedding inference, paid APIs, model answer quality, non-Windows runtime, or the complete Python suite. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing memory behavior; no new feature flag or channel expansion. - Minimum rollout channel: Normal release after maintainer review and CI qualification. - Stable/default behavior changed: The advertised tool can enumerate recent memories on backends that expose get_user_memories. Other memory operations retain their dispatch paths. - Kill switch / disable path: Use the existing memory-disable path, or revert this commit. - Unsafe override required: No. - Qualification impact: Related tests and original local gates passed; hosted CI is tracked separately. - Rollback path: Revert this commit; no dependency or storage-schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove the fix - [x] Relevant new and existing tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Listing is supported through LocalBackend's user and session APIs; the session adapter delegates to an existing core query that applies both filters before LIMIT. This patch does not add an enumeration API to DirectMem0. The author's review covered the initial patch and evidence; the session-isolation review correction was additionally reviewed and validated with Codex. Developed and validated with Codex assistance. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4830" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
cd6f438fec |
fix(memory): validate embedding dimensions before writes (#4831)
## Description A mismatched embedding reaches the vector index after the primary memory row has already been saved. A rejected add leaves a half-written row; a rejected supersession also ends the old version and removes its index entry before failing. This occurs with a vector_dimension that differs from the embedder's actual output. Validate each newly produced embedding's shape before the first primary-store mutation. Apply this to add, the whole add_batch, content updates and supersession. An invalid replacement leaves the old row, cached object and version chain intact. Refs #4362, item 2's half-write boundary. The configured vector dimension remains explicit; this does not automatically resize or migrate an index. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Reject non-vector and wrong-width embeddings before saving rows. - Validate an entire batch before saving any member. - Validate a content update before mutating a store-returned object, preserving update()'s existing object identity and metadata-only refresh behavior. - Validate supersession before ending the old version or removing its indexes. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook) - [x] Regression tests added - [x] Standalone runtime validation completed ### Test Output ```text Final regressions against original production code: 7 failed, 1 passed pytest tests/test_memory_embedding_validation.py tests/test_memory/test_core_operations.py tests/test_memory/test_hierarchical.py tests/test_memory/test_core_metadata_index_sync.py tests/test_memory/test_clear_scope_consistency.py -q: 65 passed, 51 skipped Original pre-commit and commit-msg hooks: passed Whole-tree Ruff check and format --check: passed Original pre-push make ci-precheck: passed Rust workspace: 1749 passed, 3 ignored Native extension: rebuilt, installed and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` The owning-suite skips require Hugging Face model files unavailable offline. Existing datetime/optional-model deprecation warnings remain visible in the logs. ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing dev venv; clean production baseline `9f2cc017863f540c61f8747b3c744a97776ebaf2`, patch `50d126b1b8379390e72d3e47216519f9d297b4a1`. Explicit checkout on PYTHONPATH; no paid/provider endpoint. The loopback service returns controlled vectors, not trained-model inference. - Exact command / steps: Save the script below as `proof.py`. From each checkout run `python proof.py vectors <fresh-output-directory> before` on main, or `python proof.py vectors <fresh-output-directory> after` on the patch. Identical scripts ran outside pytest. The script creates real HierarchicalMemory with SQLite, FTS5, HNSW(dimension=384) and the Ollama HTTP adapter. A bound loopback server first returns a 768-dimensional embedding for a rejected add, then 384 dimensions for a valid original, then 768 dimensions for a rejected supersession. It inspects primary rows, the original version and the live vector index. <details> <summary>Standalone reproduction script</summary> ```python """Standalone real SQLite/vector/FTS/HTTP and agent-file boundary reproductions. Usage: python memory_http_proof.py {list,vectors,sync} FRESH_OUTPUT {before,after} The loopback embedding service returns controlled vectors, not model inference. """ import asyncio from datetime import datetime, timedelta, timezone import hashlib from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import json import os from pathlib import Path import platform import sqlite3 import sys import threading import headroom import headroom._core from headroom.memory.backends.local import LocalBackend, LocalBackendConfig from headroom.memory.config import EmbedderBackend, MemoryConfig, VectorBackend from headroom.memory.core import HierarchicalMemory from headroom.memory.models import Memory from headroom.memory.sync import sync, sync_import from headroom.memory.sync_adapters.codex_agent import CodexAdapter from headroom.memory.system import MemorySystem sys.stdout.reconfigure(encoding="utf-8") case, directory, expected = sys.argv[1:] assert case == "vectors" assert expected in ("before", "after") out = Path(directory).resolve() out.mkdir(parents=True, exist_ok=False) os.environ["HEADROOM_STATELESS"] = "0" os.environ["HEADROOM_WORKSPACE_DIR"] = str(out / "workspace") os.environ["HEADROOM_CCR_BACKEND"] = "memory" os.environ["HEADROOM_OFFLINE"] = "0" # Only the ephemeral loopback service is called. class Endpoint(BaseHTTPRequestHandler): dimension = 384 requests = [] def log_message(self, *_): pass def do_POST(self): payload = json.loads(self.rfile.read(int(self.headers["content-length"]))) Endpoint.requests.append({"path": self.path, "body": payload, "dimension": self.dimension}) assert self.path == "/api/embeddings", self.path body = json.dumps({"embedding": [1.0] + [0.0] * (self.dimension - 1)}).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) server = ThreadingHTTPServer(("127.0.0.1", 0), Endpoint) worker = threading.Thread(target=server.serve_forever, daemon=True) worker.start() url = f"http://127.0.0.1:{server.server_port}" async def new_memory(name): return await HierarchicalMemory.create(MemoryConfig( db_path=out / (name + ".db"), embedder_backend=EmbedderBackend.OLLAMA, embedder_model="nomic-embed-text", ollama_base_url=url, vector_backend=VectorBackend.HNSW, vector_dimension=384, auto_bubble=False, )) async def vectors_case(): memory = await new_memory("vectors") Endpoint.dimension = 768 result = {} try: try: await memory.add("Rejected incompatible embedding", user_id="alice") raise AssertionError("mismatch was accepted") except ValueError as exc: result["add_error"] = str(exc) result["rows_after_rejected_add"] = len(await memory.get_user_memories("alice")) Endpoint.dimension = 384 original = await memory.add("Valid original fact", user_id="alice") Endpoint.dimension = 768 try: await memory.supersede(original.id, "Rejected replacement fact") raise AssertionError("replacement mismatch was accepted") except ValueError as exc: result["supersede_error"] = str(exc) stored = await memory._store.get(original.id) result["original_still_active"] = stored.valid_until is None and stored.superseded_by is None result["vector_entries_after_rejected_supersede"] = memory._vector_index.size with sqlite3.connect(out / "vectors.db") as db: result["all_primary_rows"] = db.execute("SELECT COUNT(*) FROM memories WHERE user_id = 'alice'").fetchone()[0] assert result["rows_after_rejected_add"] == (0 if expected == "after" else 1), result assert result["original_still_active"] == (expected == "after"), result assert result["vector_entries_after_rejected_supersede"] == (1 if expected == "after" else 0), result assert result["all_primary_rows"] == (1 if expected == "after" else 3), result return result finally: await memory.close() try: result = asyncio.run(vectors_case()) result.update({ "case": case, "expected": expected, "python": platform.python_version(), "source": str(Path(headroom.__file__).resolve()), "native": str(Path(headroom._core.__file__).resolve()), "http_requests": Endpoint.requests, "not_tested": "Trained model inference, paid APIs, provider answer quality, non-Windows runtime, non-local memory backends.", }) (out / "result.json").write_text(json.dumps(result, indent=2), encoding="utf-8", newline="\n") print(json.dumps(result, indent=2)) finally: server.shutdown() server.server_close() worker.join(timeout=5) ``` </details> - Observed result: Main leaves one row after the rejected add; after the rejected replacement it has three primary rows, the original is no longer active and its vector was removed. The patch leaves zero rows after the rejected add, then retains exactly the valid original row and its one vector after the rejected replacement. - Not tested: General cross-component transactions for arbitrary storage/index I/O failures, automatic dimension discovery or index migration, remote vector backends, trained model inference, paid APIs, non-Windows runtime, or the complete Python suite. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing memory behavior; no new feature flag or channel expansion. - Minimum rollout channel: Normal release after maintainer review and CI qualification. - Stable/default behavior changed: Embedding shape errors are reported before primary writes. Matching vectors and explicit auto_embed=False operations retain their existing behavior. - Kill switch / disable path: Use the existing memory-disable path, or revert this commit. - Unsafe override required: No. - Qualification impact: Related tests and original local gates passed; hosted CI is tracked separately. - Rollback path: Revert this commit; no dependency or storage-schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove the fix - [x] Relevant new and existing tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Broader regression exposed update()'s object-identity contract; the final fix preserves it without changing the existing fixture. The author's review covered the initial patch and evidence; this compatibility adjustment was additionally reviewed and tested with Codex. Developed and validated with Codex assistance. The author reviewed the changes and before/after evidence. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4831" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
a6a825c55d |
fix(memory): include older memories in agent sync (#4832)
## Description The memory sync engine builds its deduplication and export sets from only the latest 500 user memories. Older rows are omitted from agent files, and an existing older fact can be imported again as a duplicate. Use the user's full snapshot for synchronization, including the refetch after an import and the standalone import/export entry points. Carry the existing MemoryFilter limit=None semantics through LocalBackend and HierarchicalMemory; ordinary queries keep their default limit of 100. Refs #4362, item 9 only. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Request the full user-scoped snapshot at each sync entry point. - Document and type the existing unbounded query option without changing default query limits. - Cover a 501-row store, older-fact deduplication, import-triggered refetch, forced/ordinary sync, idempotence and user isolation. ## Testing - [x] Relevant unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] Type checking passes (original full-package mypy hook) - [x] Regression tests added - [x] Standalone runtime validation completed ### Test Output ```text Final regressions against original production code: 5 failed, 2 passed pytest tests/test_memory_sync_full_store.py tests/test_memory_sync.py tests/test_memory/test_core_operations.py tests/test_memory/test_clear_scope_consistency.py -q: 57 passed, 48 skipped Original pre-commit and commit-msg hooks: passed Whole-tree Ruff check and format --check: passed Original pre-push make ci-precheck: passed Rust workspace: 1749 passed, 3 ignored Native extension: rebuilt, installed and import-verified Critical Python: 177 passed, 8 skipped commitlint: passed ``` The owning-suite skips require Hugging Face model files unavailable offline. Existing datetime/optional-model deprecation warnings remain visible in the logs. ## Real Behavior Proof - Environment: Windows 11, CPython 3.13.9, existing dev venv; clean production baseline `9f2cc017863f540c61f8747b3c744a97776ebaf2`, patch `64e537fa2459d3540522bf934a32ee2e8e089ea7`. Explicit checkout on PYTHONPATH; no paid/provider endpoint. The loopback service returns controlled vectors, not trained-model inference. - Exact command / steps: Save the script below as `proof.py`. From each checkout run `python proof.py sync <fresh-output-directory> before` on main, or `python proof.py sync <fresh-output-directory> after` on the patch. Identical scripts ran outside pytest. The script initializes a real LocalBackend and SQLite store, seeds 501 chronologically ordered Alice rows plus a Bob row, and runs sync into a real CodexAdapter AGENTS.md file. It then presents the oldest existing fact from a second agent file and calls sync_import. The unpatched duplicate path makes an actual Ollama HTTP request to a loopback embedding service; the patched deduplication path makes none. <details> <summary>Standalone reproduction script</summary> ```python """Standalone real SQLite/vector/FTS/HTTP and agent-file boundary reproductions. Usage: python memory_http_proof.py {list,vectors,sync} FRESH_OUTPUT {before,after} The loopback embedding service returns controlled vectors, not model inference. """ import asyncio from datetime import datetime, timedelta, timezone import hashlib from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer import json import os from pathlib import Path import platform import sqlite3 import sys import threading import headroom import headroom._core from headroom.memory.backends.local import LocalBackend, LocalBackendConfig from headroom.memory.config import EmbedderBackend, MemoryConfig, VectorBackend from headroom.memory.core import HierarchicalMemory from headroom.memory.models import Memory from headroom.memory.sync import sync, sync_import from headroom.memory.sync_adapters.codex_agent import CodexAdapter from headroom.memory.system import MemorySystem sys.stdout.reconfigure(encoding="utf-8") case, directory, expected = sys.argv[1:] assert case == "sync" assert expected in ("before", "after") out = Path(directory).resolve() out.mkdir(parents=True, exist_ok=False) os.environ["HEADROOM_STATELESS"] = "0" os.environ["HEADROOM_WORKSPACE_DIR"] = str(out / "workspace") os.environ["HEADROOM_CCR_BACKEND"] = "memory" os.environ["HEADROOM_OFFLINE"] = "0" # Only the ephemeral loopback service is called. class Endpoint(BaseHTTPRequestHandler): dimension = 384 requests = [] def log_message(self, *_): pass def do_POST(self): payload = json.loads(self.rfile.read(int(self.headers["content-length"]))) Endpoint.requests.append({"path": self.path, "body": payload, "dimension": self.dimension}) assert self.path == "/api/embeddings", self.path body = json.dumps({"embedding": [1.0] + [0.0] * (self.dimension - 1)}).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) server = ThreadingHTTPServer(("127.0.0.1", 0), Endpoint) worker = threading.Thread(target=server.serve_forever, daemon=True) worker.start() url = f"http://127.0.0.1:{server.server_port}" def backend_config(name): return LocalBackendConfig( db_path=str(out / (name + ".db")), embedder_backend="ollama", embedder_model="probe-fixed", ollama_base_url=url, vector_dimension=384, ) async def sync_case(): backend = LocalBackend(backend_config("sync")) try: await backend._ensure_initialized() now = datetime.now(timezone.utc) memories = [Memory( id=f"alice-{i:04}", content=f"Distinct memory {i}", user_id="alice", created_at=now - timedelta(seconds=i), ) for i in range(501)] memories.append(Memory(content="Bob's private fact", user_id="bob")) await backend._hierarchical_memory._store.save_batch(memories) adapter = CodexAdapter(out / "AGENTS.md") first = await sync(backend, adapter, "alice", state_path=out / "sync-state.json", force=True) exported = {m.content for m in await adapter.read_memories()} older_adapter = CodexAdapter(out / "older-agent.md") await older_adapter.write_memories([{"content": "Distinct memory 500"}]) imported = await sync_import(backend, older_adapter, "alice") result = { "first_exported": first.exported, "oldest_exported": "Distinct memory 500" in exported, "other_user_exported": "Bob's private fact" in exported, "reimported_existing_oldest": imported, "agent_file_sha256": hashlib.sha256((out / "AGENTS.md").read_bytes()).hexdigest(), } with sqlite3.connect(out / "sync.db") as db: result["alice_primary_rows"] = db.execute("SELECT COUNT(*) FROM memories WHERE user_id = 'alice'").fetchone()[0] assert result["first_exported"] == (501 if expected == "after" else 500), result assert result["oldest_exported"] == (expected == "after"), result assert result["reimported_existing_oldest"] == (0 if expected == "after" else 1), result assert result["alice_primary_rows"] == (501 if expected == "after" else 502), result assert result["other_user_exported"] is False return result finally: await backend.close() try: result = asyncio.run(sync_case()) result.update({ "case": case, "expected": expected, "python": platform.python_version(), "source": str(Path(headroom.__file__).resolve()), "native": str(Path(headroom._core.__file__).resolve()), "http_requests": Endpoint.requests, "not_tested": "Trained model inference, paid APIs, provider answer quality, non-Windows runtime, non-local memory backends.", }) (out / "result.json").write_text(json.dumps(result, indent=2), encoding="utf-8", newline="\n") print(json.dumps(result, indent=2)) finally: server.shutdown() server.server_close() worker.join(timeout=5) ``` </details> - Observed result: Main exports 500 memories, omits the oldest and imports that existing fact as a new row (502 Alice rows). The patch exports all 501, includes the oldest, imports zero duplicates and keeps 501 Alice rows. Bob's row is excluded in both runs. - Not tested: Very large-store throughput or peak memory, concurrent sync snapshot consistency, changes to the existing fingerprint sampling policy, remote memory backends, trained model inference, paid APIs, non-Windows runtime, or the complete Python suite. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing memory behavior; no new feature flag or channel expansion. - Minimum rollout channel: Normal release after maintainer review and CI qualification. - Stable/default behavior changed: Agent sync covers all user rows; ordinary get_user_memories calls keep the default 100-row limit and explicit limits still apply. - Kill switch / disable path: Use the existing memory-disable path, or revert this commit. - Unsafe override required: No. - Qualification impact: Related tests and original local gates passed; hosted CI is tracked separately. - Rollback path: Revert this commit; no dependency or storage-schema change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove the fix - [x] Relevant new and existing tests pass locally - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Synchronization already materializes a snapshot and hash set. Covering the full store makes their size scale with the user's retained memories; this patch does not introduce a paging protocol or alter the existing no-op fingerprint algorithm. Developed and validated with Codex assistance. The author reviewed the changes and before/after evidence. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4832" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
d58b3ef19b |
fix(cache): clear older skips when storing successful results (#4819)
## Description
Concurrent compression requests can finish with different verdicts for
the same content key. If a no-op marks the key as skipped before another
request stores a successful result, the old put leaves that skip in
place. The next request returns the original even though the successful
compression is present in memory and SQLite.
Remove the older skip under the existing memory lock when admitting a
successful result. First-sight mark_skip and explicit move_to_skip
retain their original behavior. The initial PR also routed mark_skip
through disk invalidation; review identified the unnecessary DELETEs
that added to new non-compressible blocks, so that part has been
removed.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Clear an older skip when a successful result is admitted.
- Cover both existing skip APIs with memory-only and SQLite-backed
caches.
- Trace real SQLite statements to preserve the memory-only first-sight
skip path.
## Testing
- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook in WSL)
- [x] Regression tests added
- [x] Standalone runtime validation completed
### Test Output
```text
Four successful-result regressions failed on original main.
The added first-sight SQL regression failed on the initial PR and passes after review revision.
pytest tests/test_router_result_cache.py tests/test_router_cache_policy.py tests/test_transforms/test_content_router.py tests/test_content_router_concurrency.py tests/test_compression_determinism.py tests/test_cache tests/test_cold_prefix.py:
561 passed, 3 skipped
Original pre-commit and commit-msg hooks: passed
Whole-tree Ruff check and format --check: passed
Original pre-push make ci-precheck: passed
Rust workspace: 1749 passed, 3 ignored
Native extension: rebuilt, installed and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```
Optional-model skips and the existing token-count approximation warning
remain visible in the gate logs.
## Real Behavior Proof
- Environment: Windows 11, CPython 3.13.9, existing dev venv; clean
baseline `c05fbdddc873078dd97cb9ede841e75ef7ed0bef`, patch
`b7f8872ac59c41abd12e2274364bcfecba67ceb7`. Explicit checkout on
PYTHONPATH; no provider credentials. The proof enables loopback HTTP;
model downloads remain disabled.
- Exact command / steps: Save the following script as `proof.py` and run
`python proof.py` from either checkout with `PYTHONPATH=.`. Identical
before/after scripts ran outside pytest.
```python
"""Two actual router requests with real remote HTTP, then SQLite replay."""
from concurrent.futures import ThreadPoolExecutor
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
from pathlib import Path
import tempfile
import threading
from headroom.cache.result_store import ResultStore
from headroom.tokenizers.estimator import CharacterCounter
from headroom.transforms.content_router import CompressionCache, ContentRouter, ContentRouterConfig
os.environ['HEADROOM_CCR_BACKEND']='memory'
os.environ['HEADROOM_OFFLINE']='0' # This script binds and calls loopback only.
os.environ['HEADROOM_FREEZE_BLOCK_DECISION']='0'
first_seen, release_first = threading.Event(), threading.Event()
lock=threading.Lock();calls=[]
short='The deployment is healthy and the requested records are present.'
class Handler(BaseHTTPRequestHandler):
def do_POST(self):
payload=json.loads(self.rfile.read(int(self.headers['Content-Length'])))
with lock:
index=len(calls);calls.append(payload)
if index==0:
first_seen.set()
if not release_first.wait(15): raise RuntimeError('controller did not release response')
text=short
else:
text=payload['content']
data=json.dumps({'compressed':text,'original_tokens':len(payload['content'].split()),'compressed_tokens':len(text.split()),'compression_ratio':0.1 if index==0 else 1.0}).encode()
self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
def log_message(self,*args):pass
server=ThreadingHTTPServer(('127.0.0.1',0),Handler)
thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()
os.environ['HEADROOM_KOMPRESS_ENDPOINT']=f'http://127.0.0.1:{server.server_port}'
payload=[{'role':'tool','tool_call_id':'t1','content':'The service contains useful deployment observations for the current investigation. '*80}]
tokenizer=CharacterCounter()
try:
with tempfile.TemporaryDirectory(prefix='headroom-tier-proof-') as d:
router=ContentRouter(ContentRouterConfig(ccr_inject_marker=False, result_cache_disk=False))
disk=ResultStore(Path(d)/'cache.db',1800)
router._cache=CompressionCache(store=disk,namespace='proof:')
def apply():return router.apply(payload,tokenizer,force_kompress=True).messages[0]['content']
# The real HTTP client is built once before sharing the router.
router._get_kompress()
with ThreadPoolExecutor(max_workers=2) as pool:
earlier=pool.submit(apply)
assert first_seen.wait(10),'first router call did not reach HTTP'
later=pool.submit(apply)
noop=later.result(timeout=10)
release_first.set()
compressed=earlier.result(timeout=10)
replay=apply()
reopened=ContentRouter(ContentRouterConfig(ccr_inject_marker=False,result_cache_disk=False))
reopened._cache=CompressionCache(store=ResultStore(Path(d)/'cache.db',1800),namespace='proof:')
disk_replay=reopened.apply(payload,tokenizer,force_kompress=True).messages[0]['content']
print(json.dumps({'noop_is_original':noop==payload[0]['content'],'successful_request_is_compressed':compressed==short,'next_request_reuses_success':replay==short,'reopened_sqlite_reuses_success':disk_replay==short,'skip_entries':router._cache.skip_size,'http_calls':len(calls)},indent=2))
reopened._cache._store._db.close()
disk._db.close()
router._get_kompress()._client.close()
finally:
release_first.set();server.shutdown();server.server_close();thread.join(5)
```
The first-sight SQLite trace was also run outside pytest with this
script:
```python
"""Trace actual SQLite statements for first-sight negative cache entries."""
import json
from pathlib import Path
import tempfile
from headroom.cache.result_store import ResultStore
from headroom.transforms.content_router import CompressionCache
with tempfile.TemporaryDirectory(prefix='headroom-skip-io-') as directory:
store=ResultStore(Path(directory)/'cache.db',1800)
cache=CompressionCache(store=store)
statements=[]
store._db.set_trace_callback(statements.append)
for i in range(100):cache.mark_skip(f'new-{i}')
print(json.dumps({'new_skip_entries':cache.skip_size,'sqlite_statements':len(statements),'deletes':sum(s.startswith('DELETE ') for s in statements)}))
store._db.close()
```
- Observed result: Two real ContentRouter.apply calls reach a bound
loopback RemoteKompress endpoint. The controller lets the no-op finish
first, then releases the successful compression. On clean main,
next_request_reuses_success=false and skip_entries=1, while reopening
SQLite reads the successful result. With the patch both the next request
and reopened cache reuse that result, skip_entries=0, and the endpoint
still receives only the original two calls. The trace script records 100
DELETEs for 100 new skips on the initial PR, versus zero SQL statements
after this revision.
- Not tested: Trained-model quality, paid endpoints, production
throughput, cross-process in-memory coherence, network filesystems, or
the complete Python suite. HTTP responses are deterministic synthetic
fixtures; actual router, HTTP client and SQLite operations are used.
## Runtime Rollout Safety
- Rollout-managed feature(s): Existing cache behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: A successful store supersedes an
older skip. First-sight negative classification keeps its existing
memory-only path.
- Kill switch / disable path: Revert this commit to restore the prior
successful-store behavior; no new configuration is introduced.
- Unsafe override required: No.
- Qualification impact: Related tests and original local gates passed;
hosted CI is tracked separately.
- Rollback path: Revert this commit; no dependency or storage-schema
change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Developed and validated with Codex assistance. I reviewed the initial
changes and runtime evidence; the review follow-up was checked with
Codex.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4819"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
48f650ad04 |
test: make the trust_remote_code scan see every truthy value and run without git (#4785)
## Description
`tests/test_no_trust_remote_code.py` scans the tracked tree so that
nothing re-enables HuggingFace remote code execution. Its regex captured
the assigned value only when it started with a letter
(`([A-Za-z_][\w.]*)`), but the library tests the flag for truthiness, so
these also enable it and were not seen: `trust_remote_code=1`,
`trust_remote_code=(True)`, `trust_remote_code="yes"`, `setattr(cfg,
"trust_remote_code", True)` and `kwargs.setdefault("trust_remote_code",
True)`. The test also called `pytest.skip` when `git ls-files` failed
(for example in an exported tarball), so the guard silently stopped
running.
This PR widens the value rule (a quoted string, or a name or number,
with an optional leading `(`, treating anything except the exact name
`False` as enabling the flag), adds a second pattern for the `"name",
value` form, matches across a line break, and falls back to scanning
`headroom/**/*.py` when git is unavailable instead of skipping. The scan
of the current tree finds no offender with the wider rules.
Part of #4783
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `tests/test_no_trust_remote_code.py`: `_VALUE`, a wider `ASSIGNMENT`,
a new `NAMED_ARGUMENT` limited to
`setattr`/`getattr`/`setdefault`/`get`/`pop` calls, a `_values()` helper
that reports `(line, value)`, a git-less fallback in `_tracked_files()`,
and 17 new cases: calls that do not set the flag (`log.info("...",
True)`) must be ignored, every truthy shape must be flagged, a value on
the next line must be flagged, and an explicit `False` (including
`(False)`, `kwargs["..."] = False` and `setattr(..., False)`) must be
accepted.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
`mypy headroom` was not run (no change under `headroom/`). `ruff check`
and `ruff format` (0.17.3) are clean.
### Test Output
The old regex on the shapes the old test never saw (`re.findall` with
the old pattern):
```text
old regex matches=True trust_remote_code=True
old regex matches=False trust_remote_code=1
old regex matches=False trust_remote_code=(True)
old regex matches=False trust_remote_code="yes"
old regex matches=False setattr(cfg, "trust_remote_code", True)
old regex matches=False kwargs.setdefault("trust_remote_code", True)
```
The new tests with the old scan rule (value must start with a letter, no
`"name", value` pattern):
```text
6 failed, 10 passed (the =1, ="yes", mapping =1, setattr, setdefault and setattr(..., False) cases fail)
```
The new tests with this branch's scan:
```text
tests/test_no_trust_remote_code.py 19 passed (the tree scan finds no offender)
same file run from a copy of the tree without .git 19 passed, 0 skipped
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, pytest 9.2.0, `main` at
`3c66c91e`.
- Exact command / steps: `python -m pytest
tests/test_no_trust_remote_code.py`; then the same file from a copy of
the repository without `.git`; then the new tests with the scan logic
swapped for the old rule.
- Observed result: as above. The tree scan passes on the current tree
with the wider rules, so no existing code needs to change.
- Not tested: CI, and a deliberately injected `trust_remote_code=1` in a
real module (the shapes are covered by the unit cases instead).
## Runtime Rollout Safety
- Rollout-managed feature(s): none; test only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for the library and proxy. The
scan is stricter and no longer skips without git.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (this
file; the whole suite was not run)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- A value such as `"False"` (a string) is truthy in Python, so the scan
flags it; I think that is right for this guard.
- The git-less fallback scans only `headroom/`: an exported tarball has
no other tracked-file list.
AI assistance: this change, its tests and the runs above were done with
Claude (Sonnet 5.5), operated by me (HetCreep).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4785"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
dc94c46a1f |
test: remove the developer's GitHub and Copilot tokens from every test's environment (#4790)
## Description Any test that enters the proxy lifespan starts the Copilot quota poller (`headroom/subscription/copilot_quota.py`), which reads `GITHUB_COPILOT_GITHUB_TOKEN`, `GITHUB_TOKEN`, `COPILOT_GITHUB_TOKEN` or `GITHUB_COPILOT_API_TOKEN` from the environment and sends it as a `Bearer` header to `api.github.com`. `tests/conftest.py` scrubs only `HEADROOM_*`, so a developer with `GITHUB_TOKEN` in the shell has that token used by the test run. `tests/test_subscription_poll_credential_policy.py:1-8` already treats using a credential nobody meant for this purpose as a defect. This PR adds an autouse fixture that removes the nine Copilot/GitHub token variables that `copilot_quota.py` and `copilot_auth.py` read. Tests that need one still set it with `monkeypatch.setenv`, which runs after the fixture. Part of #4789 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `tests/conftest.py`: `_COPILOT_TOKEN_ENV_VARS` and the autouse fixture `_scrub_developer_copilot_tokens`, placed before `_scrub_developer_headroom_env`. - `tests/test_conftest_scrubs_copilot_tokens.py`: five names must be absent from `os.environ`, `discover_github_token()` finds nothing, and a test can still set its own token. No product code changes. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `mypy headroom` was not run (no change under `headroom/`). `ruff check` and `ruff format --check` (0.17.3) are clean on the two files. ### Test Output With `GITHUB_TOKEN` and `GH_TOKEN` set to dummy values in the shell, the new file: ```text this branch: 7 passed conftest from main: 3 failed, 4 passed (GITHUB_TOKEN and GH_TOKEN still in the environment, and discover_github_token() returns the dummy token) ``` The Copilot and GitHub-token related test files, with the same dummy variables set: ```text test_cli/test_wrap_copilot.py, test_cli/test_wrap_opencode.py, test_copilot_auth.py, test_copilot_quota.py, test_offline_egress_chokepoint.py this branch: 1 failed, 435 passed, 4 skipped conftest from main: 1 failed, 435 passed, 4 skipped test_release_workflows.py, test_subscription_poll_credential_policy.py, test_copilot_subscription_smoke.py this branch: 86 passed ``` The one failure is `tests/test_offline_egress_chokepoint.py::TestUpdateCheckOffline::test_explicit_update_check_never_constructs_request` (`AttributeError: module 'headroom.cli.main' has no attribute 'name'`). It fails the same way with `main`'s `conftest.py` and passes when run alone with either, so it depends on test order and is not caused by this change; I did not investigate it. ## Real Behavior Proof - Environment: Windows 11, Python 3.12, pytest 9.2.0, the compiled `headroom._core`, `main` at `3c66c91e`. - Exact command / steps: `GITHUB_TOKEN=ghp_dummy_not_real GH_TOKEN=dummy2 python -m pytest tests/test_conftest_scrubs_copilot_tokens.py`, once with this branch's `conftest.py` and once with `main`'s; then the two groups of files above with the same variables. - Observed result: as in Test Output. Only dummy values were used, never a real token (the runs of the other files with the dummy variables may have sent them to GitHub, where they are rejected). - Not tested: CI, Linux and macOS, the whole suite, and a network capture of the poller (the 53 lookups in #4789 come from an instrumented run, not from this PR's proof). ## Runtime Rollout Safety - Rollout-managed feature(s): none; test only. - Minimum rollout channel: n/a - Stable/default behavior changed: no. - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes (the files listed above; the whole suite was not run, and one order-dependent failure exists on `main`) - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Not in this PR: the other items of #4789 (credential readers in `test_copilot_auth.py`, writes to `~/.headroom`, tests that reach the network). - If a CI job relies on a token reaching a test, that test would now need `monkeypatch.setenv`; I found none in the files above. AI assistance: this change, its tests and the runs above were done with Claude (Sonnet 5.5), operated by me (HetCreep). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4790" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
327d92414f |
fix(cache): keep literal cache_control fields in the response-cache key (#4799)
## Description The proxy response-cache key policy strips `cache_control` recursively at every dict depth, with a single carve-out for JSON Schema `properties` names (#327). But `cache_control` is overloaded: on a content block, a `system` entry or a tool definition it is a transport prompt-cache directive, while inside tool arguments and JSON Schema literals it is ordinary application data the model actually sees. So two requests that differ only in that data get the same key. With `cache_enabled` on by default, the second caller is replayed the first caller's response for the full TTL. Reproduced on `main` (`e995eb9f`) against the real `SemanticCache` API, no mock: ```python msgs = [{"role": "assistant", "content": [{"type": "tool_use", "id": "c1", "name": "configure", "input": {"cache_control": "first"}}]}] changed = copy.deepcopy(msgs); changed[0]["content"][0]["input"]["cache_control"] = "second" c = SemanticCache() await c.set(msgs, "m", b'{"id":"first"}', {}, partition="caller") print("replayed:", (await c.get(changed, "m", partition="caller")) is not None) ``` Before (this branch's parent, `e995eb9f`): ```text changed tool payload replayed: True changed schema replayed: True ``` Closes #4272 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/semantic_cache_key_policy.py`: `strip_cache_control` no longer recurses into fields whose value is opaque payload — `input`, `arguments`, `input_schema`, `parameters`, `schema`, `properties`, `const`, `default`, `enum`, `example`, `examples`, `text` — so a `cache_control`-named field there stays in the key. Everywhere else (content blocks, `system`, tool definitions) it is still dropped, so a moved transport breakpoint still does not fragment the key. The narrower `properties`-only carve-out falls out of the general rule. - `tests/test_proxy_semantic_cache_key_policy.py`: covers the mandatory-partition precondition (`ValueError` when no partition is passed), which was the one uncovered line Codecov flagged in this change. - `tests/test_semantic_cache_literal_payloads.py`: new regression suite covering tool `input`, tool `input_schema` `default`, and `response_format` schema `const`, each through the real `SemanticCache` (changed request misses, original still hits), plus a guard for the decoded-dict `function.arguments` shape Headroom accepts (`handlers/openai.py` parses either), a guard that OpenAI's JSON-*string* `arguments` is passed through verbatim, and guards that transport `cache_control` is still ignored and nested transport annotations are still dropped. One pure module changed; no provider, no native rebuild. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `mypy` is unchecked because the pre-commit hook fails on 11 pre-existing Windows-only `attr-defined` errors in files this PR does not touch (`headroom/fileperms.py`, `headroom/ccr/mcp_server.py`, `headroom/proxy/interceptors/astgrep.py`: `os.fchmod`, `os.O_DIRECTORY`, `fcntl`); neither changed file produces a mypy error. ### Test Output Regression tests fail before the fix. With `headroom/proxy/semantic_cache_key_policy.py` reverted and the new tests kept: ```text $ git show HEAD~1:headroom/proxy/semantic_cache_key_policy.py > headroom/proxy/semantic_cache_key_policy.py $ uv run pytest tests/test_semantic_cache_literal_payloads.py -q FAILED tests/test_semantic_cache_literal_payloads.py::test_literal_cache_control_does_not_alias_cached_responses[tool_input] FAILED ... [tool_arguments_dict] FAILED ... [tool_schema] FAILED ... [response_schema] FAILED tests/test_semantic_cache_literal_payloads.py::test_literal_cache_control_changes_the_key[tool_input] FAILED ... [tool_arguments_dict] FAILED ... [tool_schema] FAILED ... [response_schema] FAILED tests/test_semantic_cache_literal_payloads.py::test_strip_cache_control_still_drops_nested_transport_annotations 9 failed, 3 passed in 0.64s ``` The three that pass against the old policy are the guards: the OpenAI string-`arguments` case (a string is never rewritten), the schema-property-name case (#327), and the moved-transport-breakpoint case. After the fix: ```text $ uv run pytest tests/test_semantic_cache_literal_payloads.py tests/test_proxy_semantic_cache_key_policy.py \ tests/test_semantic_cache_key_policy.py tests/test_proxy_response_cache_partition.py \ tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_integration.py \ tests/test_openai_response_cache_key.py tests/test_proxy_openai_cache_key_integration.py -q 94 passed, 1 warning in 28.39s $ uv run pytest <the four policy/partition files> -q \ --cov=headroom.proxy.semantic_cache_key_policy --cov-report=term-missing --cov-branch Name Stmts Miss Branch BrPart Cover headroom\proxy\semantic_cache_key_policy.py 54 0 16 0 100% ``` $ uv run pytest tests/test_cache/test_semantic.py tests/test_openai_response_cache_key.py \ tests/test_proxy_openai_cache_key_integration.py tests/test_proxy_response_cache_store_guard.py \ tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_integration.py \ tests/test_shared_context.py -q 133 passed, 1 warning in 34.11s $ uv run pytest tests/test_cache_control_move_bust.py tests/test_anthropic_fork_lineage_cache.py \ tests/test_cross_turn_cache_safety.py tests/test_cache/test_prefix_tracker.py -q 113 passed, 1 warning in 6.21s $ uv run ruff check headroom/proxy/semantic_cache_key_policy.py tests/test_semantic_cache_literal_payloads.py All checks passed! $ uv run ruff format --check <both files> 1 file already formatted ``` ## Real Behavior Proof - Environment: Windows 11, `uv run` against the project venv (Python 3.13.11), branch `fix/4272-literal-payload-cache-keys` cut from upstream `main` `e995eb9f`. No mocked cache and no provider: the reproducer calls the real `SemanticCache.set`/`get` in `headroom/proxy/semantic_cache.py`. - Exact command / steps: run the reproducer above as a script under `uv run python` — store a response under a tool input containing `{"cache_control": "first"}`, deep-copy the messages, change that value to `"second"`, then `get` with the changed messages; repeat for a tool `input_schema` `default` and a `response_format` schema `const`. - Observed result: before the fix the changed request hit the cache in both cases (`changed tool payload replayed: True`, `changed schema replayed: True`). After the fix both print `False`, and the original request still hits (asserted in each regression test), so the key was not merely widened. - Not tested: the real OpenAI HTTP path with a decoded-dict `arguments` field, which is a nonstandard client encoding (`arguments` is normally a JSON string, covered by `test_openai_arguments_string_is_not_rewritten`); the full `make ci-precheck` (Rust/native rebuild) and the end-to-end proxy HTTP path; the change is confined to the pure key policy, which the tests above exercise directly through the real `SemanticCache`. `headroom/cache/prefix_tracker.py` has the same blunt recursive strip, but it serves prefix-content equality rather than response replay and is left as a separate follow-up. ## Runtime Rollout Safety - Rollout-managed feature(s): none; the proxy response cache (`cache_enabled`) is not a rollout-gated feature. - Minimum rollout channel: n/a - Stable/default behavior changed: yes, deliberately — for requests whose tool data or JSON Schema values contain a `cache_control`-named field, two distinct requests that used to share one key now get distinct keys. Requests without such a field produce the same key as before, so hit rate there is unchanged. - Kill switch / disable path: `cache_enabled = false`, or the existing proxy config for the response cache; no new switch. - Unsafe override required: no - Qualification impact: none; no rollout-managed surface touched. - Rollback path: revert this commit. Cache keys are process-local (`SemanticCache` is in-memory), so a revert needs no cache migration or invalidation. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — not needed: the key policy's docstrings describe the new rule, and no user-facing docs claim `cache_control` is stripped from payloads. - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes `headroom/cache/prefix_tracker.py:165` has the same blunt recursive strip, used for prefix-content equality rather than response replay. Worth a follow-up, deliberately out of scope here so the response-replay fix stays a one-module change. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
973c12d2af |
fix(cache): replay every parallel tool result (#4818)
## Description
An Anthropic user message can hold several parallel `tool_result`
blocks. The session compression cache recorded and replayed only the
first one, but a hit for that result could freeze the whole message.
Later results then reached the provider in their raw form instead of the
previously compressed form.
Visit every result when recording, replaying and marking stable content.
Freeze a message only if every result is replayable or textless. Reuse
the existing single-result shape guard, so images, tool references,
multiblock boundaries and text-block metadata retain their established
protection. Cache seeding checks the paired tool-call IDs before
associating original and compressed text.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Cache and replay each parallel result independently without mutating
caller input.
- Stop the frozen prefix on a later miss or unsupported text layout.
- Cover a textless first result, later text-block metadata, changed IDs
and partial replay. Update the existing later-result limitation test to
assert actual replay.
## Testing
- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook)
- [x] Regression tests added
- [x] Standalone runtime validation completed
### Test Output
```text
New regressions against original production code: 5 failed, 1 passed
pytest tests/test_compression_cache.py tests/test_session_engine.py tests/test_cache_replay_shapes.py tests/test_token_headroom_mode.py tests/test_cache_prefix_safety.py tests/test_proxy_anthropic_cache_stability.py tests/test_prefix_tracker_restart.py:
189 passed
Original pre-commit and commit-msg hooks: passed
Whole-tree Ruff check and format --check: passed
Original pre-push make ci-precheck: passed
Rust workspace: 1749 passed, 3 ignored
Native extension: rebuilt, installed and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```
Optional-model skips and the existing token-count approximation warning
remain visible in the gate logs. The restart tests also report the
environment's existing Starlette/httpx deprecation warning.
## Real Behavior Proof
- Environment: Windows 11, CPython 3.13.9, existing dev venv; clean
baseline `c05fbdddc873078dd97cb9ede841e75ef7ed0bef`, patch
`a09ad47dc8ceb04b42858a67eecb83a7ee246485`. Explicit checkout on
PYTHONPATH; no provider credentials. Headroom runs offline; the parallel
proof uses a real SDK to send only to its bound loopback receiver.
- Exact command / steps: Save the following script as `proof.py` and run
`python proof.py` from either checkout with `PYTHONPATH=.`. Identical
before/after scripts ran outside pytest.
```python
"""Production cache/session APIs followed by real Anthropic SDK loopback HTTP."""
import copy
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import threading
from anthropic import Anthropic
from headroom.cache.compression_cache import CompressionCache
from headroom.proxy.session_engine import FREEZE_POLICY_REPLAYABLE, prepare_turn
captured = []
class Handler(BaseHTTPRequestHandler):
def do_POST(self):
captured.append(json.loads(self.rfile.read(int(self.headers['Content-Length']))))
data = json.dumps({'id':'local','type':'message','role':'assistant','model':'local-fixture','content':[{'type':'text','text':'received'}],'stop_reason':'end_turn','stop_sequence':None,'usage':{'input_tokens':1,'output_tokens':1}}).encode()
self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
def log_message(self,*args): pass
def message(a,b):
return {'role':'user','content':[{'type':'tool_result','tool_use_id':'t1','content':a},{'type':'tool_result','tool_use_id':'t2','content':b}]}
server = ThreadingHTTPServer(('127.0.0.1',0),Handler)
thread = threading.Thread(target=server.serve_forever,daemon=True);thread.start()
raw = message('first original '*20,[{'type':'text','text':'second original '*20,'cache_control':{'type':'ephemeral'}}])
short = message('first compressed',[{'type':'text','text':'second compressed','cache_control':{'type':'ephemeral'}}])
messages=[{'role':'assistant','content':[{'type':'tool_use','id':'t1','name':'inspect','input':{}},{'type':'tool_use','id':'t2','name':'inspect','input':{}}]},raw,{'role':'user','content':'continue'}]
before = copy.deepcopy(messages)
cache = CompressionCache();cache.update_from_result([raw],[short])
prep = prepare_turn(cache,messages,policy=FREEZE_POLICY_REPLAYABLE)
try:
with Anthropic(api_key='local-fixture',base_url=f'http://127.0.0.1:{server.server_port}',max_retries=0) as client:
client.messages.create(model='local-fixture',max_tokens=16,messages=prep.pipeline_input)
wire = captured[-1]['messages'][1]
print(json.dumps({'all_results_replayed':wire==short,'second_result':wire['content'][1]['content'],'frozen_count':prep.frozen_message_count,'caller_unchanged':messages==before,'requests':len(captured)},indent=2))
finally:
server.shutdown();server.server_close();thread.join(5)
```
- Observed result: With the same original/compressed message pair, clean
main freezes the two-message prefix but the real Anthropic SDK sends the
second result as its original text. With the patch, the HTTP receiver
gets both compressed results, including the second text block's
cache_control; frozen_count remains 2 and caller_unchanged remains true.
- Not tested: Live Anthropic service acceptance, model inference,
billing, task accuracy, or the complete Python suite. The receiver
records actual SDK JSON and returns a synthetic response.
## Runtime Rollout Safety
- Rollout-managed feature(s): Existing cache behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Cached replay covers every tool
result in a message; a later cache miss or unsupported shape keeps that
message mutable.
- Kill switch / disable path: Revert this commit to restore the prior
replay behavior; no new configuration is introduced.
- Unsafe override required: No.
- Qualification impact: Related tests and original local gates passed;
hosted CI is tracked separately.
- Rollback path: Revert this commit; no dependency or storage-schema
change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Developed and validated with Codex assistance. I reviewed the changes
and runtime evidence.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4818"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
d7a2fcf092 |
test: make the loopback-gating tests able to tell the guard from a missing key (#4784)
## Description
`tests/test_proxy_loopback_gating.py` says non-loopback callers get a
404 on the gated routes, but `test_non_loopback_caller_gets_404` used a
default `TestClient`, whose peer is the string `"testclient"` (not an
IP), and asserted only the 404 status. Two of the twelve gated routes,
`/v1/telemetry/tools/{hash}` and `/v1/toin/pattern/{prefix}`, answer 404
on their own when the key is absent, and `test_loopback_caller_allowed`
accepts `{200, 404, 422}`. So removing the guard from those routes, or
making it refuse every parseable IP, did not turn the test red.
The guard's denial body is `{"detail": "Not Found"}` (what FastAPI
answers for an unknown path); a detail route answers with its own
message. This PR (1) uses a routable remote peer (`203.0.113.9`,
TEST-NET-3) with a loopback `Host`, so only the client-IP check can
refuse it, (2) asserts the guard's exact denial body, and (3) asserts
that a loopback caller never receives that body.
Part of #4783
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `tests/test_proxy_loopback_gating.py`: a `GUARD_DENIAL` constant with
a comment saying why; `test_non_loopback_caller_gets_404` uses
`client=("203.0.113.9", 4242)` and `base_url="http://127.0.0.1"` and
asserts `resp.json() == GUARD_DENIAL`; `test_loopback_caller_allowed`
also asserts `resp.json() != GUARD_DENIAL`. No product code changes.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
`mypy headroom` was not run (no change under `headroom/`). `ruff check`
and `ruff format --check` (0.17.3) are clean.
### Test Output
Whole file, product unchanged:
```text
pytest tests/test_proxy_loopback_gating.py 147 passed
```
The same file run against two mutations of the product (each restored
afterwards), with the old tests (`origin/main`) and the new ones:
```text
Mutation A: loopback_guard.py `return address.is_loopback` -> `return True`
old tests: 20 failed, 127 passed
new tests: 32 failed, 115 passed (the 12 non-loopback cases now fail too)
Mutation B: remove `dependencies=[Depends(_require_loopback)]` from
/v1/telemetry/tools/{signature_hash} and /v1/toin/pattern/{hash_prefix:path}
old tests: 147 passed (nothing noticed)
new tests: 2 failed, 145 passed (the two routes)
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, pytest 9.2.0, the compiled
`headroom._core`, `main` at `3c66c91e`.
- Exact command / steps: `create_app(ProxyConfig(...))` and three
`TestClient`s (default peer, `client=("203.0.113.9", 4242)`,
`client=("127.0.0.1", 12345)`, the last two with
`base_url="http://127.0.0.1"`); `GET` each of
`/v1/telemetry/tools/example`, `/v1/toin/pattern/example`,
`/v1/feedback/example`; then the two mutations above.
- Observed result: both non-loopback peers got `404 {"detail":"Not
Found"}` on all three; the loopback peer got `404 {"detail":"No
telemetry found for signature: example"}`, `404 {"detail":"No TOIN
pattern found with hash starting with: example"}` and `200`
respectively. The mutation results are in Test Output.
- Not tested: CI, other platforms, and the two non-GET gated routes
under mutation B (they are covered by the new body assertion but I did
not mutate them).
## Runtime Rollout Safety
- Rollout-managed feature(s): none; test only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (this
file; the whole suite was not run)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- The vanilla `TestClient` (peer `"testclient"`) is also refused by the
guard; I did not keep a test for it because it is not a
loopback-versus-remote distinction.
AI assistance: this change, its tests and the runs above were done with
Claude (Sonnet 5.5), operated by me (HetCreep).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4784"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
854a696544 |
fix(proxy): retry a TLS error only while the request is being written (#4776)
## Description A corrupt TLS record after the handshake surfaces as a raw `ssl.SSLError` (for example `[SSL: RECORD_LAYER_FAILURE] record layer failure`, or the `SSLV3_ALERT_BAD_RECORD_MAC` bursts in #1615). httpcore maps `ssl.SSLError` to `ConnectError` only in `start_tls`; its `read`/`write` maps cover only anyio's resource errors and timeouts, and anyio's `TLSStream` re-raises a non-EOF `SSLError` as it is. So the error is not an `httpx.TransportError`, and the #1639 retry in `_retry_request` and `_stream_response` lets it through. That error alone does not say whether upstream received the request. This PR retries it only when httpcore reports that writing the request failed: its documented `trace` request extension emits `send_request_headers.failed` or `send_request_body.failed`. Upstream then never had a complete request to run. A TLS error raised after the request was written propagates, as on main, so a POST upstream may have accepted is never sent twice. In the streaming path it still ends as the SSE 502, now with `upstream_tls_error` instead of a generic `api_error`. What this no longer fixes: the case the first version of this PR targeted, a corrupt record in the reply on a reused HTTP/1.1 keep-alive connection, happens after the request was written. It is ambiguous, so it still fails the request (502), as on main. What it does fix: a TLS failure while the request is still being written, for example on a shared HTTP/2 connection where a large body waits for flow-control window and the read for that window hits the corrupt record. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `helpers.py`: `RequestWriteTrace`, an httpcore `trace` callback that records whether `send_request_headers` or `send_request_body` failed. `forbids_retry(exc)` is true for a raw `ssl.SSLError` raised after a complete write. - `server.py` (`_retry_request`) and `handlers/streaming.py` (`_stream_response`): each attempt attaches a fresh trace, and a raw `ssl.SSLError` is retried only when that trace shows the write failed. Every other retry is unchanged. The streaming exhaustion handler still catches `ssl.SSLError`, for the 502 `upstream_tls_error` SSE body. - `tests/test_h2_stream_reset_retry.py`: the SSL cases now run through real httpcore over a fake socket. An accepted POST sends one POST and propagates, in the buffered and the streaming path. A failed write is retried, in both paths. The review's harness (an `httpx.MockTransport` that records the POST, then raises `ssl.SSLError`) sends one POST. - Three test fakes of `http_client.post` accept the new `extensions` kwarg. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_h2_stream_reset_retry.py -q 11 passed in 0.39s # source reverted to the previous head's retry ( |
||
|
|
998d71080c |
fix(proxy): keep Copilot requests on Copilot when an extension advises another provider (#4894)
## Description A proxy in front of GitHub Copilot (started by `headroom wrap vscode` / `wrap copilot`) holds only a Copilot token. When a proxy extension published cross-provider route advice (`request.state.headroom_route`), `BackendResolver.for_request` built a translating LiteLLM backend for it without looking at the upstream. A GPT-family Copilot call advised onto a Claude model was sent to Anthropic directly and failed: ``` LiteLLM backend initialized (provider=anthropic, region=None) route advice: claude-sonnet-5-5 via anthropic (...) /stats -> failed_by_provider: litellm-anthropic ``` The resolver now takes the request's upstream and ignores cross-provider advice when that upstream is Copilot. Companion change: headroomlabs-ai/routemegood#49 stops the routing extension publishing such advice in shadow mode and for holdouts. This PR is the safety property for apply mode. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/route_advice.py`: `for_request` takes `upstream_url`. When `is_copilot_upstream_url(upstream_url)` it returns the default backend and leaves `body["model"]` alone, with one info log line (model id and provider only). - `headroom/proxy/handlers/openai.py`, `handlers/anthropic.py`: all five call sites pass the request's upstream. The Anthropic sites are included because `wrap vscode` points the Anthropic target at Copilot as well. - `handlers/openai.py`: chat-stream CCR eligibility asks the resolver whether a backend will serve the request, instead of treating any advice as "routed to backend". Ignored advice no longer disables CCR tool injection. - `tests/test_route_advice_native_providers.py`: a resolver-level test (GHE Copilot host) and an end-to-end test through `create_app`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text CI on this PR: 34 checks passed, 12 skipped, 0 failed (lint, commitlint, test shards 1-4, test-macos). Locally: ruff 0.17.4 check + format --check clean on the 4 changed files; mypy clean on the 3 source files. tests/test_route_advice_native_providers.py: 8 passed. With the Copilot check disabled, test_copilot_chat_stays_on_copilot_despite_cross_provider_advice fails with: assert ['anthropic'] == [] ``` ## Real Behavior Proof - Environment: macOS 27.2.1, Python 3.12.13, core run from source (this branch vs `origin/main` |
||
|
|
ac932eb7c4 |
feat(telemetry): split beacon sessions by harness, model and day (#4881)
## Description
After backfilling the Beacon corpus, six questions about Oct 19 – Nov 14
could not be answered from what the client sends:
1. **Which harness, for a quarter of sessions.** 228,441 of 933,044
schema-v2 sessions (36.2M turns) carry no client. 30.2M of those turns
came through `/v1/compress`.
2. **Tokens and savings per harness.** `clients` has turn counts only;
34,553 sessions mix harnesses and cannot be divided.
3. **Harness version.** Only the Headroom version is reported.
4. **Tokens and savings per model.** `models` is a list, so savings
cannot be priced.
5. **Real installs versus churned ids.** Linux proxies show 19,916
installs for about the traffic of 4,258 on macOS.
6. **Per-day figures.** Always-on proxies report one session that grows
for weeks (34% of savings sat in multi-day sessions).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
All client changes are in `headroom/telemetry/session.py`, from data the
outcome funnel and the HTTP middleware already have. No handler changes.
`schema_version` stays 2; the fields are additive.
| Field | Content |
|---|---|
| `client_usage` | Per-harness rows of `n, original, saved, tool_saved,
output, cache_read, cache_write, cache_write_inferred, cache_write_5m,
cache_write_1h, uncached`. Covers every turn: unclaimed turns are
`via_mcp`, `via_compress_api` or `unidentified`. |
| `model_usage` | The same columns per public model id; anything else
pools into `other`. |
| `client_versions` | `(client, version, n)` from the User-Agent, e.g.
`claude_code 2.1.311`. Request counts, not turns. |
| `install` | `age_days` (id file mtime), `container`, `ci`. |
| rollover | A session at least 12h old is closed by the first request
on a later UTC day (`ended: "rollover"`). Sessions that merely run past
midnight are untouched. |
- `clients` and `models` are unchanged.
- `deploy/beacon/worker.js` learns the new keys and their nested shapes.
- `headroom/proxy/auth_policy.py` gains
`client_version_from_user_agent`; `headroom/proxy/server.py` calls it
for every POST.
**Pricing a usage row (review feedback).** Every column is the raw
per-turn counter and sums to its session total. Where a provider reports
no write count, Headroom infers one as input minus cached; those are the
same tokens as `uncached`. `cache_write_inferred` is that part, so
billed input is `uncached + cache_read + (cache_write -
cache_write_inferred)`, the rule `CostTracker.record` applies, and
`cache_write_5m` / `cache_write_1h` give the TTL split.
`test_a_usage_row_prices_inferred_cache_writes_the_way_the_cost_tracker_does`
covers the 1000-not-2000 case.
**Privacy.**
- Harness names in the new fields are checked against a closed list
(`_KNOWN_CLIENTS`), because `classify_client` returns the caller's
`X-Client` header as typed. Unknown names become `other`. The existing
`clients` field has the same gap and is not changed here.
- A version is sent only if it directly follows a known harness prefix
and is one to three numeric parts.
- The Worker holds the new string leaves to a slug or version pattern;
its test asserts injected text, hostnames and wrong types are dropped.
## Testing
- [x] Unit tests pass (`pytest`) — the suites listed below, not the
whole repository
- [x] Linting passes (`ruff check .`) — on the changed files
- [ ] Type checking passes (`mypy headroom`) — 8 errors in
`headroom/cache/prefix_tracker.py` and `headroom/ccr/mcp_server.py`,
identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_beacon_schema_v2.py tests/test_telemetry_session_selfcheck.py \
tests/test_telemetry.py tests/test_beacon_gzip.py tests/test_auth_policy.py tests/test_header_isolation.py -q
============================= 165 passed in 11.75s =============================
$ node deploy/beacon/test-rollup.mjs
ok
$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
```
Not run: the full Python suite. 3 tests in
`tests/test_proxy_telemetry_env.py` fail the same way on `main` without
this change.
## Real Behavior Proof
- Environment: macOS (darwin arm64), Python 3.12.13, this branch's proxy
(`service.version 0.45.0-dev`) with an isolated
`HEADROOM_WORKSPACE_DIR`, `HEADROOM_BEACON=on`,
`HEADROOM_TELEMETRY_ENDPOINT=http://127.0.0.1:8791/v1/logs`. Receiver:
this branch's `deploy/beacon/worker.js` `fetch()` served by a 30-line
Node HTTP listener with the R2 binding replaced by a directory.
Providers: Anthropic `claude-haiku-4-5-20251001` and OpenAI
`gpt-4o-mini`, live.
- Exact command / steps:
1. `node receiver.mjs deploy/beacon/worker.js ./r2 8791`
2. `headroom proxy --port 8799` (from this branch)
3. Four POSTs through the proxy: two `/v1/messages` with `User-Agent:
claude-cli/2.1.311 (external, cli)`, one `/v1/messages` with
`User-Agent: curl/8.4.0`, one `/v1/chat/completions` with `User-Agent:
codex-cli/0.45.0`.
4. `kill -INT` the proxy so it sends its closing report.
- Observed result: the three Anthropic calls returned 200 (the second
and third were served by Headroom's response cache); the OpenAI call
returned 429 from the provider (account quota). The receiver logged
`POST /v1/logs -> 204` and stored one object,
`sessions/dt=2026-11-15/hh=15/<uuid>.json`. Its new fields, as stored
after the allowlist:
```json
"session": {"seq": 0, "duration_s": 5, "turns": 4, "ended": "shutdown", "final": true}
"tokens": {"original": 28, "input": 28, "output": 4, "saved": 0, "cache_read": 0, "cache_write": 14, "uncached": 28}
"clients": [{"client": "claude", "n": 1}, {"client": "claude_code", "n": 2}, {"client": "codex", "n": 1}]
"client_usage": [
{"client": "claude", "n": 1, "original": 0, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 0},
{"client": "claude_code", "n": 2, "original": 14, "saved": 0, "tool_saved": 0, "output": 4, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14},
{"client": "codex", "n": 1, "original": 14, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 14, "cache_write_inferred": 14, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14}]
"model_usage": [
{"model": "claude-haiku-4-5-20251001", "n": 3, "original": 14, "output": 4, "cache_write": 0, "cache_write_inferred": 0, "uncached": 14, "...": "other columns 0"},
{"model": "gpt-4o-mini", "n": 1, "original": 14, "output": 0, "cache_write": 14, "cache_write_inferred": 14, "uncached": 14, "...": "other columns 0"}]
"client_versions": [{"client": "claude_code", "version": "2.1.311", "n": 2}, {"client": "codex", "version": "0.45.0", "n": 1}]
"install": {"age_days": 0, "container": false, "ci": false}
```
Both tables' `n` sum to `session.turns` (4) and `original` to
`tokens.original` (28). The OpenAI row is the reviewed case:
`cache_write` 14 is entirely inferred, and `uncached + cache_read +
(cache_write - cache_write_inferred)` is 14, not 28. Nothing from the
User-Agent beyond harness and version reached the stored object.
- Not tested:
- The Worker on Cloudflare itself (Workers runtime, real R2).
`worker.js` ran under Node.
- A successful OpenAI turn, a Responses API turn with a reported write,
and Anthropic turns with cache reads or 5m/1h writes (covered by unit
tests only).
- `via_compress_api`, `via_mcp` and `unidentified` labels, a session
mixing harnesses on one model, and the 12-hour rollover (unit tests
only).
- Codex over WebSocket, which never passes the HTTP middleware and so
reports no version.
- gzip transport (`HEADROOM_BEACON_GZIP=1`) and a heartbeat (`ended:
"active"`); only the closing report was observed.
- Linux, Windows, containers and CI.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; the beacon is not behind a rollout
channel.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, for installs with the beacon on
(the default). The session report carries four more keys (about 5 KB
more per report for a typical session with gzip off), and a session at
least 12 hours old now ends at the first request on a later UTC day, so
always-on installs report about one session per day instead of one that
grows for weeks. No request-path behaviour changes.
- Kill switch / disable path: `HEADROOM_BEACON=off`,
`HEADROOM_TELEMETRY=off`, `DO_NOT_TRACK=1` or offline mode disable all
of it, as before. Server side, removing a key from `ALLOWED_KEYS` in
`deploy/beacon/worker.js` drops that field for every install without a
client release.
- Unsafe override required: no
- Qualification impact: none known.
- Rollback path: revert this PR. Rows already stored keep the extra
keys; readers use `union_by_name`.
- Deploy order: the Worker must be deployed before a client release that
sends these fields, or they are dropped on arrival. Production is still
the 2026-10-08 Worker build.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — none
made; `docs/content/docs/security-model.mdx` and `api-reference.mdx`
describe beacon fields and have not been updated
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — for the
suites listed under Testing
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- `client_versions` misses WebSocket sessions.
- `/v1/compress` traffic is now visible but still not attributed to the
integration that called it.
- Tool-call kinds, tool-definition size, compaction and subagent turns
are not included: they need the request body in each handler and a
privacy design for tool names.
- An earlier review suggested splitting rollover into its own PR since
it changes what a session row means; it is kept here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
00436a8b11 |
feat(savings): book modelled output savings per request, as a token and dollar range (#4878)
## Description
Benchmark factors registered with `register_modelled_factors` reached
`/stats` only as a percentage. They never reached the per-request
rollup, so modelled output savings had no tokens or dollars in the
lifetime totals that the local dashboard and the control-plane heartbeat
read. This PR books each shaped request's output saving as a low–high
range of tokens and dollars, from the same tier the dashboard reports.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue) — an unshaped
treatment request was booked a baseline saving
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `SavingsRecorder.request_savings_range(labels, output_tokens) -> (low,
high)` replaces `estimate_request_savings` in the outcome funnel.
- It counts only requests that were actually shaped, marked by the
`output_shaper:verbosity:L<n>` label. The treatment stratum label is
attached before shaping is attempted, so it proves nothing.
- A holdout (control) request books nothing.
- A response with no reported output usage (output 0) books nothing.
- The tier comes from `SavingsLedger.select_tier(level)`, the function
`best_estimate` now delegates to. Lifetime dollars therefore always come
from the tier that the dashboard's percentage comes from:
- **measured:** the request's own stratum's control mean minus its
output, with low = high. It applies only if that stratum qualifies at
the cluster gate the measurement used. A stratum outside the measurement
books 0, because the measured headline excludes it too.
- **estimated:** the request's stratum baseline, with low = high.
- **modelled:** the factor band, `observed × r/(1−r)` at each end.
- `SavingsTracker`:
- Books the low end everywhere it booked before (history, rollups,
OTel).
- Adds `lifetime.output_tokens_saved_high` and
`output_savings_usd_high`, priced per model at output rates.
- Both state builders write the new keys, so a fresh tracker has them.
- Dashboard: the Output Tokens Saved tile adds "≈ $low–$high saved on
output · low–high tokens, all time", read through a component getter.
OSS is unchanged. It ships an empty factor table, so the modelled tier
never appears.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): clean on the touched files
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ git log --oneline -1
|
||
|
|
d9ae7ad721 |
fix(proxy): stop a "*" origin setting from waiving the WebSocket Origin check (#4846)
## Description The WebSocket routes (`/v1/responses`, `/v1/live`) check the browser `Origin` against `HEADROOM_WS_ORIGINS`, falling back to `HEADROOM_CORS_ORIGINS`. A `*` entry in either made every Origin acceptable. `HEADROOM_CORS_ORIGINS=*` is a documented value for CORS (which pages may *read* a response), so loosening CORS for a remote dashboard also let any page open in the user's browser open these sockets, and the proxy then adds its own credentials upstream. `*` now matches nothing in the Origin check. A list that holds only `*` keeps the loopback default; explicit origins in either variable keep working. CORS itself is unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: `_allowed_ws_origins_from_env` drops `*` entries (empty result means the loopback default); `_is_allowed_websocket_origin` no longer short-circuits to `True` on `*`. `providers/codex/live.py` uses the same helper, so `/v1/live` is covered too. - `tests/test_openai_codex_routing.py`: policy test for `*` via either variable and mixed with an explicit origin, plus a handler test that a foreign Origin is closed with 1008 under `HEADROOM_CORS_ORIGINS=*`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # new tests on origin/main (before the fix) $ pytest tests/test_openai_codex_routing.py -k wildcard FAILED tests/test_openai_codex_routing.py::test_websocket_origin_policy_ignores_wildcard[None-*] FAILED tests/test_openai_codex_routing.py::test_websocket_origin_policy_ignores_wildcard[*-None] FAILED tests/test_openai_codex_routing.py::test_websocket_origin_policy_ignores_wildcard[None-*, https://dash.example.com] FAILED tests/test_openai_codex_routing.py::test_handle_openai_responses_ws_closes_foreign_origin_under_wildcard_cors ======================= 4 failed, 43 deselected in 1.33s ======================= # with the fix $ pytest tests/test_openai_codex_routing.py tests/test_codex_live.py ======================= 57 passed, 7 warnings in 16.31s ======================== $ ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_routing.py All checks passed! $ mypy headroom/proxy/handlers/openai.py --ignore-missing-imports Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, `headroom proxy --host 127.0.0.1 --no-optimize` started with `HEADROOM_CORS_ORIGINS='*'`, one run on origin/main and one on this branch. - Exact command / steps: open `ws://127.0.0.1:<port>/v1/responses` with the `websockets` client, once with `Origin: https://page.example` and once with `Origin: http://127.0.0.1:<port>`. - Observed result: ```text == origin/main (HEADROOM_CORS_ORIGINS=*) Origin=https://page.example -> handshake accepted, socket open Origin=http://127.0.0.1:18941 -> handshake accepted, socket open == this branch (HEADROOM_CORS_ORIGINS=*) Origin=https://page.example -> handshake refused HTTP 403 Origin=http://127.0.0.1:18942 -> handshake accepted, socket open ``` - Not tested: `/v1/live` end to end (same helper, covered by the unit tests' shared code path). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: only for operators who set `*` in `HEADROOM_WS_ORIGINS` or `HEADROOM_CORS_ORIGINS`. Native clients (Codex, SDKs) send no Origin and are unaffected; loopback pages still connect. - Kill switch / disable path: list the browser origins that need the WebSocket routes explicitly in `HEADROOM_WS_ORIGINS`. - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes `HEADROOM_WS_ORIGINS` is not documented anywhere, so there was no doc to update. A companion PR applies the same Origin rule to the HTTP routes; once both land, `*` is ignored there too because they share `_is_allowed_websocket_origin`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4846" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Reported by @JoaoMarcos44. Thank you for your contribution! 🩵 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
bae62bc87b |
fix(compression): bound the JSON-block and truncation-banner scans (#4845)
## Description
Two scans over tool output ran in quadratic time:
1. **Mixed-content JSON detection** (`_has_valid_json_block_with_text`,
default-on through `ContentRouter` → `is_mixed_content`) and the section
splitter try every line that starts with `{` or `[` as a JSON block. A
candidate that never balances walks to the end of the content, so N such
lines cost N²/2 line steps (20,000 lines of `{` took about a minute).
The existing per-line memo removes the character work but not the line
walk.
2. **ast-grep truncation-banner detection** (`_detect_truncation`): the
banner regex has unbounded runs on both sides of the `showing lines A-B
of T total` phrase, so an unclosed banner was rescanned once per phrase.
Fixes:
1. `_JsonBlockScanner` answers every candidate of one pass with exactly
the result of walking it. Candidates are walked directly until they have
walked as many lines as the content has. After that, all candidates are
answered together:
- Every walk is a chain of (line, string-state) nodes with one successor
each, so all walks form one forest, and each node is scanned once.
- With `D(v)` the bracket and brace sums from node `v` to the end of its
chain, a walk from `u` balances at `w` exactly when `D(next(w)) >= D(u)`
in both sums. So each candidate ends just before its nearest ancestor
that satisfies this.
- One depth-first pass with an undoable tree of (brace sum, depth)
stacks, keyed by bracket sum, finds that ancestor in O(log² n). There is
no budget.
2. The banner is found in two linear steps: locate `[truncated: partial
view`, take the span to the next bracket (it must be `]`), then match
the phrase inside that span. Same matches as the old regex (checked
against it on 3,000 random layouts).
Fix adapted from a proposal by HetCreep.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/mixed_content.py`: `_JsonBlockScanner` (direct
walks, then a forest solve over all candidates), used by
`_has_valid_json_block_with_text` and `split_into_sections`.
`_extract_json_block` is unchanged from main.
- `headroom/proxy/interceptors/astgrep.py`: `_TRUNCATION_SIGNATURES` now
holds only the phrase pattern; `_iter_banners` finds the `[truncated:
partial view … ]` span and searches the phrase inside it.
- `tests/test_tool_output_scan_bounds.py`: timing tests for both shapes,
an oracle test against the previous banner regex, and checks that a real
banner and a 5,000-line JSON block inside text are still found. A JSON
object that follows 150 or 20,000 unclosed `{` lines, or a prefix whose
bracket and brace counts each reach zero but never on the same line
(once, and repeated 130 times), still gets its own JSON section, as does
one after 100 or 10,000 pairs of lines ending inside an escaped string.
The solve is checked against `_extract_json_block` on 3,000 random
inputs.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
New tests on `origin/main` code:
```text
FAILED tests/test_tool_output_scan_bounds.py::test_truncation_scan_is_linear_on_unclosed_banner
FAILED tests/test_tool_output_scan_bounds.py::test_json_block_scans_are_bounded_on_unclosed_braces
assert time.perf_counter() - start < 10.0
E assert (63611.161382791 - 63549.279589625) < 10.0
==================== 2 failed, 3 passed in 75.96s (0:01:15) ====================
```
With the fix:
```text
$ pytest -q tests/test_tool_output_scan_bounds.py --durations=3 # (first version; now 10 tests, 10 passed in 0.88s)
0.34s call tests/test_tool_output_scan_bounds.py::test_json_block_scans_are_bounded_on_unclosed_braces
0.02s call tests/test_tool_output_scan_bounds.py::test_truncation_banner_matches_previous_regex
============================== 5 passed in 1.09s ===============================
$ pytest -q tests/test_tool_output_scan_bounds.py tests/test_mixed_content_scan_cache.py tests/test_mixed_content_sections.py tests/test_content_router_mixed_fence_placeholder.py
============================= 300 passed in 5.90s ==============================
$ pytest -q tests/test_tool_result_interceptors.py tests/test_proxy_compression_executor.py tests/test_subprocess_encoding.py tests/test_runtime_env.py tests/test_content_router*.py
============================= 247 passed in 32.58s =============================
$ ruff check / ruff format --check (0.17.3) on changed files: All checks passed
$ mypy headroom/proxy/interceptors/astgrep.py headroom/transforms/mixed_content.py --ignore-missing-imports
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: macOS (Apple silicon), Python 3.12 venv, this branch vs
`origin/main`.
- Exact command / steps: called `_has_valid_json_block_with_text` +
`split_into_sections` on 20,000 lines of `{`, and `_detect_truncation`
on `"[truncated: partial view " + "showing lines 1-10 of 50 total " *
12000`.
- Observed result: JSON scans about 62 s on main, 0.34 s here; banner
scan about 13 s on main, milliseconds here. The existing scan-cache
oracle tests (which compare against the original extractor) still pass.
- Not tested: end-to-end through the proxy; both functions are pure and
covered directly.
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no change in results (exact); only
time and memory bounds change
- Kill switch / disable path: none needed
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Timings with the fix: 20,000 unclosed `{` lines take 0.13 s, the
crossed-count prefix repeated 130 times takes 0.23 s, 20,000 alternating
`{` / `}[` lines take 0.29 s, and 10,000 escaped-string line pairs take
0.14 s. With direct walking, the first takes about 62 s. Separately, on
main a balanced block nested about 10,000 deep makes `json.loads` raise
`RecursionError` in the gate; that is unchanged here.
Note: `tests/test_transforms_content_router.py` has 3 failures
(`test_json_in_html_preserves_records_and_markup`,
`test_embedded_json_mixed_tool_output_preserves_every_record[minify-*]`)
that also fail on unmodified `origin/main` in my environment, so they
are not caused by this change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reported by @JoaoMarcos44. Thank you for your contribution! 🩵
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
f5c4217648 |
fix(proxy): enforce HEADROOM_PROXY_TOKEN in the Rust front and listen on loopback by default (#4840)
## Description
The Rust front (`crates/headroom-proxy`) relays every request to the
Python proxy from `127.0.0.1`. The Python proxy treats a loopback caller
as local: it skips the `HEADROOM_PROXY_TOKEN` check and serves `/admin`,
`/debug` and `/settings` to it. The front had no inbound check of its
own, listened on `0.0.0.0:8787` by default, and dropped
`x-headroom-proxy-token` (as an `x-headroom-*` header), so a token set
on the Python side neither protected the stack nor could be presented
through the front.
This PR makes the Rust front apply the same token model as the Python
proxy (`read_proxy_token` / `scrub_proxy_token_headers` in
`headroom/proxy/server.py`, `headroom/proxy/bind_policy.py`) before
anything is forwarded, and makes its default listen address loopback.
Fix adapted from a proposal by HetCreep.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `crates/headroom-proxy/src/inbound_auth.rs`, installed as a
router-wide layer (all routes, the catch-all and WebSocket upgrades)
when `HEADROOM_PROXY_TOKEN` / `--proxy-token` is set:
- a non-loopback peer (or one with no peer address) must send the token
as `X-Headroom-Proxy-Token` or `Authorization: Bearer`; the explicit
header wins, as in Python. Constant-time compare. Otherwise `401
{"error":"unauthorized"}` and no upstream contact; a
`proxy_auth_rejected` warning is logged (no credential values).
- loopback peers are exempt (same as Python); `GET /healthz`,
`/healthz/upstream`, `/health`, `/livez`, `/readyz` stay open;
`GET`/`HEAD /api/hello` (also under `/p/<project>`) is answered locally
with an empty 200, as Python does, instead of being relayed.
- the token is removed before forwarding: `x-headroom-proxy-token`
always, `Authorization` only when it carries the token (a provider
credential is left alone).
- the token is redacted from `Debug` output of `Config`.
- `main.rs`: startup bind policy, same rule and variable as Python: a
non-loopback `--listen` with no token exits with code 2 and a message
naming both remedies, unless `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`
(then a warning is logged).
- `config.rs`: `--listen` / `HEADROOM_PROXY_LISTEN` default changes from
`0.0.0.0:8787` to `127.0.0.1:8787`; new `--proxy-token` (env
`HEADROOM_PROXY_TOKEN`, value hidden from `--help`).
- `RUST_DEV.md`: runbook exports a token for the `0.0.0.0` cutover, flag
table updated, new "Inbound authentication" section.
**Behaviour change for operators:** a Rust front started with `--listen
0.0.0.0:...` (or relying on the old default to be reachable from other
hosts) now needs `HEADROOM_PROXY_TOKEN` set, and clients on other hosts
must send it. Setups that front the port with something else that
restricts access can set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`.
Loopback clients (Claude Code, Codex, SDKs on the same host) are
unaffected. The Docker image's entrypoint is the Python proxy, so it is
unaffected.
## Testing
- [x] `cargo test -p headroom-proxy` (46 test binaries, 608 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes
New tests:
- `tests/integration_inbound_auth.rs` (11 tests, `oneshot` with an
injected `ConnectInfo` so a non-loopback peer can be simulated,
`wiremock` upstream): missing/wrong token is 401 and the upstream sees
nothing; correct token via header or Bearer is forwarded without the
token; provider `Authorization` kept when the header carries the token;
loopback and IPv4-mapped loopback peers pass; missing peer address fails
closed; health probes open but `POST /readyz` is 401; `/api/hello`
answered locally; no token configured keeps passthrough.
- `tests/integration_bind_policy.rs` (3 tests, real binary): `0.0.0.0`
with no token refuses to start; loopback starts; default `--listen` is
loopback.
- unit tests in `inbound_auth.rs` for token parsing, constant-time
compare, bind policy and exemptions.
### Test Output
Before (the bind-policy test file copied onto a clean `origin/main`
worktree; `integration_inbound_auth.rs` does not compile there since
main has no token setting at all):
```text
test default_listen_address_is_loopback ... FAILED
test loopback_listen_without_token_still_starts ... ok
test non_loopback_listen_without_token_refuses_to_start ... FAILED
test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20.01s
```
After:
```text
$ cargo test -p headroom-proxy --test integration_inbound_auth --test integration_bind_policy
test default_listen_address_is_loopback ... ok
test non_loopback_listen_without_token_refuses_to_start ... ok
test loopback_listen_without_token_still_starts ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.01s
test missing_peer_address_fails_closed ... ok
test remote_peer_with_bearer_token_is_forwarded_without_it ... ok
test remote_peer_without_token_gets_401_and_upstream_is_not_contacted ... ok
test token_header_wins_over_a_provider_bearer ... ok
test no_token_configured_keeps_todays_passthrough ... ok
test loopback_peer_without_token_is_still_forwarded ... ok
test ipv4_mapped_loopback_peer_counts_as_loopback ... ok
test remote_peer_with_token_header_is_forwarded_without_it ... ok
test hello_probe_is_answered_locally_without_token ... ok
test remote_peer_with_wrong_token_gets_401 ... ok
test health_probes_stay_reachable_without_token ... ok
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.74s
```
## Real Behavior Proof
- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (
|
||
|
|
550644f57a | Merge branch 'main' into fix/plugin-launcher-isolated-python | ||
|
|
cfdeb1c466 |
fix(learn): render non-string Bash commands without raising (#4826)
## Description
Fixes a crash in `headroom learn` when a scanned session stores a
non-string Bash `command`.
`ToolCall.input_summary` sliced `input_data["command"]` directly,
assuming a string. A real opencode v2 database contained eight Bash
calls with `{"command": true}`, which raised:
```
TypeError: object of type 'bool' has no len()
```
Both the digest formatter (`analyzer._format_tool_call`) and the loop
sampler (`loops.detect_loops`) read `input_summary`, so a single
malformed call aborted the entire `headroom learn --all --apply` run.
No linked issue: this was found while validating the opencode v2 scanner
change in #4199 (now merged). The crash is in shared `ToolCall` code, so
the fix is submitted separately.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `ToolCall.input_summary` now coerces a non-string `command` with
`str()` instead of slicing the raw value; `None` renders as an empty
string.
- `input_summary` treats a non-mapping `input_data` as `{}` before
calling `.get()`, matching the defensive shape already used in
`loops._identity_input`.
- Added `TestInputSummaryToleratesMalformedInput` with six cases:
bool/int/`None`/list commands, a non-mapping input, and an end-to-end
`detect_loops` + `_build_digest` case.
- No behavior change for valid string commands; the other summary
branches already use `str()`/f-strings.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] Type checking passes (`mypy headroom`) — not run; the touched
module was not type-checked as part of this change.
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_learn/test_loop_weighting.py -k TestInputSummaryToleratesMalformedInput -q
6 passed, 89 deselected, 1 warning in 0.78s
$ .venv/bin/python -m pytest tests/test_learn/ tests/test_cli_learn.py -q
455 passed, 7 skipped, 2 warnings in 32.88s
$ uvx ruff@0.16.8 check headroom/learn/models.py tests/test_learn/test_loop_weighting.py
All checks passed!
$ uvx ruff@0.16.8 format --check headroom/learn/models.py tests/test_learn/test_loop_weighting.py
2 files already formatted
```
All six new tests fail against the unpatched `models.py`.
## Real Behavior Proof
- Environment: Linux x86_64; Python 3.14.8; `uv sync --frozen`; PR head
`22e6a6f8fb05f79dcc82ea57a24bfbd6615305b8`. No provider credentials,
network, or private data required.
- Exact command / steps: save the script below as
`proof_malformed_opencode.py` at the repo root, then run
`.venv/bin/python proof_malformed_opencode.py`. It creates a temporary
opencode v2 SQLite database (`project` / `session_v2` /
`session_message`) containing three Bash calls whose input is
`{"command": true}`, then runs `OpenCodePlugin.scan_project` →
`detect_loops` → `_build_digest`.
- Observed result: with this PR, `sessions: 1 calls: 3`, `summaries:
['True', 'True', 'True']`, `loops: [('Bash', 3, 'True')]`, `digest
chars: 684`, `digest contains malformed call: True`. With
`headroom/learn/models.py` reverted to `origin/main`, the same script
exits with `TypeError: object of type 'bool' has no len()` at
`models.py:67` (`return cmd[:100] + "..." if len(cmd) > 100 else cmd`).
The original private database scan (28 sessions / 7,882 calls, 8
malformed commands, 29 loops, 319,885-char digest) is additional
evidence but is not reproducible outside that machine.
- Not tested: a real LLM analysis call (requires provider credentials),
Windows path handling, other agents' databases, and `mypy`. The scanner
and analyzer logic are unchanged; only the shared display property is
guarded.
<details>
<summary>proof_malformed_opencode.py</summary>
```python
"""Reproduce the malformed-command crash with a synthetic opencode v2 DB."""
import json
import sqlite3
import tempfile
from pathlib import Path
from headroom.learn.analyzer import _build_digest
from headroom.learn.loops import detect_loops
from headroom.learn.plugins.opencode import OpenCodePlugin
with tempfile.TemporaryDirectory() as tmp:
tmp_path = Path(tmp)
project_path = tmp_path / "project"
project_path.mkdir()
db_path = tmp_path / "opencode.db"
conn = sqlite3.connect(db_path)
conn.executescript(
"""
CREATE TABLE project (id TEXT PRIMARY KEY, worktree TEXT NOT NULL, name TEXT);
CREATE TABLE session_v2 (
id TEXT PRIMARY KEY, project_id TEXT NOT NULL, time_created INTEGER NOT NULL
);
CREATE TABLE session_message (
id TEXT PRIMARY KEY, session_id TEXT NOT NULL, type TEXT NOT NULL,
seq INTEGER NOT NULL, time_created INTEGER NOT NULL,
time_updated INTEGER NOT NULL, data TEXT NOT NULL
);
"""
)
conn.execute("INSERT INTO project VALUES (?, ?, ?)", ("p", str(project_path), "proof"))
conn.execute("INSERT INTO session_v2 VALUES (?, ?, ?)", ("s", "p", 1))
content = [
{
"type": "tool",
"id": f"call-{i}",
"name": "bash",
"state": {
"status": "completed",
"input": {"command": True},
"content": [{"type": "text", "text": "exit code 0"}],
},
}
for i in range(3)
]
conn.execute(
"INSERT INTO session_message VALUES (?, ?, ?, ?, ?, ?, ?)",
(
"m",
"s",
"assistant",
1,
1,
1,
json.dumps({"content": content}, separators=(",", ":")),
),
)
conn.commit()
conn.close()
plugin = OpenCodePlugin(db_path=db_path)
project = plugin.discover_projects()[0]
sessions = plugin.scan_project(project)
print("sessions:", len(sessions), "calls:", sum(len(s.tool_calls) for s in sessions))
print("summaries:", [c.input_summary for c in sessions[0].tool_calls])
loops = detect_loops(sessions)
print("loops:", [(lp.tool, lp.count, lp.sample_input) for lp in loops])
digest = _build_digest(project, sessions, loops=loops)
print("digest chars:", len(digest))
print("digest contains malformed call:", "Bash: True" in digest)
```
</details>
## Runtime Rollout Safety
- Rollout-managed feature(s): `headroom learn` local session scanning
and digest formatting only; no proxy, provider, or runtime request path.
- Minimum rollout channel: N/A — library bug fix, not gated by a rollout
channel.
- Stable/default behavior changed: No. Valid string commands render
exactly as before; only malformed inputs that previously raised now
render.
- Kill switch / disable path: N/A — revert the commit to restore the
prior behavior.
- Unsafe override required: No.
- Qualification impact: None; no network, model, or rollout behavior is
touched.
- Rollback path: revert this commit/PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation — N/A, no
public API or docs change
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — no UI change.
## Additional Notes
The malformed value is real session data, not a synthetic edge case:
session `ses_e6bcaf904fd4blosS7bPwIvQ1f` in a local opencode v2 DB
stores `{"command": true}` for 8 Bash calls. The fix only affects
display/summary rendering, so loop identity, failure classification, and
recommendations are unchanged.
Co-authored-by: JavierLeon9966 <javier@lumineproxy.org>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
93331f2874 |
fix(proxy): retain final backend streaming usage (#4814)
## Description
Translated OpenAI streams (`/v1/chat/completions` through a backend such
as `anyllm` or LiteLLM) recorded the **first nonzero** usage snapshot
instead of the last one.
`_absorb` in `_stream_openai_via_backend` wrote a counter only when the
accumulator was still falsy:
```python
if key in usage and not stream_state.get(key):
stream_state[key] = usage[key]
```
A provider that reports climbing cumulative snapshots on successive
content chunks therefore had every later frame thrown away. Replaying
the two frames from the issue through the real proxy path, `/stats`
recorded 10 input / 1 output tokens and `/metrics` recorded 2 cache
reads — while the client received both frames intact. The bug is in
accounting after delivery, not a dropped upstream event.
Closes #4263
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `_parse_sse_usage_from_buffer` (openai branch): absent counters no
longer default to `0`. A frame that omits a field was previously
indistinguishable from one reporting an explicit zero, which is a
prerequisite for "absent keeps the last value".
- `_absorb`: assigns unconditionally, so the latest cumulative snapshot
wins. An absent field keeps its earlier value; an explicit zero still
replaces it.
- Removed `_parse_completion_tokens_from_sse_chunk`. It existed to catch
completion-only frames, but the buffered SSE parser already handles
those, and its `not stream_state["output_tokens"]` guard latched the
output count a second time.
- Replaced that helper's unit tests with eight regressions driving the
real `_stream_openai_via_backend` path through stream finalization, in
Chat and Responses shapes.
- `docs/content/docs/proxy.mdx`: one paragraph on the `POST
/v1/chat/completions` page documenting the accounting rule, in the
published docs site.
Net: -100/+113 lines, of which most deletions are the removed helper and
its implementation-only tests.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_streaming_usage_parser.py -q
........ [100%]
8 passed in 1.01s
$ ruff check headroom/proxy/handlers/streaming.py tests/test_streaming_usage_parser.py
All checks passed!
$ ruff format --check headroom/proxy/handlers/streaming.py tests/test_streaming_usage_parser.py
2 files already formatted
$ mypy headroom/proxy/handlers/streaming.py
Success: no issues found in 1 source file
```
Regression set — the new tests against the unfixed handler (`git stash`
of `streaming.py` only):
```text
$ python -m pytest tests/test_streaming_usage_parser.py -q
FAILED ...[latest-snapshot-chat] assert (10, 1, 2, 3) == (13, 5, 9, 1)
FAILED ...[latest-snapshot-responses]
FAILED ...[partial-chat] assert (10, 1, 2, 3) == (10, 5, 2, 3)
FAILED ...[partial-responses]
FAILED ...[explicit-zero-chat] assert (10, 1, 2, 3) == (0, 0, 0, 0)
FAILED ...[explicit-zero-responses]
6 failed, 2 passed in 1.07s
```
Six fail before the fix, all eight pass after — matching the issue
exactly.
Wider streaming and parser suites:
```text
$ python -m pytest tests/test_streaming_usage_parser.py tests/test_streaming_sse_malformed_events.py \
tests/test_provider_billed_input.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_handler_helpers.py \
tests/test_proxy_streaming_request_logger.py tests/test_route_advice.py tests/test_h2_stream_reset_retry.py \
tests/test_proxy_streaming_resilience.py tests/test_proxy_streaming_ratelimit_headers.py -q
3 failed, 607 passed in 10.21s
```
The three failures are
`tests/test_proxy_cache_ttl_metrics.py::test_stats_endpoint_reports_*`,
which raise `SystemExit: 78` from `_check_rust_core()` in `server.py`.
They are environmental (unbuilt Rust core in this checkout) and fail
identically on unmodified `upstream/main` with this branch stashed —
unrelated to this change.
## Real Behavior Proof
- Environment: Windows, Python 3.13.11, this branch, `PYTHONPATH` set to
the checkout.
- Exact command / steps: replayed the issue's two usage snapshots as a
Chat stream through `_stream_openai_via_backend` with a stub backend,
then read the resulting `RequestOutcome`.
- Observed result: the corrected counts from the issue; previously 10 /
1 / 2.
```text
provider_input_tokens=13 (expect 13)
output_tokens =5 (expect 5)
cache_read_tokens =9 (expect 9)
```
- Not tested: a live `anyllm`/LiteLLM upstream over real HTTP (no
credentials in this environment). The replay drives the identical
handler and parser path the proxy uses; the frames are byte-for-byte the
issue's.
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: token accounting for translated
OpenAI streams now records the final snapshot. This is the fix; counts
move up from truncated to complete.
- Kill switch / disable path: none beyond reverting this commit.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: `git revert`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Authorship is credited to @oxura, who diagnosed this in #4263 and wrote
the original patch on their fork; they were holding it back on the
ten-open-PR limit. This PR carries their change rebased onto current
`upstream/main` (their branch was cut from `fbf7bc32`, and
`streaming.py` has since drifted ~200 lines), with the commit message
and tests adapted to the current tree. Their `Authored-by` trailer is on
the commit.
One hunk from their patch was redirected: they documented the rule in
`wiki/proxy.md`, which `AGENTS.md:38` says is unpublished and must not
receive new content, so the paragraph now lives on the `POST
/v1/chat/completions` page in `docs/content/docs/proxy.mdx`. Thanks to
Devin Review for flagging it.
Co-authored-by: oxura <grozafroza44@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
199912e020 |
test(cli): assert the --learn/--no-learn warning text (#4800)
## Description `TestLearnNoLearnConflict::test_both_flags_warns_and_exits_zero` ends with `assert "both --learn and --no-learn" in result.output or (result.output is not None), ...`. The right side of the `or` is true for any run (`result.output` is a string), so the assertion cannot fail and the test passes without the warning. This PR drops the `or` so the test checks the warning text it is named for. Part of #4783 (found in a second pass; listed there as a test that cannot fail). It is a one-line test change, no product code. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) - [x] Test-only change ## Changes Made - `tests/test_cli_proxy_improvements.py`: the warning assertion is now `assert "both --learn and --no-learn" in result.output, result.output`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `ruff check` and `ruff format --check` (0.17.3) are clean on the file. ### Test Output ```text tests/test_cli_proxy_improvements.py -k TestLearnNoLearnConflict this branch: 4 passed mutation: headroom/cli/proxy.py `if learn and no_learn:` -> `if False and learn and no_learn:` test_both_flags_warns_and_exits_zero, this branch: FAILED test_both_flags_warns_and_exits_zero, main: passed ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, click 8.5.1, pytest 9.2.0, `main` at `3c66c91e`. - Exact command / steps: ran the four tests of `TestLearnNoLearnConflict`; then disabled the warning branch in `headroom/cli/proxy.py` locally and ran `test_both_flags_warns_and_exits_zero` with this change and with `main`'s test (the product file was restored afterwards). - Observed result: as in Test Output. `main`'s assertion passes with the warning removed; this change fails. - Not tested: other click versions (the assertion relies on `result.output` containing stderr, which is how click 8.5.1's `CliRunner` behaves), the whole suite, CI. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a - Stable/default behavior changed: no (test only). - Kill switch / disable path: n/a - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes (the four tests of the class; the whole suite was not run) - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) AI assistance: this change and the runs above were done with Claude (Sonnet 5.5), operated by me (HetCreep). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4800" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
d379efe90c |
test: skip tests marked live by default (#4786)
## Description
`pyproject.toml` declares a `live` marker ("opt-in multi-turn tests that
hit real upstream APIs; require provider keys"), and two test files
carry it: `tests/test_thinking_signature_scope_live.py` ("Opt-in:
requires a real key and is gated behind `pytest.mark.live`. Run with
`pytest -m live ...`") and `tests/test_realignment_live_multi_turn.py`
(which describes a "default `pytest -m "not live"` run that never
touches the network"). `addopts` was `-v --tb=short` with no `-m`
filter, so a plain `pytest` run on a machine with `ANTHROPIC_API_KEY` in
the environment, or in the project's `.env` (loaded by
`tests/_dotenv.py`), collected these tests and called the real API.
This PR adds `-m 'not live'` to `addopts`, so the default run deselects
them. `pytest -m live <file>` still selects them (a later `-m` replaces
this one).
Part of #4783
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `pyproject.toml`: `addopts = "-v --tb=short -m 'not live'"`, with a
three-line comment saying why and how to opt in.
- `tests/test_realignment_live_multi_turn.py`: the module-wide `live`
marker is replaced by `@pytest.mark.live` on the eight tests that need a
provider key, so the offline
`test_classify_auth_mode_routes_payg_vs_oauth` keeps running by default.
- `tests/test_live_marker_default_deselect.py`: collection-only test
that the default run deselects live tests, keeps the offline one, and
that `-m live` selects them again (fails against the old `addopts`).
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
This changes pytest configuration only, so no new test is added; the
proof is the collection result below (`--collect-only`, with a dummy
`ANTHROPIC_API_KEY`, nothing contacted).
### Test Output
```text
Before (origin/main pyproject.toml):
pytest tests/test_thinking_signature_scope_live.py tests/test_realignment_live_multi_turn.py --collect-only -q
15 tests collected
After (this branch):
same command -> 1 test collected (the offline test_classify_auth_mode_routes_payg_vs_oauth), 14 deselected
same command with -m live -> 14 tests collected
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, pytest 9.2.0,
`ANTHROPIC_API_KEY=sk-ant-dummy-collect-only` in the environment.
- Exact command / steps: `python -m pytest
tests/test_thinking_signature_scope_live.py
tests/test_realignment_live_multi_turn.py --collect-only -q`, with the
pyproject from `origin/main` and with this branch's, and once more with
`-m live`.
- Observed result: before, 15 tests collected (they would run and call
the API when a key is present); after, the 14 provider-calling tests are
deselected and the offline one is still collected; with `-m live`, 14
collected.
- Not tested: an actual live run (no provider key was used), and CI (the
CI workflows I read do not export a provider key to pytest, so this PR
is aimed at developer machines).
## Runtime Rollout Safety
- Rollout-managed feature(s): none; test configuration only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for the library and proxy. The
default `pytest` run no longer collects tests marked `live`.
- Kill switch / disable path: run `pytest -m live ...` (or pass any `-m`
expression).
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes (the
whole suite was not run; see Testing)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- If a CI job runs `pytest -m live` deliberately it keeps working,
because the later `-m` wins. I did not find such a job.
AI assistance: this change and the runs above were done with Claude
(Sonnet 5.5), operated by me (HetCreep).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4786"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
6665940656 |
fix(memory): return stored id from a background DirectMem0 save (#4759)
## Description `DirectMem0Adapter.save_memory(background=True)` returned an id from a second `_generate_id` call. That helper salts with the current time, so the returned id never matched the Qdrant point the background task wrote, and a later `memory_update` / `memory_delete` with it found nothing. The direct-write ids are now generated once, passed to the background task, and returned. On the Mem0 extraction path (no pre-extracted data), Mem0 assigns ids itself, so no id is known up front. The save is marked `_id_pending`, and `memory_save` reports `memory_id: None` instead of an invented id. The real id is available from `get_task_status(task_id)`. Refs #4362 (item 6 only). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/memory/backends/direct_mem0.py`: generate fact ids before scheduling the background task, then store and return those same ids. `_write_facts_to_qdrant` accepts optional `memory_ids`. Mem0-path background saves are marked `_id_pending`. Fixed the `_generate_id` docstring, which said the ids were "deterministic". - `headroom/memory/system.py`: `memory_save` returns `memory_id: None` when the id is pending. - Background saves snapshot `facts` before generating ids and schedule that same snapshot, so a caller mutating its list after queueing can't make the task fail. The returned `_all_memory_ids` is a copy for the same reason. - `tests/test_memory/test_direct_mem0_background_ids.py`: 4 regression tests, using a fake Qdrant client and a fake Mem0 client. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_memory/test_direct_mem0_background_ids.py -v test_background_save_returns_stored_fact_id PASSED test_mutating_lists_after_queueing_does_not_break_the_save PASSED test_background_mem0_extraction_does_not_invent_an_id PASSED test_memory_save_tool_withholds_a_pending_id PASSED 4 passed in 0.36s ``` ## Real Behavior Proof - Environment: macOS; Qdrant v1.15.5 server in Docker (`localhost:6333`); `mem0ai` 2.2.1 and `qdrant-client` 1.20.0 (real `Memory.get/update/delete`); embeddings from Ollama 0.40.6 `nomic-embed-text` through its OpenAI-compatible API (`OPENAI_BASE_URL=http://localhost:11434/v1`). `DirectMem0Adapter` ran unmodified, with `enable_graph=False`. - Exact command / steps: a script on a fresh 768-dim collection: `save_memory(facts=[...], background=True)` → append to the caller's `facts` list → `flush_pending()` → `get_memory` / `update_memory` / `get_memory` / `delete_memory` / `get_memory`, all using the **returned** id. Run once on this branch and once on `main` (`a385ea8e`) as a control. - Observed result: Returned IDs resolve to the stored points; before/after results are below. | | this branch | `main` | |---|---|---| | task status | completed | completed | | returned id is a stored Qdrant point | **True** | False | | `get_memory(returned id)` | `'Prefers Python over JavaScript'` | `None` | | `update_memory` → `get_memory` | `'Prefers Rust over Python'` | `Memory with id 0ad08f… not found` | | `delete_memory(returned id)` | `True`, 0 points left | `False`, 2 points left | | fact appended after queueing | not stored (snapshot) | stored under an id never returned to the caller | - Not tested: the Mem0 LLM-extraction background path (`_id_pending`; it needs an OpenAI LLM and none was available); Neo4j graph writes; hosted Qdrant; OpenAI's own embedding API. ## Runtime Rollout Safety - Rollout-managed feature(s): None - Minimum rollout channel: NA - Stable/default behavior changed: Only background saves on `DirectMem0Adapter` (opt-in via `background=true` or `async_writes=True`). The proxy doesn't use background saves. Synchronous saves are unchanged. - Kill switch / disable path: Don't use background saves. - Unsafe override required: NA - Qualification impact: None - Rollback path: Revert this PR ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4759" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
6b41b6e451 |
feat(proxy): add cli and settings knob for ccr_inject_system_instructions (#4642)
<!-- maintainer-current-main --> ## Maintainer latest-main checkpoint Published `d13ba17dff65a5b31253ef3cf0775ffc2504d8da` contains current upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 150 CLI/settings/Gemini tests passed. Earlier live behavior proofs below retain their commit/platform scope; contributor runtime logic is unchanged. The common shard-2 failure was independently confirmed on main's hosted job 127069149772 and locally (four failures, six passes): the Gemini memory fixture mocked an unused body reader and provided no request stream. The maintainer test-only follow-up now constructs a real FastAPI/ASGI Request carrying JSON bytes and exercises the actual body reader. All six test-function ASTs/assertions are unchanged, and all 10 parameterized Gemini cases pass. Touched-file Ruff/format and whitespace checks pass. This fixes the inherited fixture instead of asking contributors to run CI or weakening the memory/content assertions. Fresh exact-head hosted qualification is requested. Nothing merged. <!-- /maintainer-current-main --> ## Description Closes #4636. `ccr_inject_system_instructions` (which instructs models how to retrieve compressed context when CCR markers are present) previously existed as a field on `ProxyConfig` but could only be toggled programmatically via the Python API. Operators deploying `headroom proxy` as a container or standalone service had no CLI flag, environment variable, or settings store entry to enable it without wrapping the process in custom Python code. This PR exposes `ccr_inject_system_instructions` as: 1. CLI flag: `--ccr-inject-system-instructions` 2. Environment variable: `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS` 3. Curated settings store field: `ccr_inject_system_instructions` in `settings_store.SETTINGS` 4. Server environment configuration: loaded in `_proxy_config_from_env()` The default remains `False` preserving exact existing behavior. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `headroom/cli/proxy.py`: Add `@click.option("--ccr-inject-system-instructions", is_flag=True, envvar="HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS", ...)`, add parameter to `proxy()`, and thread it into `ProxyConfig`. - `headroom/settings_store.py`: Add `SettingField("HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS", "ccr_inject_system_instructions", ...)` to `SETTINGS` registry so it is validated, saved to `settings.json`, and surfaced on `/settings`. - `headroom/proxy/server.py`: Read `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS` in `_proxy_config_from_env()` for multi-worker and container deployments. - `docs/content/docs/proxy.mdx`: Document `--ccr-inject-system-instructions` and `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS` in the CCR configuration table. - `tests/test_cli_proxy_env.py`: Add test cases verifying `--ccr-inject-system-instructions`, `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS`, `settings.json`, and `_proxy_config_from_env()` correctly populate `ProxyConfig.ccr_inject_system_instructions`. ## Testing - [x] Unit tests pass (`pytest tests/test_proxy/test_settings_store.py tests/test_cli_proxy_env.py`) - [x] Linting passes (`ruff check`) - [x] Formatting passes (`ruff format --check`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy/test_settings_store.py -q 40 passed, 1 warning in 2.18s $ pytest tests/test_cli_proxy_env.py -k "test_ccr_inject_system_instructions or test_settings_file_reaches_ccr_inject_system_instructions or test_proxy_config_from_env_reads_ccr_inject_system_instructions" -q 5 passed, 91 deselected in 3.42s $ ruff check headroom/cli/proxy.py headroom/settings_store.py headroom/proxy/server.py tests/test_cli_proxy_env.py All checks passed! $ ruff format --check headroom/cli/proxy.py headroom/settings_store.py headroom/proxy/server.py tests/test_cli_proxy_env.py 4 files already formatted ``` ## Real Behavior Proof - Environment: Windows 11 x86_64, Python 3.14.5, Click CliRunner test harness and settings store test harness. - Exact command / steps: Executed `runner.invoke(main, ["proxy", "--ccr-inject-system-instructions"])`, verified captured ProxyConfig; tested env var `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS=1`, verified ProxyConfig; tested `settings.json` with `{"ccr_inject_system_instructions": true}`, verified ProxyConfig; tested `_proxy_config_from_env()`. - Observed result: In all test cases, ProxyConfig.ccr_inject_system_instructions resolved to True when enabled and False by default, matching the behavior of sibling knobs like HEADROOM_CCR_INLINE_RESOLVE and HEADROOM_CCR_BUFFERED_CHAT_STREAMING. - Not tested: Live Kubernetes deployment or multi-node container cluster. ## Runtime Rollout Safety - Rollout-managed feature(s): none, opt-in configuration knob - Minimum rollout channel: n/a - Stable/default behavior changed: no, default remains False - Kill switch / disable path: omit the flag or unset HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS - Unsafe override required: no - Qualification impact: none while disabled - Rollback path: git revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Follows the exact pattern established by `--ccr-inline-resolve` and `--ccr-buffered-chat-streaming` (#4631). <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4642" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ## Maintainer runtime and integration qualification Published `9c0b0a10d63227554350a3ba2cb936e09cea1286` includes upstream main `ee975d49c0b742e34b09df50a01c7b5680e85cdc` with a clean merge and no contributor logic changes. Main is actively advancing, so final qualification must check the then-current main again. Windows/Python 3.12.13: `python -m pytest tests/test_cli_proxy_env.py tests/test_proxy/test_settings_store.py -q` reports **140 passed** with the existing development native extension. Touched-file Ruff, format and whitespace checks pass; no fresh native build claimed. Real CLI proof (no replaced run_server or provider transport): launch the actual Click entry point `from headroom.cli import main; main()` in six isolated subprocesses with `proxy --mode token --no-optimize --no-cache --no-rate-limit --no-ccr-proactive-expansion --no-memory-tools --no-memory-context --disable-kompress --anthropic-api-url http://127.0.0.1:<fixture-port>`. Each process has an isolated workspace/settings file, offline/beacon opt-out and a real CompressionStore entry created before launch; GET `/v1/retrieve/{hash}` confirms ownership. Send `/v1/messages` with that stored hash in a historical tool-result marker, through the actual TCP proxy to a local HTTP Anthropic-shaped provider fixture, and inspect the received upstream JSON. Observed: all six calls return HTTP 200. Default: no `Compressed Context Available` instructions. CLI flag, environment `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS=1`, and a settings file with `{"ccr_inject_system_instructions":true}` each produce the instructions upstream. `--no-ccr` and `--lossless`, both requested alongside the flag, produce neither instructions nor a retrieval tool despite the owned historical marker. This verifies the disabled-CCR finding is addressed on the real request path. The documented warm-prefix/cache-mode skip remains an intentional limitation; this run uses token mode. Not tested: live cloud models, Kubernetes/multiple nodes, every provider wire shape, warm-cache replay or a fresh native build. The local HTTP fixture is synthetic; request/configuration/ownership paths and TCP transport are real. Hosted CI must finish on the final head; nothing merged. ### Maintainer current-main requalification Exact head `b38e5a5937cf09c3b5127f3f066e0384db235955`, upstream main `ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`. Clean integration with upstream/main |
||
|
|
e9301f34c1 |
feat(tray): opt-in system tray icon with savings counter, dashboard and update alert (#4726)
## Description
Adds an **opt-in system tray icon** for Headroom while it runs (Windows
first; macOS/Linux planned). The icon is the Headroom logomark and its
right-click menu shows a compact savings counter (tokens and estimated $
saved), opens the dashboard, alerts when an update is available, and
lets you quit.
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `headroom/tray/` package: `core.py` (GUI-free stats
polling/formatting), `icon.py` (logomark drawn with Pillow, no binary
assets in the wheel), `app.py` (pystray app), `launcher.py` (spawns the
tray as a child of the proxy).
- `headroom proxy --tray` starts the tray with the proxy; `headroom tray
--port N` attaches to a running proxy.
- Menu: live `▼ tokens · ~$ saved` counter (from `/stats` →
`persistent_savings.lifetime`, polled every 5s), Open Dashboard, "Update
available" (only when newer; reuses `headroom.update_check`, honours
`HEADROOM_UPDATE_CHECK=off`; one-time notification), Quit.
- Icon states: normal, orange dot when an update exists, grey when the
proxy is offline.
- New optional extra `headroom-ai[tray]` (`pystray`, `pillow`).
Unsupported platform / missing extra: `--tray` prints why and the proxy
starts normally.
- Docs: `docs/content/docs/system-tray.mdx` + nav entry; screenshots in
`docs/public/assets/tray/`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
> python -m ruff check headroom\tray headroom\cli tests\test_tray_core.py
All checks passed!
> python -m mypy headroom\tray headroom\cli\tray.py
Success: no issues found in 7 source files
> python -m pytest tests\test_tray_core.py tests\cli tests\test_offline_egress_chokepoint.py tests\test_proxy\test_settings_store.py -q
293 passed, 2 skipped
```
(Scoped runs; I did not run the full suite or `mypy headroom` over the
whole package.)
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.13, 3840x2160 display, `pystray`
+ `pillow` 12.3.0, branch based on `upstream/main` @
|
||
|
|
66e0086952 |
feat(opencode): add repeatable --extra-model flag for custom wrap-time models (#2364)
## Description
`headroom wrap opencode` injects a hardcoded model list
(`HEADROOM_OPENCODE_MODELS`) into the `headroom` provider block of
`opencode.json`. There's no way to add custom models (for example a
free-tier DeepSeek) at wrap time. Hand-patching `opencode.json` is the
only workaround, and the next wrap overwrites it.
This PR adds a repeatable `--extra-model` flag and a
`HEADROOM_OPENCODE_EXTRA_MODELS` env var. Both take
`id[=name[=context_window[=output_limit]]]` specs. Extra models are
merged into the injected provider block alongside the built-ins, and
flag specs win over env specs on id collision. `name` defaults to the
id, `context_window` to 200000, and `output_limit` to `min(16384,
context_window)`.
Fields are separated by `=` rather than `:` because model ids routinely
contain `:`. Ollama ids are `name:tag` (`qwen2.5-coder:7b`), and
OpenRouter free-tier ids end in `:free`. With a `:` separator those were
silently cut down to a different id (Devin review, 2026-10-03).
The PR also fixes the wrap/unwrap lifecycle the flag depends on. A wrap
that starts without an `opencode.json` used to write the file with no
backup and no record of what Headroom put in it, so `headroom unwrap
opencode` left the injected models and the proxy base URL behind while
reporting that OpenCode was no longer routed through Headroom, and a
re-wrap under `--openai-api-url` kept an extra model Headroom had
injected but no longer advertised. Wrap now records what it owns in
`opencode.json.headroom-managed.json`, and unwrap uses that record: it
removes the model ids and the base URL Headroom wrote, keeps everything
else in the file, drops an extra model that is no longer requested, and
deletes the config file when Headroom created it and nothing else is
left in it.
Closes #2359
## Type of Change
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [x] Documentation update
- [ ] Refactoring (no functional changes)
## Changes Made
- `headroom/providers/opencode/config.py`:
- New `parse_extra_model_spec()`, which splits on `=` with an optional
fourth `output_limit` field and raises `ValueError` on an empty id or a
non-integer / non-positive limit.
- New `extra_models_from_env()`, which reads comma-separated specs from
`HEADROOM_OPENCODE_EXTRA_MODELS`.
- An `extra_models` parameter threaded through
`headroom_provider_entry()`, `_render_provider_block()` and
`inject_opencode_provider_config()`.
- With `keep_user_entries` (#3125's `--openai-api-url` gate),
`--extra-model` entries win over a kept user entry with the same id.
- New managed-state sidecar (`opencode.json.headroom-managed.json`)
recording the model ids and base URL Headroom injected and whether
Headroom created the config file. `inject_opencode_provider_config()`
writes it on every injection; `record_opencode_config_origin()` marks a
config that Headroom creates because none existed.
- `remove_managed_opencode_provider()` drops the recorded model ids, the
recorded base URL, and the provider block once nothing else is left in
it. Entries the user added or edited are preserved.
- `_drop_stale_managed_models()` prunes an extra model Headroom injected
but no longer advertises, including an extra that overrode a built-in id
and is dropped again, so a re-wrap without the flag restores the
built-in defaults rather than keeping the old override.
- `snapshot_opencode_config_if_unwrapped()` no longer snapshots a config
Headroom wrote itself as pre-wrap content.
- `headroom/providers/opencode/runtime.py`: extras also reach the
launch-time `OPENCODE_CONFIG_CONTENT`.
- `headroom/cli/wrap.py`:
- A repeatable `--extra-model` option on `wrap opencode`. Env and flag
specs are parsed up front, and a malformed spec raises
`click.BadParameter`. The merged dict goes to both
`inject_opencode_provider_config` call sites (prepare-only and launch)
and to the launch env. Also adds a help text and docstring example.
- `wrap opencode` records the config origin before the first config
mutation.
- `unwrap opencode` undoes the recorded provider entries when there is
no pre-wrap backup, keeps the rest of the file, and removes a config
Headroom created once nothing Headroom-owned is left in it.
- `docs/content/docs/opencode.mdx`: new "Adding models" subsection
(format, defaults, Ollama/OpenRouter example, built-in id override, the
env var's no-comma limitation) and an env-var table row. The subsection
and the "How It Works Under The Hood" unwrap step now describe the
managed-state record and the automatic cleanup it enables.
- Tests:
- Spec parsing: full spec with output, defaults, the output cap at small
contexts, colon-bearing ids (`qwen2.5-coder:7b`,
`deepseek/deepseek-chat:free`, a name containing `:`), and invalid specs
including a bad output limit.
- Env parsing, provider-entry merge, and end-to-end inject. Also
@JerrettDavis's collision test across both `keep_user_entries` values.
- CLI: the flag reaches `opencode.json`, the env var persists, both
reach `OPENCODE_CONFIG_CONTENT` on launch, the flag wins over a kept
user entry under `--openai-api-url`, and a malformed spec exits with a
usage error.
- Managed state: the sidecar records the injected ids and base URL, a
config the user already had is not marked as Headroom-created, a re-wrap
does not snapshot Headroom's own config, a dropped extra model is pruned
(including an override of a built-in id, which returns to the built-in
defaults), a user model and a user-edited model survive, and the
provider removal keeps unrelated user entries.
- Lifecycle: a fresh wrap followed by an unwrap removes the config file
again (with and without the MCP entry), user content added after a fresh
wrap survives the unwrap, a re-wrap without the extra model drops the
stale one, and a directly injected provider entry is undone by the
unwrap callback as well.
- A user setting that is false, zero or an empty string is content, not
an empty section: such a setting survives the unwrap instead of costing
the user the whole file.
## Testing
- [x] Unit tests pass locally
- [x] Lint/format/type checks pass locally
- [x] New tests added for new functionality
The new parsing tests fail against the previous `:`-based parser:
```text
FAILED tests/test_providers_opencode_config.py::test_parse_extra_model_spec_keeps_colons_in_ids[qwen2.5-coder:7b-...]
FAILED tests/test_providers_opencode_config.py::test_parse_extra_model_spec_keeps_colons_in_ids[deepseek/deepseek-chat:free=DeepSeek Free-...]
FAILED tests/test_providers_opencode_config.py::test_parse_extra_model_spec_full
...
================ 21 failed, 2 passed, 103 deselected in 0.86s =================
```
With this branch (head
|
||
|
|
d6df127d57 |
feat(compress): accept Gemini and Bedrock Converse bodies on the gateway turn (#4679)
## Description
`/v1/compress` in gateway mode (a request with a `gateway` block) now
accepts two more native request bodies:
- Gemini `generateContent`
- Bedrock Converse
It compresses their text slots and returns `body` in the same native
shape. This lets a gateway like LiteLLM compress native Gemini and
Bedrock passthrough traffic, which it couldn't before.
`config.mode: "ccr"` is rejected for these shapes with a 400. Every
other mode is marker-free for them.
(Was stacked on #4617, which has merged; the branch now carries current
`main`.)
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### How
This works the same way `gateway_responses` does for Codex: it builds a
view instead of translating the body.
- Every compressible text slot becomes one chat-shaped pseudo message.
- Compressed text is written back by its path in the body.
- Everything else stays byte-identical: `thoughtSignature`, thought
parts, `functionCall`, `inlineData`, `toolUse`, `cachePoint`,
`reasoningContent`, `tools`/`toolConfig`.
- Tool calls appear in the view, so tool-name rules still apply. `Read`
stays verbatim, for example.
- A JSON slot is only written back if it still parses to the same JSON
type.
- If the compressed list doesn't line up with the slots, the original
transcript is forwarded and a warning is logged.
### CCR on native bodies (review fix)
A native body can't carry `headroom_retrieve`. Its `tools`/`toolConfig`
are forwarded untouched, and `create_ccr_tool_definition` only builds
chat, Anthropic and Responses shapes. It also never gets a re-drive. So
a CCR marker in a native body points at content the model can never
fetch. Before this fix, `config.mode: "ccr"` on a Gemini or Converse
gateway turn did exactly that: log output came back as a marker with
`ccr_hashes: [<hash>]` and `obligations: []`.
- `handle_compress` now answers `ccr` on a native view with `400
invalid_request`. The message names the shape, explains why, and lists
the supported modes.
- The check runs where the mode is chosen, right after `attach_view` has
recorded the shape. That is before any compression, session work or CCR
store write.
- **The rejection does not depend on `gateway.can_redrive`.** Even a
gateway that can re-drive has no way to put the retrieval tool into a
native request, so there is no capability under which native `ccr`
works.
- **Without a `gateway` block nothing changes.** A Gemini body is
already a 400 (no `messages`). A Converse body goes through the plain
messages path, where its untyped blocks are opaque: 0 tokens saved, no
markers in any mode, `ccr` included (checked). So the rule is
consistent: reject `ccr` wherever Headroom reads a native shape.
- The bypass header still returns the body untouched with no
compression, in any mode.
- I checked every other mode on the same marker-producing input
(default, `lossless`, `lossy_inline`, `lossless_then_lossy`): no marker
and no hash in any of them. These modes build pipelines with
`ccr_inject_marker=False, ccr_enabled=False`. Stale-Read notices are
hashless in marker-free modes since #4678.
- **Second defect, same family:** with `HEADROOM_DEDUPE=1`, cross-turn
dedup wrote `[↑NL same as msg 4]` into a Converse tool result. That `4`
counts view messages (system blocks, tool calls, result slots), not the
caller's `messages`, so it named the wrong message. Native views now set
`cross_turn_dedup_recoverable=False`, as `lossless` mode already did.
- The `not self.native_shape` guards on CCR arming and re-drive in
`GatewayTurn.finish` stay as defence in depth.
### Caller contract
**Gemini**
- Send top-level `contents` as a list, with no `messages` or `input`.
- `systemInstruction`/`system_instruction`, `tools` and
`generationConfig` are optional and pass through.
**Converse**
- Send `messages` where every message has role `user` or `assistant`,
and every content block has no `type` key.
- At least one block must use a Converse key: `text`, `toolUse`,
`toolResult`, `cachePoint`, and so on.
- `system` is optional, as a list of blocks.
**Both shapes**
- `config.mode`: omit it (marker-free default), or use `lossless`,
`lossy_inline` or `lossless_then_lossy`. `ccr` is a 400.
- Only gateway turns get a view. Without a `gateway` block the endpoint
behaves exactly as before. A client-sent `_headroom_native_view` key is
dropped, and `BODY_CONTROL_KEYS` strips the real one before the body is
returned.
- `config.frozen_message_count` and `config.protect_recent` count native
messages: Gemini `contents` entries and Converse `messages`.
`session.frozen_message_count` comes back in native units. The
`/v1/usage` reply's `frozen_message_count` still counts view messages,
and the docs say so.
- No re-drive is ever armed, so `obligations` never contains `redrive`,
and request hooks run stream-safe-only. Tool compaction and output
steering are skipped.
- Relayed usage can be:
- Gemini `usageMetadata`, where `promptTokenCount` already includes
`cachedContentTokenCount`.
- Converse `usage`, where `inputTokens` excludes `cacheReadInputTokens`
and `cacheWriteInputTokens`.
- The whole provider response that carries either.
- Every gateway answer, fail-open ones included, carries `wire_shape`:
`gemini`, `converse`, `openai_responses` or `messages`. A caller that
sent a native body and gets anything else, or no field at all, is
talking to an older Headroom and should forward its original body.
- Top-level `messages` in the answer is the Converse list for Converse
and the chat-shaped view for Gemini.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): no new errors; the 8
reported are in `ccr/mcp_server.py` and `cache/prefix_tracker.py` and
are identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed (local replay, below)
New in `tests/gateway/test_native_shapes.py`:
- `test_ccr_mode_is_rejected_for_native_bodies[gemini|converse]`: 400,
`invalid_request`, message asserted.
- `test_other_modes_forward_native_bodies_without_ccr_markers`: 4 modes
x 2 shapes, on log output large enough to produce a marker. Checks no
hash, no marker, the shape echoed, non-message keys byte-identical and
the native list mapped back.
- `test_ccr_still_works_on_a_chat_gateway_turn`: the same log output in
a chat body with `ccr` still carries a marker. This proves the input
above is marker-producing, and that chat `ccr` didn't regress.
- `test_native_bodies_get_no_dedup_pointer_into_view_messages`.
- `test_native_turns_are_never_asked_to_redrive` now runs in the default
mode, because `ccr` is a 400.
### Test Output
```text
# before the fix (new tests only, rest of the file passing)
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[gemini]
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[converse]
FAILED tests/gateway/test_native_shapes.py::test_native_bodies_get_no_dedup_pointer_into_view_messages
3 failed, 25 passed
# after
tests/gateway/test_native_shapes.py: 28 passed
# tests/gateway tests/test_gateway_turn_unit.py tests/test_compress_api.py
# tests/test_proxy_compress_endpoint.py tests/test_compress_session_mode.py
# tests/test_compress_session_shared.py tests/test_compress_passthrough.py tests/test_compress_failure.py
1 failed, 507 passed, 4 skipped
# the one failure, test_lossy_inline_emits_no_ccr_markers[1-json_minifier], fails identically on main
|
||
|
|
9321a415a6 |
fix(memory): scope TrafficLearner's SQLite reads to its user_id (#4850)
## Description
`TrafficLearner` saves its rows under its configured `user_id`, but its
two direct SQLite reads ignored `user_id`:
- `_hydrate_persisted_state` (startup) loaded every user's
`traffic_learner` rows into the dedup maps, then applied `LIMIT
dedup_window`. Bob's learner could bind a pattern hash to Alice's row
and later bump *her* evidence instead of creating his own row. Other
users' rows also used up his window.
- `_load_persisted_patterns_from_sqlite` (used by
`_collect_all_patterns` when writing learned context files) returned
every user's patterns, so Alice's learned text ended up in Bob's files.
Both queries now filter on `user_id = ?`. In hydration the filter runs
before the `LIMIT`. Only matters when several user IDs share one memory
database; a single-user setup (`user_id="default"`) is unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/memory/traffic_learner.py`: `WHERE user_id = ?` in both
reads. `_load_persisted_patterns_from_sqlite(db_path, user_id)` takes
the user explicitly (private helper; its only caller is updated).
- `tests/test_memory/test_traffic_learner.py`: new
`TestUserScopedPersistence`, which checks that hydration leaves another
user's row alone and collection excludes other users. Hand-built test
schemas gain the `user_id` column that the real `memories` table has,
and direct helper calls pass `"default"`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# new tests against origin/main (
|
||
|
|
491afe98b1 |
fix(proxy): reject a private x-headroom-base-url on chat and responses with 400 (#4857)
## Description
When the upstream guard rejected an `x-headroom-base-url` override (an
address on a private, loopback, link-local or otherwise non-public
network, or a host outside `HEADROOM_ALLOWED_BASE_URLS` in allowlist
mode), `_resolve_openai_upstream_base` logged a warning and returned
`None`. `/v1/chat/completions` and `/v1/responses` then sent the request
— the prompt and the client's `Authorization`, meant for the gateway the
client named — to the configured `OPENAI_API_URL` instead. Every other
route that reads the header (`headroom/providers/proxy_routes.py`:
Anthropic messages, Gemini, models, embeddings, audio, passthrough)
refuses with `400 Rejected unsafe upstream base URL`.
The OpenAI chat and responses handlers now refuse the same way.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
## Changes Made
- `headroom/proxy/handlers/openai.py`: `_resolve_openai_upstream_base`
raises `HTTPException(400, "Rejected unsafe upstream base URL")` instead
of returning `None` when the guard rejects the override (malformed /
non-http(s) values are unchanged: still ignored).
- `docs/content/docs/security-model.mdx`: one behaviour for all routes
(the page previously documented the two different outcomes).
-
`tests/test_upstream_guard.py::test_openai_routes_refuse_a_private_override_instead_of_using_the_default[chat|responses]`:
real app, configured OpenAI target is a local recording server; asserts
400 and that the configured target is never contacted.
**Sibling PRs that touch the same code** (merge order note):
`fix/upstream-guard-off-loop` adds `_resolve_openai_upstream_base_async`
next to this function; whichever merges second must make the async
resolver raise the same 400 (that PR's rejection branch is the one place
to change). `fix/upstream-idna-pin-once` touches the call sites only.
Suggested order: idna-pin-once, off-loop, then this PR rebased.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# New test on origin/main (
|
||
|
|
2f58877732 |
fix(proxy): check x-headroom-base-url off the event loop in chat and responses (#4856)
## Description
`handle_openai_chat` and `handle_openai_responses` validated
`x-headroom-base-url` with the synchronous `is_safe_upstream_url`, which
waits for the DNS lookup (up to `HEADROOM_UPSTREAM_RESOLVE_TIMEOUT_S`, 3
s by default) on the event-loop thread. Chat did this three times per
request and responses twice, so one slow-resolving host held up every
other in-flight request for several seconds. The other routes in
`headroom/providers/proxy_routes.py` already use
`is_safe_upstream_url_async`.
Both handlers now resolve the header once, through a new
`_resolve_openai_upstream_base_async` that uses the async guard (lookup
on a worker thread, pin scope opened on the request's task), and reuse
that result for the destination and the extra-header decision.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement
## Changes Made
- `headroom/proxy/handlers/openai.py`: header parsing split into
`_openai_base_url_candidate`; sync `_resolve_openai_upstream_base` (kept
for `_resolve_openai_upstream`) and new
`_resolve_openai_upstream_base_async` share it;
`_openai_upstream_from(request, custom)` picks the upstream from an
already-resolved override; chat and responses call the async resolver
once.
- `tests/test_proxy/test_openai_upstream_guard_off_loop.py`: asserts the
guard runs once and not on the event-loop thread, for chat and
responses.
- `tests/test_proxy_openai_cache_key_integration.py`: its stub now
patches the async resolver the handler calls.
**Sibling PRs that touch the same code** (merge order note):
`fix/upstream-idna-pin-once` adds the identical `_openai_upstream_from`
and resolve-once hunks (this PR is a superset of that part);
`fix/openai-base-url-reject` changes the rejection branch of
`_resolve_openai_upstream_base`, which after this PR must be mirrored in
`_resolve_openai_upstream_base_async`. Suggested order: idna-pin-once,
then this PR, then reject (rebasing reject means raising the 400 from
the shared rejection path in both resolvers).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# New test on origin/main (
|
||
|
|
3300923387 | Merge branch 'main' into fix/plugin-launcher-isolated-python | ||
|
|
4850666011 |
test(memory): reset the sticky tool tracker between memory gate tests (#4873)
## Description
On main,
`test_memory_call_the_proxy_did_not_inject_reaches_the_client[not_injected-anthropic]`
and `[not_injected-anthropic_stream]` fail when
`tests/test_memory_tool_user_scope.py` runs earlier in the same process
(CI shard `test (3)`).
This is a test-isolation problem, not a regression in the proxy. Memory
tool definitions are sticky per session (`SessionToolTracker`, a
process-wide tracker in `headroom/proxy/helpers.py`). The user-scope
test sends an identical Anthropic request with memory tools injected.
When the later `not_injected` case sends the same request, the session
tracker replays those tool definitions, so the proxy really does offer
`memory_delete` upstream on that request. Running the call is then the
intended behaviour. Each PR passed alone because nothing else in its own
shard had injected tools for that request first.
The fix resets the tracker before and after each test in
`tests/test_memory_tools_only_when_injected.py`. The same reset is
already used in `test_memory_tool_stream.py` and
`test_corrupt_golden_bytes_recovery.py`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `tests/test_memory_tools_only_when_injected.py`: an autouse fixture
calls `_reset_session_tool_tracker_for_test()` around each test. No
production code changes.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) (tests-only change)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
Same order as CI (`tests/test_memory_tool_user_scope.py` then
`tests/test_memory_tools_only_when_injected.py`), on origin/main
|
||
|
|
fda75de505 |
fix(memory): run only the memory tools the proxy injected on the request (#4870)
## Description
With memory enabled (`--memory`, `--learn`, `wrap/init --memory`), the
proxy ran any upstream tool call named
`memory_list`/`memory_save`/`memory_search`/`memory_update`/`memory_delete`,
and hid it from the client. That held even when the proxy had not
offered those tools on the request (`--no-memory-tools`), or when the
client had declared a tool with that name itself.
The proxy now runs only the memory tools it injected on the current
request. Every other call reaches the client unchanged. The proxy does
not run it, does not withhold it, and sends no continuation upstream.
Anthropic streaming already followed this rule
(`_server_memory_tool_names`). That helper is now shared, and every
executor uses it.
**Behaviour change**
- With `--no-memory-tools`, memory tool calls from the model now reach
the client instead of running on the proxy.
- A memory-named tool that the client declares is now the client's. The
proxy no longer runs it on any path, including the streaming path that
used to run it as a side effect.
- Injected memory tools behave as before: they run on the proxy, are
hidden from the client, and the turn continues upstream.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/memory_handler.py`: `server_memory_tool_names(tools,
client_tools)` moved here from `AnthropicHandlerMixin`. It now reads
Anthropic, Chat and Responses tool shapes through `extract_tool_name`.
`handle_memory_tool_calls` takes a required `tool_names` and skips every
other call.
- `handlers/openai.py`, Chat Completions (non-streaming): records the
injected names and passes them to the executor.
- `handlers/openai.py`, Responses (HTTP): runs a `function_call` only if
its name was injected. Before, any name in `MEMORY_TOOL_NAMES` ran.
- `handlers/openai.py`, Responses over WebSocket: the injected names are
kept per `response.create`, like the user id. The relay holds back and
runs only those calls.
- `handlers/anthropic.py`, non-streaming: passes the injected names it
already computed for streaming.
- `handlers/streaming.py`: removes the branch that ran memory calls
after the stream had already reached the client. It ran only for tools
the proxy had not injected. Memory handling in the stream (SSE
buffering, the Claude Code credential warning) now turns on only when
the proxy injected memory tools.
- Tests: new `tests/test_memory_tools_only_when_injected.py`. Existing
fixtures that relied on the old behaviour now inject the tool they call:
`test_codex_project_context.py`, `test_memory_handler_native_ops.py`,
`test_memory_handler_project_isolation.py`,
`test_memory_tool_stream.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
The new tests use `create_app` with a fake memory backend and a
`MockTransport` upstream. The WebSocket test uses a scripted upstream
socket. For each path (chat, Responses, Responses over WebSocket,
Anthropic, Anthropic streaming) they check two cases. First, injection
is off and the model calls `memory_delete`. Second, injection is on and
the client declares its own `memory_delete`. Each case expects zero
backend calls, exactly one upstream request, and the call in the
client's response. The control cases inject `memory_list` and expect it
to run as before.
On `origin/main` (
|
||
|
|
48df6df497 |
fix(memory): scope memory_update and memory_delete to the calling user (#4869)
## Description
The `memory_delete` tool called `backend.delete_memory(memory_id)`
without the user id. `LocalBackend.update_memory` and `delete_memory`
ignored `user_id`, and `DirectMem0Adapter` did the same. When users
share one store (`global` storage mode, or the qdrant-neo4j backend), a
memory id that belonged to another user could be deleted or overwritten.
The handler now passes the effective user id to `delete_memory`. Both
backends now treat an id owned by someone else as not found: delete
returns `False`, and update raises the same not-found `ValueError` it
raises for an unknown id. A caller that passes no `user_id` (the CLI,
`MemorySystem`) keeps the old unscoped behaviour.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/memory_handler.py`: `_execute_delete`, and the
delete+save fallback in `_execute_update`, pass
`user_id=effective_user_id`. The fallback now returns `not_found` when
nothing was deleted, instead of saving a replacement.
- `headroom/memory/backends/local.py`: `update_memory` and
`delete_memory` check the stored owner when `user_id` is given.
- `headroom/memory/backends/direct_mem0.py`: `update_memory` and
`delete_memory` check the owner through `get_memory` when `user_id` is
given. If the lookup fails, the id is treated as not found.
- `headroom/memory/backends/mem0_system_adapter.py`: `delete_memory`
with a `user_id` used to delete anyway when the owner lookup raised (it
caught every exception) or when the record had no owner. It now deletes
only when the stored owner matches. A `ValueError`/`OSError` from the
lookup is logged and the delete is refused; anything else propagates.
`Mem0Backend` (`backends/mem0.py`) is the raw client that this adapter
wraps; the proxy never calls it directly.
- Tests: a new `tests/test_memory_tool_user_scope.py` runs through
`create_app` with a real `LocalBackend` in `global` mode and a
`MockTransport` upstream. There is also one `DirectMem0Adapter` test.
Fake backends in `test_memory_handler_native_ops.py` now accept
`user_id`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
In the test, alice saves a memory. bob's request then gets an upstream
turn that calls `memory_delete`, or `memory_update`, with alice's id.
The test passes when alice's memory is unchanged. The controls check
that bob can still delete and update his own memory.
On `origin/main` (
|
||
|
|
2e4ff6f9c7 |
fix(proxy): keep upstream credentials out of manifests, env and TOIN (#4866)
## Description
Proxy secrets (upstream API keys, proxy token, licence) were stored or
passed in clear:
- Bench manifests and `ProxyDeploymentPlan.to_dict()` serialized the
full `ProxyConfig`, keys included.
- With `--workers N`, the whole config, keys included, went to each
worker in the `HEADROOM_PROXY_CONFIG_JSON` env var, which other
processes of the same user can read (`ps eww`, `/proc/<pid>/environ`).
- TOIN stored query field names in clear, although its module docstring
says field names are hashed; the text before a `:` can be a credential
(`<token>:x-oauth-basic`).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/models.py`: `PROXY_CONFIG_SECRET_FIELDS` (one list)
and `redact_proxy_config_secrets`.
- `headroom/testing/harness.py`, `ProxyDeploymentPlan.to_dict()`:
serialize the redacted config; the in-memory plan keeps real values.
- `headroom/proxy/server.py` (`--workers N`): secrets go to a 0600 temp
file named by `HEADROOM_PROXY_CONFIG_SECRETS_FILE`; workers read it, the
env JSON carries the config without them, and the file is removed when
the server stops. A worker that cannot read the file raises
`ConfigurationError` instead of falling back to env defaults (which
could drop the proxy token).
- `headroom/telemetry/toin.py`: query patterns hash field names; stored
patterns with raw field names are dropped on load and the store is
rewritten at once (a private `_import_patterns` reports whether anything
was dropped).
- Tests: `tests/test_proxy_config_secrets_at_rest.py` (manifest,
`to_dict`, worker env carry no secret; workers still get the real
values); `tests/test_toin_retention.py` expectations updated to hashed
field names.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_proxy_config_secrets_at_rest.py tests/test_toin_retention.py tests/test_stateless_toin.py tests/test_toin_*.py ...
125 passed
# new tests against origin/main
tests/test_proxy_config_secrets_at_rest.py: 3 failed (+1 for the fail-closed worker, which fails on the first commit too)
test_legacy_raw_query_keys_are_removed_on_load: fails without the load-time rewrite
# every test file importing the changed modules (73 files), this branch
2 failed, 1504 passed: tests/test_read_maturation_handler_nobust.py, both also failing on origin/main
(an earlier run on the first commit also hit tests/test_proxy_health.py failures that fail the same way on origin/main,
and one tests/test_proxy_ccr.py timing failure that main showed in a sibling test; that file passes 3/3 alone on both)
ruff check / format --check: clean; mypy: no new errors (same 8 pre-existing as origin/main)
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, the real `headroom proxy --workers 2
--stateless --no-telemetry` CLI, isolated `HOME`/`TMPDIR`,
`--openai-api-url` pointed at a local HTTP server that prints the
`x-gateway-auth` header it receives, `--openai-extra-headers
'{"x-gateway-auth":"FAKE-GATEWAY-0123"}'`.
- Exact command / steps: start the proxy, run `ps eww` on each worker
process, list `$TMPDIR`, POST `/v1/chat/completions`, stop the proxy
with SIGTERM.
- Observed result:
- origin/main: both worker processes' environments contain
`FAKE-GATEWAY-0123` (via `HEADROOM_PROXY_CONFIG_JSON`).
- this branch: neither worker's environment contains it; `$TMPDIR` holds
one `headroom-proxy-config-*.json` with mode `-rw-------`; the upstream
still receives `x-gateway-auth: FAKE-GATEWAY-0123` (workers load it from
the file); after SIGTERM the file is gone.
- On both, the parent's own command line holds the value because the
test passed it as a CLI flag; that is outside this change.
- Not tested: SIGKILL of the parent (the file stays in `$TMPDIR`).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: manifests and `to_dict()` show
secrets redacted; TOIN query-pattern keys are hashed and old raw-name
keys are dropped once on load (learned query statistics for those keys
restart).
- Kill switch / disable path: none.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
dffc4dcc57 |
fix(proxy): keep credentials out of proxy logs and 400 debug dumps (#4865)
## Description Credentials reached proxy logs and on-disk debug dumps: - The inbound request log line printed the full query string, so `?key=<api key>` (Google routes) and similar parameters were logged in clear. The streaming upstream-error lines and three passthrough failure lines printed the raw path with its query. - `should_redact_key` did not match `apikey` or headers such as `Ocp-Apim-Subscription-Key`, `X-Functions-Key`, `x-goog-api-key`. - The Anthropic and OpenAI 400 debug dumps kept a prefix of auth header values and were written with the default umask (world-readable on most systems). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/query_log_policy.py`: `redact_query_string_for_log` keeps parameter names (percent-encoded, so a `%0A` in a name cannot start a new log line) and drops every value; used by the inbound request line in `server.py`. - `query_log_policy.py` / `server.py`: `AccessLogQueryFilter` on `uvicorn.access`, installed by `create_app` in every process, replaces query values in uvicorn's access log (previously only LiteLLM's filter, when present, masked them). - `handlers/streaming.py`, `handlers/anthropic.py`, `handlers/openai.py`: error and passthrough log lines use the existing `_path_for_log`. - `wire_debug_redaction_policy.py`: `apikey` and any `*_key` / `*-key` name are secret. - `handlers/_debug_dump.py`: `redact_dump_headers` masks secret headers fully, including names configured in `*_extra_headers`; `write_owner_only_json` writes dumps with dir 0700 / file 0600, and the shared dump writer now uses it too. - `tests/test_credential_log_redaction.py`: query values never appear in the request/error log lines; dumped headers are fully masked; dump files are 0600 in a 0700 directory. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest -q tests/test_credential_log_redaction.py 16 passed # same file against origin/main 9 failed, 7 passed $ pytest -q tests/test_proxy tests/test_log_safety.py tests/test_image_log_redaction.py tests/test_credential_log_redaction.py 693 passed ruff check / format --check: All checks passed mypy (changed modules): no new errors (same 8 pre-existing errors as origin/main, in prefix_tracker.py and ccr/mcp_server.py) ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, the real `headroom proxy` CLI (`--stateless --no-telemetry`, isolated `HOME`), origin/main and this branch on separate ports. - Exact command / steps: `curl "http://127.0.0.1:<port>/stats?key=AIzaFAKEKEY0123456789&alt=sse"` and `curl "http://127.0.0.1:<port>/stats?x%0AINJECTED%20fake%20log%20line=1"`; grep `proxy_inbound_request` in the proxy's stderr. - Observed result: - origin/main: `... path=/stats query=key=AIzaFAKEKEY0123456789&alt=sse ...` - this branch: `... path=/stats query=key=<redacted>&alt=<redacted> ...` and `... query=x%0AINJECTED%20fake%20log%20line=<redacted> ...` (one line). - uvicorn access log: real uvicorn at `info` and `debug`, LiteLLM's filter removed, `POST ...?key=<canary>`: without this change the access line contains the canary; with it the line shows `key=<redacted>`. `test_uvicorn_access_log_drops_query_values` runs real uvicorn the same way. - 400 dumps: covered by `test_upstream_error_dump_masks_credentials_and_is_owner_only` (real handler, upstream response stubbed); not run against a live provider. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: log text only (query values shown as `<redacted>`, more header names masked); 400 dump files are now owner-only. - Kill switch / disable path: none needed. - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
c2292ed0c6 |
fix(gemini): apply the request rate limit to Cloud Code and countTokens (#4862)
## Description
`handle_google_cloudcode_stream` (Cloud Code Assist / Antigravity
`:streamGenerateContent`) and `handle_gemini_count_tokens`
(`:countTokens`) never called the request rate limiter
(`TokenBucketRateLimiter.check_request`) that
`handle_gemini_generate_content` uses. With a per-minute request limit
configured, requests on those two routes were never counted or refused.
The check is now one helper, `_enforce_request_rate_limit`, called by
all three routes. It uses the same identity rule
(`rate_limit_identity`), the same 429 detail and the same
`record_rate_limited` metric. It runs after the internal `x-headroom-*`
headers are stripped, the same point where generateContent already ran
it.
Fix adapted from a proposal by HetCreep.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/gemini.py`: new
`_enforce_request_rate_limit`. generateContent now calls it in place of
its inline block, and the Cloud Code stream and countTokens handlers
call it too. The token check in generateContent now computes its
identity inline, since the local `rate_key` moved into the helper.
- `tests/test_gemini_rate_limit_routes.py`: at 1 request/minute, four
calls to each route return `[200, 429, 429, 429]` and only the first
reaches upstream.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
New test on `origin/main` code:
```text
E assert [200, 200, 200, 200] == [200, 429, 429, 429]
E assert [200, 200, 200, 200] == [200, 429, 429, 429]
============================== 2 failed in 1.42s ===============================
```
With the fix:
```text
$ pytest -q tests/test_gemini_rate_limit_routes.py
============================== 2 passed in 5.21s ===============================
$ pytest -q tests/*gemini* tests/*cloudcode* tests/*rate_limit* tests/*antigravity* (20 files)
======================= 147 passed, 27 skipped in 26.52s =======================
$ ruff check / ruff format --check (0.17.3): All checks passed
$ mypy headroom/proxy/handlers/gemini.py --ignore-missing-imports
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS (Apple silicon), Python 3.12 venv. The real
`headroom proxy` runs from this branch and from `origin/main`, with
`--rpm 1 --no-optimize` and `--gemini-api-url` / `--cloudcode-api-url`
pointed at a local stand-in upstream (`http.server` that returns
`{"totalTokens": 3}` or one SSE chunk).
- Exact command / steps:
```text
headroom proxy --port 18912 --rpm 1 --no-optimize --gemini-api-url
http://127.0.0.1:18911 --cloudcode-api-url http://127.0.0.1:18911
4x curl -X POST
http://127.0.0.1:18912/v1beta/models/gemini-2.0-flash:countTokens -H
'x-goog-api-key: test' -d '{"contents":[...]}'
4x curl -X POST
'http://127.0.0.1:18912/v1internal:streamGenerateContent?alt=sse' -H
'authorization: Bearer test' -d
'{"model":"gemini-2.0-flash","project":"p","request":{"contents":[...]}}'
```
- Observed result:
```text
this branch: countTokens 200 429 429 429 cloudcode stream 200 429 429
429
origin/main: countTokens 200 200 200 200 cloudcode stream 200 200 200
200
```
The unit tests also drive the handlers with a real Starlette `Request`
whose JSON body arrives in two ASGI chunks.
- Not tested: real Google upstreams (the change happens before the
upstream call).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only when a request rate limit is
configured. Cloud Code stream and countTokens requests now count toward
it and get 429 when over, like generateContent. Clients that call
countTokens before every generation (Gemini CLI) now use two requests
from the bucket per turn.
- Kill switch / disable path: existing rate-limit settings (raise or
disable the request limit)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Other routes that do not consult the request limiter today (Bedrock
invoke, batch endpoints, the Responses WebSocket) are left out of this
PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
5503383bef |
fix(proxy): bound brotli output per step and lossless inverse expansion (#4861)
## Description
Two places could allocate far more memory than the input size justifies:
1. **Brotli request bodies** (`_brotli_bounded` in
`headroom/proxy/helpers.py`): the body was fed to
`Decompressor.process()` in 64 KiB input slices, with the 100 MB cap
checked after each slice. One slice of a highly compressed body (a
1,681-byte body holding 1 GiB of zeros) expands in full inside a single
`process()` call before the check runs.
2. **Lossless compaction self-checks** (`compact_lossless` in
`headroom/transforms/lossless_compaction.py`): to confirm a fold
round-trips, it runs `expand_runs` / `unfold_repeated_blocks` on the
candidate. Those inverses read their repeat counts from the text, so a
marker line already present in tool output (`... (repeated 2000000
times)`, or chained `... (repeats N lines from N lines back)` lines that
double each time) made the check allocate millions of lines.
Fixes:
1. `process()` is called with `output_buffer_limit=64 KiB`. Pending
output is drained with `process(b"")` before the next slice (brotli
accepts only empty input while it holds unprocessed input) and again
after the last slice until `is_finished()`, with the cap checked after
every step. brotli older than 1.2.0 has no output limit, so it is now
refused for `br` bodies with a clear upgrade message, the same way a
brotli without `Decompressor` already was.
2. `expand_runs` and `unfold_repeated_blocks` take an optional
`max_chars`. Every line is charged its length plus its newline before it
is appended: a run costs `count * (len(line) + 1)`, and a block copy is
costed by summing the copied lines. Going over the limit raises a
private `CompressionError` subclass before anything is allocated, and
the content is returned unchanged. The extra character for the dropped
last newline is allowed only when the input has no trailing newline.
`compact_lossless` bounds each inverse by the text it must reproduce:
the content itself, or for config, the run-collapsed text that the block
inverse must rebuild (it can be longer than the content). Called without
`max_chars`, both public functions behave exactly as before.
Fix adapted from a proposal by HetCreep. Change from that proposal: with
real brotli 1.2.1, `can_accept_more_data()` is already True once all
input is consumed even while output is still pending, so stopping there
cut off normal bodies (600 KB of text failed as "truncated stream").
This version drains until `is_finished()`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/helpers.py`: `_brotli_bounded` pulls output in limited
steps, drains pending output, and requires `can_accept_more_data`
(brotli>=1.2.0).
- `headroom/transforms/lossless_compaction.py`: `_OutputBudget` and
`max_chars` on `expand_runs` / `unfold_repeated_blocks`;
`compact_lossless` passes per-stage limits and returns the content
unchanged when one is exceeded.
- `tests/test_lossless_expansion_limit.py` (new), using tracemalloc so
it works on any OS:
- a run marker (log/text/config) and chained block markers stay under an
8 MB peak;
- a 1 MiB line expanded 1000x with 1000 shrinking markers, and 1000
block copies of it, stay under 16 MB;
- the exact `max_chars` boundary holds for both inverses, with and
without a trailing newline;
- a config fold whose run-collapsed intermediate is longer than the
content is still kept;
- the block inverse on 400 KB of short lines stays under 6 MB;
- real repeats still fold.
- `tests/test_request_body_decompression_limits.py`: fake decompressors
for the per-step bound, the fail-closed path on old brotli, and draining
output that is still pending after the last slice.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
New tests on `origin/main` code:
```text
E AssertionError: peak 133.5MB: the marker count was expanded
E AssertionError: peak 217.4MB: the block markers were unfolded
E AssertionError: peak 128.0MB suggests one slice was fully expanded
FAILED tests/test_lossless_expansion_limit.py::test_run_marker_in_input_is_not_expanded[log]
FAILED tests/test_lossless_expansion_limit.py::test_run_marker_in_input_is_not_expanded[text]
FAILED tests/test_lossless_expansion_limit.py::test_run_marker_in_input_is_not_expanded[config]
FAILED tests/test_lossless_expansion_limit.py::test_chained_block_markers_in_config_are_not_expanded
FAILED tests/test_request_body_decompression_limits.py::test_brotli_step_output_is_bounded
FAILED tests/test_request_body_decompression_limits.py::test_brotli_without_output_limit_fails_closed
```
With the fix:
```text
$ pytest -q tests/test_request_body_decompression_limits.py tests/test_lossless_expansion_limit.py
======================== 32 passed, 2 skipped in 1.68s =========================
# same file with brotli 1.2.1 importable (the 2 skips are real-brotli tests):
$ pytest -q tests/test_request_body_decompression_limits.py
============================== 29 passed in 4.32s ==============================
$ pytest -q tests/test_lossless_expansion_limit.py tests/test_lossless_excluded_compaction.py tests/test_transforms_config_compressor.py tests/test_request_body_decompression_limits.py tests/test_lossless_diff_fold_guard.py tests/test_lossless_timestamp_guard.py tests/test_bash_search_lossless_fold.py
======================== 177 passed, 2 skipped in 6.90s ========================
$ ruff check / ruff format --check (0.17.3): All checks passed
$ mypy headroom/proxy/helpers.py headroom/transforms/lossless_compaction.py --ignore-missing-imports
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: macOS (Apple silicon), Python 3.12, brotli 1.2.1.
- Exact command / steps: built a 1 GiB-of-zeros brotli stream (1,681
bytes compressed) and called `_brotli_bounded`. Then round-tripped
`b""`, `b"x"`, 600 KB of text, 300 KB of random bytes, 50 MB of zeros
and 5 MB of repeated random data, plus a stream with its last 3 bytes
cut off.
- Observed result: `refused: Decompressed brotli request body exceeds
100MB`, max RSS growth 100 MB, 0.09 s. All round trips were exact. The
cut-off stream gave `Failed to decompress brotli request body: truncated
stream`.
- Lossless: `compact_lossless(("X" * 1 MiB) + "\n... (repeated 1000
times)\n" + "a\n... (repeated 1 times)\n" * 1000, "text")` returns the
input unchanged with a 3.1 MiB tracemalloc peak. With a line limit only,
it peaked at 1002 MiB.
- Not tested: a full proxy request carrying `Content-Encoding: br` (the
reader dispatches to the same function).
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: `br` request bodies are now refused
(400 with an upgrade message) when the installed `brotli` is older than
1.2.0. brotli is an optional install, and gzip/deflate/zstd are
unaffected. Lossless compaction leaves content unchanged only when it
carries markers asking for more lines than it has.
- Kill switch / disable path: upgrade `brotli` to >=1.2.0
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
The memory tests use tracemalloc peaks, not RLIMIT_AS, so they run on
macOS as well as Linux.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reported by @JoaoMarcos44. Thank you for your contribution! 🩵
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
5d44855c6f |
fix(proxy): withhold per-project savings and quota stats from /stats network callers (#4859)
## Description `GET /stats` removes `recent_requests` / `request_logs` for a caller that is not loopback, not a trusted dashboard CIDR peer and has not presented the proxy token. The same payload still carried `savings.per_project`, `persistent_savings.projects` (project names and per-project spend) and the provider quota sections (`get_quota_registry().get_all_stats()`, e.g. account login and plan). For that same caller `/stats-lifetime` already drops `projects` and `/quota` is refused, so `/stats` was the one route that still handed them out. `/stats` and `/stats?cached=1` now drop those fields under the same rule (`_request_can_view_stats_metadata`). The nested `savings` / `persistent_savings` dicts are rebuilt rather than edited in place, so the shared cached snapshot still serves loopback callers in full. Token-authenticated callers keep seeing everything, as documented in "What the proxy token unlocks"; that docs table now lists the extra fields. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py` (`/stats`): for callers without stats-metadata access, also pop every quota tracker key and drop `savings.per_project` and `persistent_savings.projects`; docstring updated. - `docs/content/docs/security-model.mdx`: the `/stats` row of "What the proxy token unlocks" lists per-project savings and the quota sections. - `tests/test_proxy_stats_network_gating.py`: network caller on `/stats` and `/stats?cached=1` gets no project name / quota data; loopback still gets them after a network caller primed the cached snapshot; a token holder still gets them. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_proxy_stats_network_gating.py 5 passed in 18.60s # with the server.py change stashed (origin/main code) FAILED tests/test_proxy_stats_network_gating.py::test_stats_withholds_project_names_and_quota_stats_from_network_callers[/stats] FAILED tests/test_proxy_stats_network_gating.py::test_stats_withholds_project_names_and_quota_stats_from_network_callers[/stats?cached=1] 2 failed, 3 passed in 7.10s $ pytest -q tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py tests/test_stats_output_reduction_gate.py tests/test_proxy_stats_network_gating.py 174 passed, 5 warnings in 44.87s ruff check / format --check (0.17.3): All checks passed, 2 files already formatted mypy headroom/proxy/server.py: Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.12, `create_app(ProxyConfig(stateless=True, ...))` under FastAPI `TestClient`, one recorded request for a project. - Exact command / steps: `GET /stats` once as a network peer (`203.0.113.9`, Host `203.0.113.5`) and once as loopback; print whether `savings.per_project` and `persistent_savings.projects` are present. - Observed result: - origin/main: network caller gets `savings.per_project` and `persistent_savings.projects` with every project name. - this branch: network caller gets `savings.per_project: None` and no `persistent_savings.projects`; loopback still gets both in full. - Not tested: a real Copilot/Codex quota tracker (the test registers a stand-in tracker). ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: yes, for network callers without dashboard-CIDR or token access only: `/stats` no longer includes per-project savings or quota sections. Loopback dashboards, CIDR peers and token holders are unchanged. - Kill switch / disable path: configure `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` or present the proxy token, as for `/stats-lifetime` and `/quota`. - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Fix adapted from a proposal by HetCreep. `persistent_savings.storage_path` (a local filesystem path) is still in the network payload; left as is to keep this change to the fields the sibling routes already withhold. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4859" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |