Commit Graph
3781 Commits
Author SHA1 Message Date
JD Davis 6712d432c2 Merge branch 'main' into feat/https-compose-caddy 2026-11-17 01:49:10 -06:00
18067b1d19 fix(content-router): protect cache_control markers nested in tool_result content (#4898)
## Description

`ContentRouter` never rewrites a block that carries `cache_control`
(contract 1 in `_process_content_blocks`), but the guard only checks
`"cache_control" in block`. The Messages API also accepts the marker on
a `text` sub-block of a `tool_result`'s list content. That shape is not
seen as marked, so the `tool_result` is compressed and re-wrapped as
`[{"type": "text", "text": compressed}]`: the marker is dropped and the
bytes under it change, even for a tool result earlier in the
conversation. The cross-turn dedup `protected` check has the same gap.
`compress()` and `POST /v1/compress` return the router output as-is, so
callers get the rewritten block back.

This PR treats a `tool_result` whose list content carries a marker as
marked, the same shapes `count_cache_breakpoints` in
`headroom/proxy/helpers.py` already counts.

Fixes #4890

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_router.py`: new helper
`_carries_cache_control(block)`, true when the block has `cache_control`
or is a `tool_result` whose list content has a sub-block with
`cache_control`.
- Used in the per-block contract 1 guard of `_process_content_blocks`
and in the `protected` check of the cross-turn dedup. The contract 1
docstring now names the nested shape. The fresh-turn release
(`fresh_turn` and a `tool_result` in the final message) is unchanged.
- `tests/test_transforms_content_router.py`: the existing
`test_tool_result_cache_control_protected` is parametrized over
`block_marker` / `nested_marker`, plus
`test_marked_tool_result_in_history_survives_apply`, which goes through
`apply()` with the marked tool result in history.
- `tests/test_cross_turn_dedup.py`:
`test_dedup_never_rewrites_nested_cache_control_tool_result`, with an
unmarked control that must still be folded.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Source: one commit `916e8395` on `main` `5023c84e`. Runs below were
built from `git archive` of that commit in a `python:3.13-slim-bookworm`
image with `uv sync --frozen --extra proxy --extra dev`
(`headroom._core` built by maturin). ruff ran on the three touched
files, mypy on `headroom`. The new and changed tests were run 5 times in
a row. The module run covers the two touched test files plus
`test_content_router_detection_dedup.py`,
`test_responses_cross_turn_dedup.py` and
`test_openai_chat_dedup_recoverability.py`. For the reverse check,
`content_router.py` was replaced with the copy from `5023c84e` and the
tests kept.

### Test Output

```text
== RUFF CHECK RC 0
== RUFF FORMAT RC 0
== MYPY RC 0
# new and changed tests, run 1 and run 5 (runs 2-4 identical):
====================== 9 passed, 124 deselected in 0.97s =======================
====================== 9 passed, 124 deselected in 0.37s =======================
# related modules:
======================= 149 passed, 1 skipped in 12.32s ========================
# with content_router.py restored to 5023c84e (fix removed):
PASSED tests/test_transforms_content_router.py::test_tool_result_cache_control_protected[block_marker]
PASSED tests/test_transforms_content_router.py::test_marked_tool_result_in_history_survives_apply[block_marker]
FAILED tests/test_transforms_content_router.py::test_tool_result_cache_control_protected[nested_marker]
FAILED tests/test_transforms_content_router.py::test_marked_tool_result_in_history_survives_apply[nested_marker]
FAILED tests/test_cross_turn_dedup.py::test_dedup_never_rewrites_nested_cache_control_tool_result
================= 3 failed, 6 passed, 124 deselected in 0.58s ==================
```

The `block_marker` cases are the existing behaviour and pass on both;
only the nested-marker tests fail without the fix.

## Real Behavior Proof

- Environment: Linux container, Python 3.13, headroom installed from
this branch with `uv sync --frozen --extra proxy`. No upstream provider
and no API key: `/v1/compress` does not forward.
- Exact command / steps: `python -m headroom.cli proxy --host 127.0.0.1
--port 8833 --no-telemetry --no-rate-limit`, wait for `/readyz`, then
POST `/v1/compress` with `model: claude-sonnet-4-5` and a `tool_result`
holding 17289 characters of log lines with one `cache_control: {"type":
"ephemeral"}`. Four requests: marker on the `tool_result` or on its
`text` sub-block, with the tool result either in the final message or in
history (followed by an assistant and a user turn). Same image, same
steps with `content_router.py` from `main`.
- Observed result: on `main` the nested-marker requests came back as
17289 -> 29 characters with 0 markers out of 1, both in the final
message and in history. With this patch all four requests came back
17289 -> 17289 characters with 1 marker out of 1, the same as the marker
on the `tool_result` itself (which was already 1 -> 1 on `main`).
- Not tested: a real Anthropic endpoint, the `/v1/messages` forwarding
path end to end with this patch, SSE streaming, Bedrock, Windows and
macOS.

## Runtime Rollout Safety

- Rollout-managed feature(s): N/A, this is a bug fix in the existing
compression path with no flag.
- Minimum rollout channel: N/A, no rollout-managed feature.
- Stable/default behavior changed: Yes, only for a `tool_result` whose
list content carries `cache_control` on a sub-block: outside the
fresh-turn release it is now passed through unchanged, as a marker on
the block itself already was.
- Kill switch / disable path: N/A, no flag. Removing the marker on the
client side gives the old compression.
- Unsafe override required: No.
- Qualification impact: None expected. Unmarked tool results and string
content take the same path as before.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`; it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Documentation is N/A, the contract 1 docstring is the only place that
describes this. On `/v1/messages` the proxy already moved the marker
back onto the compressed `tool_result` before forwarding, so the
breakpoint count stayed the same there, but the bytes under it still
changed. That path uses the same router guard, so a marked tool result
in history should now stay uncompressed there too (not run end to end,
see Not tested). `read_maturation.py` intentionally rewrites marked
blocks while keeping their marker, so it is left alone. "New and
existing unit tests pass locally" refers to the modules listed above,
run in the container.

Written with help from Claude Code.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4898"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:52:21 -06:00
8da7e17e4e fix(kompress): honour toin_read_only in the Kompress CCR store (#4895)
## Description

Fixes #4801

The second TOIN write path of #4474. #4493 gated the router →
`SmartCrusher.crush()` path, but `store_kompress_in_ccr` still recorded
a TOIN observation unconditionally, so a Subscription request
(`CompressionPolicy.toin_read_only=True` — "never writes new pattern
observations from this request") that went through Kompress added an
observation to the shared TOIN store anyway. This change threads the
same per-call gate through the Kompress CCR store, both compressors, and
the router seam.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `store_kompress_in_ccr` gains a keyword-only `record_toin: bool =
True`; when `False` the CCR store write still happens (compressed output
stays retrievable, marker unchanged) but the
`get_toin().record_compression(...)` call is skipped.
- `KompressCompressor.compress()` / `compress_batch()` accept and
forward `record_toin` on every internal path (single, batched
reconstruction, sequential fallback); `apply()` derives it from a
`compression_policy` kwarg, the way `SmartCrusher.apply()` and
`CacheAligner` do.
- `RemoteKompressCompressor.compress()` accepts the same keyword
(keeping the drop-in parity contract) and forwards it to the shared
store function.
- `ContentRouter._try_ml_compressor` — the single seam both compressors
are called through — sends `record_toin=False` only when the request's
`_runtime_compression_policy.toin_read_only` is set, mirroring
`_invoke_smart_crusher`; the kwarg is omitted otherwise, so compressors
and test doubles without it keep working and default store calls keep
the exact pre-change 3-argument shape.
- Six regression tests in `tests/test_compression_policy_toin_gate.py`
covering the store function, local `compress()`, `compress_batch()`,
`apply()` under a Subscription policy, the remote client, and router
forwarding.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression_policy_toin_gate.py tests/test_remote_kompress_dropin.py tests/test_ccr_marker_policy.py tests/test_compression_policy.py
58 passed in the combined related run (gate file alone: 8 passed, 8 skipped — skips are the suite's own headroom._core-gated tests, no Rust wheel on this machine)

# Same five runnable new tests against unmodified main: 5 failed (they pass with this change)

$ ruff check headroom/transforms/kompress_compressor.py headroom/transforms/kompress_remote.py headroom/transforms/content_router.py tests/test_compression_policy_toin_gate.py
All checks passed!
$ ruff format --check (same four files)
4 files already formatted

# Issue repro, extended with the new gate (TOIN at a temp file):
store_kompress_in_ccr("word " * 400, "word " * 100, 400)                    -> TOIN total 0 -> 1 (reported bug; kept as the default for policy-less callers)
store_kompress_in_ccr("word " * 400, "word " * 100, 400, record_toin=False) -> TOIN total 1 -> 1, cache key still returned
store_kompress_in_ccr("word " * 400, "word " * 100, 400)                    -> TOIN total 1 -> 2
```

Type checking was not run locally (no mypy environment on this machine);
CI runs it.

## Real Behavior Proof

- Environment: Linux, Python 3.12, checkout of `main` at 5023c84e plus
this branch, TOIN pointed at a temporary file, no provider calls.
- Exact command / steps: Ran the issue's function-level repro script
(extended with the new `record_toin` gate), then the pytest and ruff
commands pasted under Testing, then re-ran the five runnable new tests
with the source changes stashed to confirm they fail on unmodified
`main`.
- Observed result: Gated store call leaves the TOIN total unchanged
while still returning a CCR cache key whose entry retrieves
successfully; ungated calls record as before. Related suites: 58 passed.
New tests: fail on `main`, pass with this change.
- Not tested: A live Subscription request through the running proxy
end-to-end (the issue itself was reported and verified at function
level; router forwarding is covered by a stub-compressor test), and the
`_core`-gated router test on this machine (skips without the Rust wheel;
runs in CI).

## Runtime Rollout Safety

- Rollout-managed feature(s): None — this honours the existing F2.2
`CompressionPolicy.toin_read_only` flag; no new flag or channel is
introduced.
- Minimum rollout channel: N/A — no rollout-managed behavior; the gate
only takes effect for requests already classified Subscription with
policy enforcement on.
- Stable/default behavior changed: No for policy-less and PAYG/OAuth
callers (default `record_toin=True` everywhere); Subscription Kompress
traffic stops writing TOIN observations, which is the documented policy
and the bug fix.
- Kill switch / disable path: The existing
`HEADROOM_PROXY_AUTH_MODE_POLICY_ENFORCEMENT` switch already reverts
Subscription requests to the PAYG policy (TOIN writes re-enabled)
without a deploy.
- Unsafe override required: No.
- Qualification impact: None — no change to compression output bytes,
CCR storage, or retrieval markers; only the TOIN learning write is
gated.
- Rollback path: Revert this PR; no data migration (TOIN observations
written before the fix are ordinary pattern data).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Documentation checklist item is N/A: no user-facing docs describe this
internal write path; the changed functions' docstrings were updated in
place. Direct callers with no policy context keep the previous
write-enabled behaviour, matching the deliberate compatibility decision
of the F2.2 SmartCrusher gate.

---------

Co-authored-by: TEJA P C <pcteja@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:51:50 -06:00
Danila KatalshovandJD Davis f9627eb462 fix(compress): verify scalar types in lossless message comparisons (#4884)
## Description

Agent mode can report a message as lossless after a pipeline extension
changes a tool argument from JSON true to 1. Python considers those
values equal even inside nested dicts. The same issue affects integers
and floats.

## Type of Change

- [x] Bug fix

## Changes Made

- Verify aligned message structures before accepting equal-valued
scalars and require their types to match. Identical objects
short-circuit during traversal.
- Exercise changes both with and without simultaneous reversible string
compression. The existing token recount test now simulates a failed
string round trip at the leaf being verified.

## Testing

- [x] Ran the agent-mode, codec, compress API and canonical pipeline
suites. The regression test failed before the fix.
- [x] Ruff lint and format checks passed on changed files.

### Test Output

```text
72 passed, 7 skipped, 1 warning in 1.56s
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, native extension from headroom-ai
0.44.0, offline.
- Exact command / steps: Run the following from the checkout with the
Python environment active.
- Observed result: The output changed from `True 0 int 1` to `True 1
bool True`. The changed tool call is now reverted.
- Not tested: Live providers or the full repository suite.

```sh
HEADROOM_OFFLINE=1 python - <<'PYTHON'
from dataclasses import replace
from headroom import compress
from headroom.hooks import CompressionHooks
from headroom.pipeline import PipelineStage

class NumericHook(CompressionHooks):
    def on_pipeline_event(self, event):
        if event.stage == PipelineStage.INPUT_COMPRESSED:
            msg = dict(event.messages[0])
            msg["content"] = [dict(msg["content"][0], input={"enabled": 1})]
            return replace(event, messages=[msg])
        return None

messages = [{"role": "assistant", "content": [
    {"type": "tool_use", "id": "call", "name": "run", "input": {"enabled": True}}
]}]
r = compress(messages, mode="agent", hooks=NumericHook())
v = r.messages[0]["content"][0]["input"]["enabled"]
print(r.lossless, r.reverted_messages, type(v).__name__, v)
PYTHON
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Lossless verification reverts
messages with scalar type changes.
- Kill switch / disable path: Existing verify_lossless option outside
agent mode.
- Unsafe override required: No.
- Qualification impact: Adds regression coverage for nested booleans,
integers and floats.
- Rollback path: Revert this patch.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:51:15 -06:00
zhengqiuyangandJD Davis 303f521812 fix(smart-crusher): keep container-shaped strings as strings in lossless tables (#4875)
## Description

A string cell in a `json` column whose text is a JSON **container**
(`"[]"`, `"{}"`, `"[1,2]"`) is
classified as `StringifiedJson` and un-escaped into
`CellValue::Scalar(parsed)`, so the cell renders
byte-identically to the real value it spells. The csv-schema decoder
cannot tell the two apart
afterwards: `expand_compacted` returns `[]` where the original had
`"[]"`, and with
`verify_lossless=True` (also forced by `mode="agent"`) the round-trip
check fails and the whole
message is reverted, silently dropping its compression.

This is the container-shaped remainder of #4053, which fixed the
scalar-looking case (`"null"`,
`"true"`, `"1"`). The formatter already knows how to render these
strings unambiguously
(`csv_formatter_json_column_quotes_strings_that_parse_as_json` covers
`"[1]"` and `'{"a":1}'`), but
`compact()` never lets them reach it as strings.

Closes #4874

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `compactor::cell_from_value`: for `StringifiedJson` that is not an
array of objects, keep the
original string (`CellValue::Scalar(v.clone())`) instead of the parsed
value. The formatter's
existing ambiguous-text rule JSON-quotes it, so the decoder returns the
string.
- Nested behaviour is unchanged: a stringified **array of objects**
still becomes a nested table, so
`test_nested_table_markers_resolve_to_source_bytes` and its CCR markers
keep working.
- Rust regression test pinning the exact wire cells (`"[]"` for the
array, `"""[1]"""`-style for the
  string).
- Python regression test at the caller boundary (`_crush` +
`expand_compacted`).

## Testing

- [x] Unit tests pass (`cargo test -p headroom-core --lib` -> 1044
passed, 1 ignored)
- [x] `cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cargo test -p headroom-core --lib compaction::
test result: ok. 74 passed; 0 failed; 0 ignored

$ cargo test -p headroom-core --lib
test result: ok. 1044 passed; 0 failed; 1 ignored

# with the fix reverted, the new Rust test fails:
left: "\"[]\""
right: "\"\"\"[]\"\"\""
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, Rust 1.100
`x86_64-pc-windows-gnu`, repo `main` at
  `fda75de5`
- Exact command / steps: `cargo test -p headroom-core --lib

compaction::compactor::tests::stringified_container_strings_keep_their_identity`;
the test builds

`[{id:0,v:[]},{id:1,v:"[]"},{id:2,v:[1,2]},{id:3,v:"[1,2]"},{id:4,v:{}},{id:5,v:"{}"}]`,
compacts
  it, and asserts the rendered `v` cells.
- Observed result: without the change the array and the string render
the same cell (`"[]"`); with
  it the string renders as the JSON string literal (`"""[]"""`), which
  `expand_compacted` maps back to `"[]"`.
- Not tested: the Python regression test could not be executed locally
because this machine cannot
build the native extension (Rust build stopped in `icu_segmenter` under
the GNU toolchain); it is
  included for CI, where the wheel is built normally.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; the change is in the lossless table
codec only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: a stringified JSON container in a
table cell now stays a string
in the densified text (it was displayed as the parsed value). This is
required for the documented
reversibility of `csv-schema`; the model sees a JSON string literal
instead of nested JSON.
- Kill switch / disable path: n/a (lossless path)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the single `cell_from_value` branch.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4875"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:50:47 -06:00
cb8414cc73 perf(bm25): lower batch allocations -- 93.4% less peak memory (#4893)
## TL;DR

- Reduce peak memory in `BM25Scorer.score_batch` for 1,000 search
results.
- Keep corpus counts between two passes and reuse reason strings within
each batch.
- Peak allocation falls from 2,471,190 to 164,222 bytes, saving
2,306,968 bytes; runtime rises from 15.41 to 22.56 ms.
- `pytest tests/test_relevance.py tests/test_relevance_extra.py -k 'bm25
or BM25' -q`: 16 passed, 32 deselected.

## Description

The batch scorer kept every document's tokens until scoring finished. It
now keeps query-term document counts, then reads each document again for
scoring.

The score formula, document lengths, and query-term counts stay the
same. A small local dictionary also reuses equal result descriptions.

## Changes Made

- `headroom/relevance/bm25.py`: `BM25Scorer.score_batch` keeps only
query-term document counts in the first pass, then tokenizes each
document again in the scoring pass.
- `headroom/relevance/bm25.py`: a small per-batch dictionary reuses
equal reason strings.
- No public interface, setting, or test file changes.

## Validation

These results use the repository's search-result generator on Linux
x86_64 with Python 3.12.13 and `HEADROOM_OFFLINE=1`.
Memory means peak Python allocations during one batch, not process RSS.
Runtime rows show median times without allocation tracing.

| Situation | Before | After | Change |
|---|---|---|---|
| 1,000 search results, peak allocation | 2,471,190 bytes | 164,222
bytes | 93.4% less |
| 100 search results, runtime | 1.46824 ms | 2.28269 ms | 55.5% slower |
| 1,000 search results, runtime | 15.40643 ms | 22.55516 ms | 46.4%
slower |
| 100 results with empty query, runtime | 0.03821 ms | 0.03777 ms |
Unchanged path; no observed slowdown |
| Single result, runtime | 0.01296 ms | 0.01295 ms | Unchanged path; no
material difference |

Memory measurements across seven fixed inputs and three runs give:

| Statistic | Value |
|---|---|
| Old -> new (mean) | 2.4 MiB -> 160.4 KiB (93.4% less peak memory) |
| 95% CI of the new/old ratio (bootstrap, 10,000 resamples) |
0.066–0.067 (93.3–93.4% less peak memory) |
| Mean ± SD of the change per input | -2.2 MiB ± 1.7 KiB |
| 95% CI of the mean change (t-interval) | -2.2 MiB to -2.2 MiB |
| Number of fixed inputs (n) | 7 (3 runs) |

This is a memory-for-time tradeoff: repeated tokenization makes nonempty
batches slower.
The later pass releases each token list after scoring its document.

Run the repository benchmarks with:

```bash
HEADROOM_OFFLINE=1 python -m pytest benchmarks/bench_relevance.py::TestBM25Benchmarks -q
```

All seven benchmarks passed. An additional local comparison checked
8,568 combinations against the original implementation with exact
results.
Those combinations cover empty documents, repeated words, UUIDs,
Unicode, and different scoring settings.

## Why this is safe and easy to maintain

The change adds six net code lines and keeps one shared scoring formula.
Scores, result order, reasons, and matched terms remain unchanged for
the checked inputs.
Existing BM25 tests cover corpus IDF, matching, and normalization.

## Type of Change

- [x] Performance improvement

## Testing

- [x] Focused BM25 unit tests pass.
- [x] `ruff check headroom/relevance/bm25.py` passes.
- [x] `ruff format --check headroom/relevance/bm25.py` passes.
- [ ] Full unit suite passes.
- [ ] Type checking passes (`mypy headroom`).
- [ ] New repository tests added.

The broader relevance run reports 37 passed, eight skipped, and three
failed on both versions.
All three failures require the unavailable `headroom._core` extension in
SmartCrusher integration tests.
No new behavior or public interface is added; existing tests exercise
the changed path.

### Test Output

```text
$ HEADROOM_OFFLINE=1 pytest tests/test_relevance.py tests/test_relevance_extra.py -k 'bm25 or BM25' -q
====================== 16 passed, 32 deselected in 0.62s =======================

$ ruff check headroom/relevance/bm25.py && ruff format --check headroom/relevance/bm25.py
All checks passed!
1 file already formatted

$ HEADROOM_OFFLINE=1 python -m pytest benchmarks/bench_relevance.py::TestBM25Benchmarks -q
============================== 7 passed in 5.81s ===============================
```

## Real Behavior Proof

- Environment: Linux x86_64, Python 3.12.13, offline mode, no provider
or model needed.
- Exact command / steps: Run this direct scoring call after applying the
patch:

```bash
HEADROOM_OFFLINE=1 python -c 'from headroom.relevance.bm25 import BM25Scorer; s = BM25Scorer(); print([(r.score, r.matched_terms) for r in s.score_batch(["shared zeta", "shared alpha", "shared beta", "shared gamma"], "shared zeta")])'
```

- Observed result: The rare term ranks the first document higher:

```text
[(0.13093333199837626, ['shared', 'zeta']), (0.010536051565782637, ['shared']), (0.010536051565782637, ['shared']), (0.010536051565782637, ['shared'])]
```

- Not tested: Full application traffic, native-extension integration,
and other operating systems.

## Runtime Rollout Safety

- Rollout-managed feature(s): None; this is the pure Python BM25 scorer.
- Minimum rollout channel: Not applicable to this local scoring path.
- Stable/default behavior changed: Lower temporary memory use with
higher batch latency; scoring values are preserved.
- Kill switch / disable path: No new setting; use the previous
implementation if needed.
- Unsafe override required: No override is required.
- Qualification impact: No provider qualification changes.
- Rollback path: Revert this single-file change.

## Review Readiness

- [x] I have performed a self-review (automated source and result review
by a separate reviewer step; see AI assistance below).
- [x] Automated source and result checks completed.
- [x] This PR is ready for human review, with the latency tradeoff
stated above.

## Checklist

- [x] The modified file follows the project's style guidelines.
- [x] Source reviewed for duplicated scoring logic and unnecessary
complexity.
- [x] `CHANGELOG.md` was not edited.
- [ ] Full repository checks completed.

## AI assistance

This PR was prepared with AI assistance from Aiden. Candidates were
tracked with Weco Observe.

Weco share link:
https://dashboard.weco.ai/share/Y27llB_ia4h2OzafJay18VxKVJ0L4m8I

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4893"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Aiden <aiden@weco.ai>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:49:54 -06:00
xunWhaleandJerrettDavis c8168a0aa0 fix(cache): invalidate cached CCR markers after store eviction (#4872)
## Description

When `ContentRouter` caches a compressed output that carries a CCR
retrieval marker (`Retrieve more: hash=...`, `Retrieve original:
hash=...`, or `<<ccr:...>>`), a subsequent `compress()` /
`ContentRouter.apply()` call could hit Tier 2 `CompressionCache` and
return that marker even after the underlying `CompressionStore` entry
was evicted by `max_entries`, cleared, or expired. Any follow-up
retrieval for that marker then returned `None`.

This change:
1. Adds `CompressionCache.invalidate(key)` to evict a stale cached entry
without marking it skipped, and tracks verified CCR marker hashes per
cached entry (`_marker_hashes`) cleaned up alongside `_results`.
2. Records only marker hashes that actually existed in
`CompressionStore` at `put()` time via
`ContentRouter._collect_verified_marker_hashes`, preserving
compatibility with synthetic/mocked router tests that do not write to
`CompressionStore`.
3. Validates tracked marker hashes via
`ContentRouter._cached_markers_retrievable` on both Tier 2 cache hit
paths (`ContentRouter.apply` and
`ContentRouter._compress_block_content`), invalidating stale entries and
falling through to fresh compression so the original content is
re-stored and retrievable.

Closes #4796

*Disclosure: This fix and PR description were written by an AI agent (沈寻
/ Shen Xun, `@xunWhale`) on behalf of `@e0_7`, and verified locally with
pre-fix failure / post-fix pass pytest outputs, repo-wide ruff/mypy, and
the issue reproduction.*

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_router.py`: added `_CCR_MARKER_HASH_RE`,
`CompressionCache.invalidate()`, `CompressionCache.get_marker_hashes()`,
`_marker_hashes` tracking inside `CompressionCache`, and
`ContentRouter._collect_verified_marker_hashes()` /
`_cached_markers_retrievable()` checks on both string and content-block
Tier 2 cache hit paths.
- `tests/test_router_result_cache.py`: added 4 regression tests covering
`CompressionCache.invalidate()`, capacity eviction under `compress()`
for both `role="tool"` string messages and Anthropic `tool_result`
content blocks, and custom `CompressionStore.clear()` invalidation.

## Testing

- [x] Unit tests pass (`pytest` on router/cache/compression suites; see
scope note below)
- [x] Linting passes (`ruff check .` & `ruff format --check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Pre-fix baseline (`0368a87`, without `content_router.py` fix):
```text
$ PYTHONPATH=. pytest tests/test_router_result_cache.py -k "capacity_eviction or store_cleared or invalidate_drops" -v
FAILED tests/test_router_result_cache.py::test_invalidate_drops_cached_result_without_marking_skipped - AttributeError: 'CompressionCache' object has no attribute 'invalidate'
FAILED tests/test_router_result_cache.py::test_compress_repopulates_ccr_store_after_capacity_eviction - assert None is not None
FAILED tests/test_router_result_cache.py::test_compress_block_content_repopulates_ccr_store_after_capacity_eviction - assert None is not None
FAILED tests/test_router_result_cache.py::test_router_invalidates_cached_marker_when_store_cleared - TypeError: CompressionCache.put() got an unexpected keyword argument 'marker_hashes'
======================= 4 failed, 28 deselected in 0.39s =======================
```

Post-fix checks:
```text
$ ruff check .
All checks passed!

$ ruff format --check .
1940 files already formatted

$ mypy headroom
Success: no issues found in 583 source files

$ PYTHONPATH=. pytest tests/test_router_result_cache.py -v
...
tests/test_router_result_cache.py::test_invalidate_drops_cached_result_without_marking_skipped PASSED [ 90%]
tests/test_router_result_cache.py::test_compress_repopulates_ccr_store_after_capacity_eviction PASSED [ 93%]
tests/test_router_result_cache.py::test_compress_block_content_repopulates_ccr_store_after_capacity_eviction PASSED [ 96%]
tests/test_router_result_cache.py::test_router_invalidates_cached_marker_when_store_cleared PASSED [100%]
============================== 32 passed in 1.93s ==============================

$ PYTHONPATH=. pytest tests/test_router_result_cache.py tests/test_router_cache_policy.py tests/test_transforms_content_router.py tests/test_compression_store.py tests/test_compression_safety_rails.py tests/test_compression_determinism.py tests/test_compression_decision.py tests/test_compression_units.py tests/test_compression_batches.py tests/test_compression_cache.py tests/test_ccr_batch_store.py -q
======================= 423 passed, 2 skipped in 23.02s ========================
```
*(Scope note: The full repository test suite was not run locally because
optional extras such as `[proxy]`/`fastapi` are not installed in this
environment; `ruff check .`, `ruff format --check .`, and `mypy
headroom` ran across the entire repository.)*

## Real behavior proof

- Environment: Debian Linux (Python 3.11.2), Headroom `main` (`0368a87`)
with `headroom._core` from `headroom-ai==0.44.0`,
`InMemoryBackend(max_entries=2)`, `kompress_model="disabled"`.
- Exact command / steps: Ran the issue #4796 reproduction (`compress` on
3 distinct 251-line tool log payloads so entry 0 is capacity-evicted
from `CompressionStore`, followed by a repeat `compress(originals[0])`
and `store.retrieve(repeat_key)`).
- Observed result:
- Before patch: `before_repeat_retrieve: None`, `repeat_key == keys[0]:
True`, `store.retrieve(repeat_key): None`
- After patch: `before_repeat_retrieve: None`, `after_repeat_key_match:
True retrieved_present: True exact_match: True`
(`store.retrieve(repeat_key).original_content` equals
`originals[0][-1]["content"]`).
- Not tested: Distributed Redis `CompressionStore` backend, `[proxy]`
extra tests (`fastapi` not installed), or live upstream LLM API calls;
`kompress_model="disabled"` was used as in the issue reproduction.

## Runtime Rollout Safety

- Rollout-managed feature(s): None (bug fix in `ContentRouter` Tier 2
cache hit validation).
- Minimum rollout channel: N/A
- Stable/default behavior changed: Only invalidates cached CCR marker
hits when their store-backed CCR entry is no longer present in
`CompressionStore`.
- Kill switch / disable path: Standard `HEADROOM_COMPRESSION_CACHE`
disable / `invalidate`.
- Unsafe override required: No.
- Qualification impact: Prevents unrecoverable CCR markers on repeat
compression after store capacity eviction.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A —
internal cache bug fix, no public API or doc change)
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4872"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:26:07 -06:00
f4e3ad4b11 fix(content-router): keep list-form tool_result text blocks on separate lines (#4739)
## Description

A `tool_result` whose `content` is a list of several `text` blocks gets
flattened with `"".join(...)` in `ContentRouter` before the
read-protection check and before compression. MCP tools that return more
than one `TextContent` and litellm-style clients send this shape. When a
block does not end with a newline, the last line of one block and the
first line of the next one become a single line. The compressed result
is written back as one block, so the model sees a line the tool never
produced.

This PR joins the blocks with a newline unless the block already ends
with one, at both flatten sites.

Fixes #4729

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/content_router.py`: new helper
`_join_tool_result_text_blocks()` that joins list-form `text` blocks
with `\n`, skipping the separator when a block already ends with a
newline. Empty blocks add no extra line.
- Used at both places that flattened with `""`: the read-protection
check and the compression path. The wire shape does not change (the
existing `[{"type": "text", ...}]` re-wrap is kept), and results that
are not compressed keep their original blocks.
- `tests/test_transforms/test_content_router.py`: one `router.apply`
test with a two-block log `tool_result` that asserts that the output was
compressed and that every log line in it is a line the tool actually
returned, plus four parametrized cases for the helper.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Exact source: head `5305d8acb5f505e635d1287cae30ed7f8d9eb595`, base
`c05fbdddc873078dd97cb9ede841e75ef7ed0bef`. The PR has three commits:
`f273fbb9` (the fix, written on `1da8930a`) and two merges of `main`
(`2cba6c5b`, `5305d8ac`), both without conflicts. `git diff
c05fbddd...5305d8ac` still touches only the two files listed above.

Upstream CI on `5305d8ac`, all `completed` / `success`: CI
[43338122484](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122484),
Merge Conflicts
[43338122336](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122336),
Changelog Guard
[43338122370](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122370),
rust
[43338122318](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122318),
Evaluation Suite
[43338122592](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122592),
Security
[43338122460](https://github.com/headroomlabs-ai/headroom/actions/runs/43338122460).

Local runs below were built from `git archive 5305d8ac` in a
`python:3.13-slim-bookworm` image with `uv sync --frozen --extra proxy
--extra dev`. ruff ran on the two touched files and mypy on
`content_router.py`. pytest ran on the content router test modules. For
the reverse check, `content_router.py` was replaced with the copy from
`c05fbddd` and the tests kept.

A wider run over the 122 test files that touch the content router gave 9
failed / 2669 passed on `5305d8ac` and 9 failed / 2664 passed on
`c05fbddd`. The 9 failures are the same set on both (memory_bridge,
offline_egress docs, tabular; they come from the test image, not this
change), and the extra 5 passes are the new tests.

### Test Output

```text
== RUFF CHECK RC 0
== RUFF FORMAT RC 0
== MYPY RC 0
# new tests, 5 consecutive runs, all identical:
======================= 5 passed, 66 deselected in 5.01s =======================
# test_content_router.py + test_content_router_ccr_retrieve_exemption.py + test_content_router_shell_small_results.py:
============================= 125 passed in 5.61s ==============================
# with content_router.py restored to c05fbddd (fix removed):
E   AssertionError: assert ['2026-11-10T...er-0 retry 0'] == []
FAILED tests/test_transforms/test_content_router.py::test_list_form_tool_result_blocks_are_not_fused
FAILED tests/test_transforms/test_content_router.py::test_join_tool_result_text_blocks[texts0-a\nb\nc]
FAILED tests/test_transforms/test_content_router.py::test_join_tool_result_text_blocks[texts1-a\nb]
FAILED tests/test_transforms/test_content_router.py::test_join_tool_result_text_blocks[texts2-a\nb]
FAILED tests/test_transforms/test_content_router.py::test_join_tool_result_text_blocks[texts3-only]
================== 5 failed, 1 passed, 65 deselected in 1.88s ==================
```

The four `test_join_tool_result_text_blocks` cases fail on the base
because the helper does not exist there. The passing test in the last
run is the control
`test_anthropic_tool_result_lossy_without_marker_stays_verbatim`.

## Real Behavior Proof

- Environment: Linux container, Python 3.13, headroom installed from
this branch with `uv sync --frozen --extra proxy`. Upstream is a local
stdlib HTTP server on 127.0.0.1:9911 that records the `/v1/messages`
body it receives and returns a fixed `ok` message. Dummy API key, no
real provider.
- Exact command / steps: `python -m headroom.cli proxy --host 127.0.0.1
--port 8799 --no-telemetry --no-rate-limit --anthropic-api-url
http://127.0.0.1:9911`, run once with the default mode and once with
`--mode token`. Then POST `/v1/messages` with a `tool_result` whose
content is two `text` blocks of 150 log lines each, the first block not
ending in a newline. Same steps against `main` for comparison.
- Observed result: the proxy compressed the tool result in all four runs
(`content_router: 3 msgs — 1 compressed (tool:log:0.03)`) and the client
got `200` with `ok`. On `main` the body forwarded upstream had 1 fused
line, `2026-11-10T10:29:00Z INFO worker-2 processed batch 149
ok2026-11-10T11:00:00Z WARN worker-0 retrying batch 0`. With this patch
it had 0 fused lines in both modes.
- Not tested: a real Anthropic or OpenAI endpoint, SSE streaming
responses, Bedrock, the OpenAI `/v1/chat/completions` handler (it goes
through the same `ContentRouter`), Windows and macOS.

## Runtime Rollout Safety

- Rollout-managed feature(s): N/A, this is a bug fix in the existing
compression path with no flag.
- Minimum rollout channel: N/A, no rollout-managed feature.
- Stable/default behavior changed: Yes, only for list-form `tool_result`
content with more than one `text` block: the blocks are now joined with
a newline instead of nothing before compression.
- Kill switch / disable path: N/A, no flag. Disabling compression for
the tool, as today, skips this path.
- Unsafe override required: No.
- Qualification impact: None expected. Single-block and string
`tool_result` content takes the same path as before.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`; it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Documentation is N/A, there is no user-facing change to document. I
checked the other provider paths: OpenAI `role: "tool"` messages with
list content are handled block by block and are not joined, so they do
not have this bug. "New and existing unit tests pass locally" refers to
the content router test modules listed above, run in the container.

Written with help from Claude Code.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4739"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:25:28 -06:00
3aafdf2dcc fix(router): keep source code out of the prose (Kompress) fallbacks (#4787)
## Description

When the code compressor declined a block of source code, the router
passed the block to Kompress. Kompress is a prose model. It drops
low-salience words, and in code those words are keywords such as `for`,
`in`, `if` and `import`. The result no longer parses. #4214 shows a
fenced Python script that comes back with `for` and `in` removed.

This PR stops that in the Python router and in the Rust live-zone
dispatcher. When the code compressor declines, the code is forwarded
unchanged.

Closes #4214

Credit: the Python change is oxura's work from
https://github.com/oxura/headroom/tree/fix/fenced-code-routing-4214-20261018
(277ca1f). Their commit is the first commit here, rebased onto `main`,
and oxura is its author. The test, docs and Rust commits were added on
top. oxura asked us to open this PR from our fork
([comment](https://github.com/headroomlabs-ai/headroom/issues/4214#issuecomment-6797572655)),
and said they have not run this branch, so the model and native results
below are ours, not theirs.

## Behaviour change to review

With `enable_code_aware=False` (the default), source code now stays
unchanged. Before this PR, it was sent to Kompress and lost keywords.
With `enable_code_aware=True`, code that the code compressor declines
also stays unchanged. Before, Kompress ran on it.
A code block the code compressor does compress is unaffected. Prose,
logs, JSON and tables are unaffected.

Side effect: a code block that Kompress used to shrink (and break) now
keeps its full size. This is the intended trade, but it means fewer
tokens saved on code when `enable_code_aware=False`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update
- [ ] New feature
- [ ] Breaking change (the default-config behaviour change above is
intended, and it only stops damage to code)

## Changes Made

Five commits:

- `596764c3` `fix(router): keep source code out of prose fallbacks`.
Author: oxura. Removes the CODE_AWARE to Kompress fallbacks in
`headroom/transforms/content_router.py`. This is 277ca1f rebased onto
`main`, and it differs from 277ca1f in two ways only: the two test-file
conflicts are resolved, and its `wiki/proxy.md` hunk is dropped, because
`AGENTS.md` says not to add content under `wiki/`.
- `7705a4e6` `test(router): repin the code-to-Kompress tests and add the
#4214 fence regression`. Author: Tao Hansen, with a `Co-authored-by:
oxura` trailer. Five tests asserted the old fallback. They are rewritten
to assert the new behaviour (code unchanged, Kompress never sees it)
instead of being deleted. Adds a regression test for the script from
#4214, with `enable_code_aware` both off and on. Updates
`docs/content/docs/code-compression.mdx` and
`docs/content/docs/how-compression-works.mdx`, which said that detected
code goes to Kompress.
- `6631668a` `fix(live-zone): leave source code unchanged when the code
compressor declines`. Author: Tao Hansen. Same fix in
`crates/headroom-core/src/transforms/live_zone.rs`: a declined
`SourceCode` block is now a NoOp, not a Kompress input. Tests:
`tests/live_zone_source_code_decline.rs` (single test, needs no model),
and new cases in `tests/live_zone_dispatch.rs`.
- `d8776283` `fix(router): turn off the code compressor's internal
Kompress fallback`. Author: Tao Hansen. Follows review: the router's
code compressor was still built with
`fallback_to_kompress=self.config.enable_kompress` (#4719), so unknown
languages, Perl and AST failures could build a `KompressCompressor`
inside `CodeAwareCompressor`. The router now always sets
`fallback_to_kompress=False`. The #4719 test that pinned `True`
(`test_enabled_kompress_keeps_code_aware_fallback`) is repinned as
`test_enabled_kompress_still_turns_off_code_aware_fallback`, and the
unknown-language, Perl (by language hint, since `detect_language()`
reports Perl-dominant code as unknown) and AST-failure paths are tested
with spies on `is_kompress_available` and `KompressCompressor`, with
`enable_kompress` both off and on.
`docs/content/docs/code-compression.mdx` notes that the router always
turns the option off.
- `c1f8464e` `fix(router): keep passthrough code from external
compressors and stale caches`. Author: Tao Hansen. Follows review:
`_try_external_compressor` skips `PASSTHROUGH` blocks (code with
`prefer_code_aware_for_code=False`, whole or as a mixed section), which
would otherwise be offered to a prose-only external compressor as
`text/plain`. The disk cache namespace now also hashes
`_ROUTING_POLICY_REVISION`, so a same-version restart does not replay
code that Kompress damaged before this change. Tests: two
external-dispatch cases with a `text/plain` external, and one namespace
test. All three fail on the previous head.

I did not touch `CHANGELOG.md`.

## Testing

- [x] Unit tests pass (`pytest`): see below, the failures are
environment failures that also fail on `main`
- [x] Linting passes (`ruff check .`, `ruff format --check .`)
- [x] Type checking passes (`mypy
headroom/transforms/content_router.py`)
- [x] New tests added for new behaviour
- [x] Manual testing performed (real model replay, below)

### Test Output

macOS, Python 3.14. Head is now `c1f8464e`.

```text
After the second review fix (c1f8464e): router, code-aware, code-compressor, cache, external-dispatch and compressor-selection tests (107 files): 2106 passed, 13 skipped. ruff clean.
After the rebase onto main c05fbddd and the review fix (d8776283):
  router, code-aware and code-compressor tests (37 files): 743 passed, 2 skipped
  tests/test_code_aware_kompress_fallback.py: 8 passed; with the router change reverted, 4 fail
    (the enable_kompress=True cases: config pin, unknown language, Perl, AST failure)
  ruff check / ruff format --check headroom tests: clean
  cargo test -p headroom-core: 1151 passed, 0 failed
Earlier run on 9df60c7c (before the rebase):
router tests (28 files incl. both changed files): 557 passed, 2 skipped, 1 failed:
  test_content_router_ml_stage_deadline.py::test_non_ml_prework_does_not_consume_the_ml_budget,
  a timing test the branch does not touch; it passes 3 of 3 times when rerun alone
ruff check .:           All checks passed!
ruff format --check headroom tests: 1714 files already formatted
mypy headroom/transforms/content_router.py: Success: no issues found in 1 source file
cargo test -p headroom-core:   22 test binaries, 1151 passed, 0 failed
cargo fmt --check / cargo clippy -p headroom-core --tests -D warnings: clean
pytest tests (full):     31 failed, 19208 passed, 513 skipped
```

Rust bug-fix tests, fail before and pass after (the router-test commit's
source with the new Rust tests, then the live-zone commit; measured
before the rebase as `9df60c7c`, and the Rust code is unchanged since):

```text
before: declined_source_code_never_starts_kompress ... FAILED (Kompress init ran once, expected 0; rerun on 7705a4e6)
        source_code_declined_by_code_compressor_is_never_kompressed ... FAILED (model loaded: Go source was Kompressed)
after:  all 11 live_zone_dispatch tests ok; live_zone_source_code_decline ok
```

The dispatch fail-before was measured on the earlier, code-identical
branch (only documentation files differ). The dispatch test only fails
before the fix when the Kompress model is loaded (`HF_HOME` with the
model, `ORT_DYLIB_PATH` set). The single-test binary fails before the
fix on any host.

Python regression: with the new tests run against `origin/main`'s router
(no fix), 9 test cases fail (the #4214 fence test in both configs, the
repinned tests, and the fenced-code keyword test). With the fix they
pass.

### Failures in the full `pytest tests` run

The full run on the branch had 31 failures, none in router or
compression code. I compared them with pristine `origin/main`
(200c838c):

- 11 `tests/test_proxy_uds.py` tests fail on `main` too. The socket path
is too long under this checkout directory (`OSError`). They fail the
same way in isolation on `main`.
- 17 other failures are in `test_proxy_health.py`,
`test_read_maturation_handler_nobust.py`, `test_stateless_no_disk.py`,
`test_memory_bridge.py`, `test_cli/*`, `test_image_compression*.py`,
`test_offline_egress_chokepoint.py` and `test_proxy_scalability.py`. A
full run on `main` fails 29 tests, and the set differs from run to run
(the two sets overlap by 17 of 31). I re-ran the 14 failures that were
not in the `main` list in isolation on the branch: 12 pass, the other 2
are the UDS tests above.
- Known flaky test:
`tests/test_content_router_ml_stage_deadline.py::test_saturated_remote_calls_fail_open_without_queueing_their_payloads`
failed once in a loaded run and passes on rerun. File repeated 3 times
on `596764c3` and on `origin/main`: 16 passed each time. It is timing
based and not caused by this change. It passed in the 558-test router
run above.

## Real Behavior Proof

- Environment: macOS arm64, Python 3.14 in the checkout's own `.venv`,
real `kompress-v2-base` int8 ONNX model from an offline copy of the
public HF cache (`HF_HOME`, `HF_HUB_OFFLINE=1`), Rust `ort` via the
venv's onnxruntime.
- Fence: a 1324-character valid Python script. The issue quotes only
four of its lines, so the rest is a reconstruction in the same style,
not the reporter's bytes.
- Exact command / steps: `ContentRouter.compress` on the fence followed
by about 40 lines of prose, for four configs (default;
`enable_code_aware=True`; `prefer_code_aware_for_code=False`;
`enable_code_aware=True` with lossless-then-lossy). Also
`headroom.compress` on a full Anthropic message list with the fence in a
`tool_result`.
- Observed result: Before (v0.44.0): in the default and code-aware
configs the fence comes back as 1144 characters and does not parse
(`import json, collections` becomes `json, collections`; `[s for s in
...]` becomes `[s s in ...]`). After (`9df60c7c`): in all four configs
and in `headroom.compress`, the fence is byte-identical, 1324
characters, and parses. Prose control: the same router on prose still
compresses. A prose-only block went from 5709 to 5357 characters
(strategy TEXT). A prose / fence / prose block went from 12760 to 12056
characters (strategy MIXED), and the fence in it is byte-identical and
parses.
- Not tested: the reporter's exact 1324-character fence (only four lines
are public, so the tests use the reconstructed fence described above);
the Rust live-zone path end to end through the proxy with a real request
(covered by `cargo test` only); platforms other than macOS; the full
proxy on a live deployment.

## Other places where code can still reach a prose model

I checked every path from code to a prose or ML compressor, in Python
and Rust.

| Path | State |
| --- | --- |
| Python `CODE_AWARE` branch, code compressor declines or is disabled |
Fixed here: the block stays unchanged |
| Python post-strategy fallback after "no savings" (`content_router.py`,
`fallback_eligible_strategy`) | Never applied to `CODE_AWARE`. It only
applies to `SMART_CRUSHER` and `CONFIG` (JSON and config files), so
unchanged |
| Python lossless-then-lossy (opt-in) after the lossless fold |
`CODE_AWARE`, `DIFF` and `PASSTHROUGH` are excluded, so unchanged |
| Python lossless-then-lossy layer for `SEARCH`, `LOG`, `HTML` (opt-in)
| Prose-like output, not code. Unchanged |
| Python relevance split for `LOG` and `SEARCH` | Not code. Unchanged |
| Python mixed content with fenced code (`_compress_mixed`) | Each fence
goes through the `CODE_AWARE` branch, so it is fixed by this change.
Covered by the new fence tests |
| Python `prefer_code_aware_for_code=False` | Already PASSTHROUGH before
this PR |
| Python `_experimental_compress_read`
(`HEADROOM_EXPERIMENTAL_READ_KEEP_RATIO`) | Deliberate opt-in experiment
that runs Kompress on file reads, which can be code. Off by default.
Left as is |
| Python size gate and deadline fallbacks in `_try_ml_compressor` | Run
only after a caller chose Kompress, so code no longer reaches them |
| Python `CodeAwareCompressor` internal fallback
(`fallback_to_kompress`: unknown language, Perl, no tree-sitter, AST
failure) | Fixed in commit 4: the router always builds it with
`fallback_to_kompress=False`. Covered by Kompress-spy tests |
| Python external compressor dispatch for `PASSTHROUGH` code | Fixed in
commit 5: never offered to an external compressor |
| Disk result cache made before this change | Fixed in commit 5: a new
namespace through `_ROUTING_POLICY_REVISION` |
| Rust `live_zone.rs` `SourceCode` arm | Fixed here (commit 3) |
| Rust `live_zone.rs` `PlainText` arm to Kompress | Only plain text. It
is the single remaining call to `kompress_or_noop` |
| Rust: prose text with an embedded fenced code block | **Not
reproduced.** I built synthetic tool results of prose around the fenced
Python script, at 6 KB, 20 KB and 71 KB. On this branch the dispatcher
classified each one as `source_code`, so commit 3 leaves it unchanged,
and Kompress was never consulted (`kompress_init_runs() == 0`). Rust has
no fence-aware splitting, but I found no input where that sends code to
Kompress, so this PR files no follow-up. |
| Python `config_compressor.py` docstring says YAML/TOML/INI "fall
through to the lossy prose path" | Documentation only: a stale comment,
not a code path. Config content has its own `CONFIG` strategy. Can be
refreshed in a later change |

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, in the default config code is no
longer sent to Kompress (see "Behaviour change to review"). Tokens saved
on code drop in return for code that stays valid.
- Kill switch / disable path: none added. Revert the PR to restore the
old fallback.
- Unsafe override required: no.
- Qualification impact: none expected. Compression of code by the code
compressor is unchanged.
- Rollback path: revert the three commits. No data, schema or config
migration.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines (`ruff`, `cargo
fmt`, `clippy` clean)
- [x] I have made corresponding changes to the documentation
(`docs/content/docs/code-compression.mdx`,
`docs/content/docs/how-compression-works.mdx`)
- [x] I have added tests that prove my fix is effective
- [x] I did **not** edit `CHANGELOG.md`
- [ ] New and existing unit tests pass locally: the only failures are
environment failures that also fail on `main`, listed above

## Additional Notes

- Rust has no fence-aware splitting like the Python router has. A
synthetic prose-plus-fence repro did not reach Kompress (see the table
above), so no follow-up issue is filed.
- One stale comment remains in `content_router.py`
(`_try_ml_compressor`, the Phase 0 size-gate note lists
`CODE_AWARE->KOMPRESS` as an entry point). It is comment-only; not
changed here to keep the review commits as they are. Say if you want it
fixed in this PR.
- Reviewers: the Python change is easiest to review commit by commit.
Commit 1 is oxura's change, and commits 2 and 3 are the tests, docs and
the Rust fix.

---------

Co-authored-by: oxura <grozafroza44@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:25:06 -06:00
Ayush Kumar JhaandJD Davis cf3284675e fix(lossless): never grep-fold Go, nginx, klog or access-log timestamps (#4901)
## Description

The lossless search fold splits each line as `path:line:content` and
factors a repeated `path` into a header. `_TIMESTAMP_ROW_RE` (#3419)
keeps timestamped log lines out of it, because a log line has the same
colon shape: folding it hoists the date and hour into a fake file header
and every row starts mid-clock. The fold round-trips byte-exactly, so
the inverse check in `compact_lossless` cannot catch it.

The guard knew ISO, `MM/DD/YYYY`, syslog and bare-clock prefixes, but
not these four common formats:

| Format | Line | Split as |
|---|---|---|
| Go `log` package | `2026/09/02 14:30:00 ERROR ...` | `2026/09/02 14` /
`30` / `00 ERROR ...` |
| nginx error log | `2026/09/02 14:30:00 [error] ...` | same |
| klog (Kubernetes components) | `E0902 14:30:00.000000 1
reflector.go:100] ...` | `E0902 14` / `30` / `00.000000 ...` |
| common/combined access log | `10.0.0.7 - - [02/Sep/2026:14:30:00
+0000] ...` | `10.0.0.7 - - [02/Sep/2026` / `14` / `30:00 ...` |

They reach the fold two ways: the bash-search fold (`grep ERROR
api.log`) and the router's lossless-first stage, which tries the search
fold on any tool output (`cat`, `tail`, `kubectl logs`).

This adds those three prefix shapes to `_TIMESTAMP_ROW_RE`, the same
place #3419 put the guard. The content detector (#3748) and the
mixed-content splitter (#4463) share the regex; neither claimed these
lines before, so for them only false positives could change, and there
are none on real grep output (below).

The access-log alternative differs from the one I suggested in #4900.
That one (`\S+ \S+ \S+ \[...`) also matched a real `grep -n` row over an
access log (`/var/log/nginx/access.log:12:10.0.0.7 - - [02/Sep/...`),
because `\S+` swallows `path:12:10.0.0.7`. The fields now exclude `:`,
which a mis-split line cannot contain before its first colon anyway. The
host field takes name/IPv4 characters only (`[\w.-]`), capped at 255 (a
DNS name is at most 253). With `[^\s:]` the check scanned every grep
row's whole path, and fold + detection on 10k grep rows took 10% and 32%
longer than main in two interleaved runs; with this class and cap it
stops early on a path and on a long unbroken line such as a base64 blob.

Closes #4900

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/lossless_compaction.py`: three alternatives in
`_TIMESTAMP_ROW_RE`: `YYYY/MM/DD hh:mm` (Go log, nginx error log),
`[IWEF]MMDD hh:mm` (klog/glog), and the access-log prefix `host ident
user [dd/Mon/yyyy:hh:mm` with a `[\w.-]{1,255}` host.
- `tests/test_lossless_timestamp_guard.py`: the four formats, plus an
access log with a 253-character host, join
`test_timestamped_logs_are_never_folded`. New
`test_grep_rows_over_those_logs_still_fold`: real `grep -n` rows over
the same logs (including a dated `2026/09/02/api.log` path) still fold
under their file header.
- `tests/test_bash_search_lossless_fold.py`:
`test_go_log_keeps_its_timestamps` runs a Go log through `ContentRouter`
as the output of `grep ERROR /var/log/api.log` (bash-search fold) and of
`cat /var/log/api.log` (lossless-first stage).

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# this branch's tests against main's lossless_compaction.py (main 66e00869):
$ pytest -q -rf tests/test_lossless_timestamp_guard.py "tests/test_bash_search_lossless_fold.py::test_go_log_keeps_its_timestamps"
FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[go-log]
FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[nginx-error]
FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[klog]
FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[access-log]
FAILED tests/test_lossless_timestamp_guard.py::test_timestamped_logs_are_never_folded[access-log-longest-host]
FAILED tests/test_bash_search_lossless_fold.py::test_go_log_keeps_its_timestamps[grep ERROR /var/log/api.log]
FAILED tests/test_bash_search_lossless_fold.py::test_go_log_keeps_its_timestamps[cat /var/log/api.log]
======================== 7 failed, 12 passed in 19.98s ========================
# same command on this branch:
============================= 19 passed in 20.52s =============================

# mutations of the fix, each run against the same tests:
== M1 drop Go/nginx alt: ======================== 4 failed, 15 passed in 21.51s ========================
== M2 drop klog alt: ======================== 1 failed, 18 passed in 15.91s ========================
== M3 drop access-log alt: ======================== 2 failed, 17 passed in 13.28s ========================
== M4 access-log alt as proposed in issue (\S+): ======================== 1 failed, 18 passed in 14.13s ========================
    FAILED tests/test_lossless_timestamp_guard.py::test_grep_rows_over_those_logs_still_fold[access-log]
== M5 host cap too small {1,15}: ======================== 1 failed, 18 passed in 13.45s ========================
== M6 host cap removed: ============================= 19 passed in 12.49s =============================
== M7 host class widened to [^\s:]: ============================= 19 passed in 12.55s =============================

# false positives, the measure used on #3419: every line of every tracked text file in
# this repo as a real grep row (git grep -n -I -e ''), old guard vs this branch:
lines=977737 grep_rows=977737 rejected_by_old_guard=0 rejected_by_new_guard=0 newly_rejected=0

# cost (M6/M7 survive because the cap and class only bound the scan):
main   fold+detect+split, 10k real grep rows: median 37.4 ms (min 34.7)
final  fold+detect+split, 10k real grep rows: median 38.7 ms (min 37.1)
1MB word-char line main   0.000 ms per match
1MB word-char line final  0.001 ms per match
1MB base64 line    main   0.000 ms per match
1MB base64 line    final  0.000 ms per match
# an earlier run with the host as unbounded [^\s:]+ (not used):
1MB word-char line   unbounded        5.571 ms per match
1MB base64 line      unbounded        6.566 ms per match

# full suite on this machine: pytest tests scripts/tests -n 8 --continue-on-collection-errors
# main 5023c84e:
= 459 failed, 18027 passed, 1154 skipped, 42 warnings, 417 errors in 9256.57s (2:34:16) =
# this branch on 66e00869:
= 454 failed, 18085 passed, 1154 skipped, 39 warnings, 417 errors in 2921.98s (0:48:41) =
# 8 failures appear only in the branch run. Each rerun 3x serially on main 66e00869 and on the branch:
[wtmain] ppp  tests/test_fastembed_pinned_snapshot.py::test_token_fixture_matches_huggingface_hub[HF_TOKEN wins over the login file]
[wtmain] ppp  tests/test_image_compression_isolation.py::test_one_timeout_does_not_destroy_a_concurrent_request
[wtmain] ppp  tests/test_offline_egress_chokepoint.py::TestLiteLLMCostMapOffline::test_import_with_offline_set_after_package_load_opens_no_socket[headroom.backends.litellm]
[wtmain] ppp  tests/test_openai_chat_buffered_ccr.py::test_a_200_with_no_usable_choice_is_not_replayed_as_an_empty_stream[payload7]
[wtmain] ppp  tests/test_openai_chat_buffered_ccr.py::test_failed_continuation_fails_closed_with_its_status[429-rate_limit_error]
[wtmain] ppp  tests/test_read_maturation_handler_nobust.py::test_quiesce_turns_config_is_honored
[wtmain] FFF  tests/test_stage_timer.py::test_concurrent_measure_calls_are_independent
[wtmain] ppp  tests/test_stateless_no_disk.py::test_cli_stateless_flag_exports_env_for_children
[wt4900] ppp  tests/test_fastembed_pinned_snapshot.py::test_token_fixture_matches_huggingface_hub[HF_TOKEN wins over the login file]
[wt4900] ppp  tests/test_image_compression_isolation.py::test_one_timeout_does_not_destroy_a_concurrent_request
[wt4900] ppp  tests/test_offline_egress_chokepoint.py::TestLiteLLMCostMapOffline::test_import_with_offline_set_after_package_load_opens_no_socket[headroom.backends.litellm]
[wt4900] ppp  tests/test_openai_chat_buffered_ccr.py::test_a_200_with_no_usable_choice_is_not_replayed_as_an_empty_stream[payload7]
[wt4900] ppp  tests/test_openai_chat_buffered_ccr.py::test_failed_continuation_fails_closed_with_its_status[429-rate_limit_error]
[wt4900] ppp  tests/test_read_maturation_handler_nobust.py::test_quiesce_turns_config_is_honored
[wt4900] FFF  tests/test_stage_timer.py::test_concurrent_measure_calls_are_independent
[wt4900] ppp  tests/test_stateless_no_disk.py::test_cli_stateless_flag_exports_env_for_children

$ ruff check .   (0.17.4)
All checks passed!
$ ruff format --check .   (0.17.4)
1954 files already formatted
$ mypy headroom/transforms/lossless_compaction.py headroom/transforms/content_detector.py headroom/transforms/mixed_content.py --ignore-missing-imports --platform linux --follow-imports=silent   (1.20.2)
Success: no issues found in 3 source files
```

`mypy headroom` is unchecked because it is not clean on this machine for
main either. With mypy 1.20.2 from the pre-commit hook and `--platform
linux`, main and this branch both report the same single error,
`upstream_guard.py:162` (`no-any-return`), which comes from that
environment having no `httpx` installed. Without `--platform linux`,
both also report the Windows-only `os` attribute errors from #4520.

## Real Behavior Proof

- Environment: Windows 11, Python 3.11.9, this repo at 5023c84e (main)
and this branch on the same base via `PYTHONPATH=.` (the branch was then
rebased onto 66e00869, whose three new commits touch only OpenCode
files), Rust core taken from the released headroom-ai 0.44.0 wheel (main
is 0.44.0), `HEADROOM_OFFLINE=1`, pure-Python content detection (the
default on Windows).
- Exact command / steps: the reproduction script from #4900
(`compact_lossless(text, "search")` on 30 lines of each format), then an
Anthropic `Bash` `tool_use` plus a 40-line `tool_result` through
`ContentRouter(ContentRouterConfig()).apply(...)` for `kubectl logs ...`
(klog), `tail -n 200 /var/log/nginx/access.log`, `cat /var/log/api.log`
and `grep ERROR /var/log/api.log` (Go log).
- Observed result: on main each format folds into a fake header
(`2026/09/02 14`, `E0902 14`, `10.0.0.7 - - [02/Sep/2026`) and each
router case applies `lossless_search`; on this branch every case comes
back byte-identical with its timestamps, and the router reports
`router:noop`.
- Not tested: a live proxy session with a real client and model
(in-process `ContentRouter` only); Linux and macOS (the change is a
pure-Python regex; CI covers ubuntu); the Rust core has no copy of this
fold, so there is nothing to port.

```text
main (5023c84e):
--- go log: 1489 -> 1083 chars
2026/09/02 14
30:00 ERROR upstream timeout req=0
--- klog: 2359 -> 2098 chars
E0902 14
30:00.000000       1 reflector.go:0] failed to list *v1.Pod: timeout
--- access log: 2359 -> 1605 chars
10.0.0.7 - - [02/Sep/2026
14:30:00 +0000] "GET /api/items/0 HTTP/1.1" 500 512
klog     N=40 5399->5048 fake_header=True first='E0902 14'
          transforms: ['router:tool_result:lossless_search']
access   N=40 3829->2815 fake_header=True first='10.0.0.7 - - [02/Sep/2026'
          transforms: ['router:tool_result:lossless_search']
go       N=40 3079->2533 fake_header=True first='2026/09/02 14'
          transforms: ['router:tool_result:lossless_search']
go-grep  N=40 3079->2533 fake_header=True first='2026/09/02 14'
          transforms: ['router:bash:lossless_search']

this branch:
--- go log: 1489 -> 1489 chars
2026/09/02 14:30:00 ERROR upstream timeout req=0
--- klog: 2359 -> 2359 chars
E0902 14:30:00.000000       1 reflector.go:0] failed to list *v1.Pod: timeout
--- access log: 2359 -> 2359 chars
10.0.0.7 - - [02/Sep/2026:14:30:00 +0000] "GET /api/items/0 HTTP/1.1" 500 512
klog     N=40 5399->5399 fake_header=False first='E0902 14:30:00.000000       1 reflector.go:100] failed to li'
          transforms: ['router:noop']
access   N=40 3829->3829 fake_header=False first='10.0.0.7 - - [02/Sep/2026:14:30:00 +0000] "GET /api/items/0 '
          transforms: ['router:noop']
go       N=40 3079->3079 fake_header=False first='2026/09/02 14:30:00 ERROR handler: upstream timeout after 90'
          transforms: ['router:noop']
go-grep  N=40 3079->3079 fake_header=False first='2026/09/02 14:30:00 ERROR handler: upstream timeout after 90'
          transforms: ['router:noop']
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: N/A (no feature flag involved).
- Stable/default behavior changed: Yes, narrowly. Lines that start with
a Go/nginx, klog or access-log timestamp no longer take the grep fold;
real grep output folds as before (0 of 977,737 real grep rows newly
rejected).
- Kill switch / disable path: `HEADROOM_LOSSLESS_COMPACTION=0` disables
every lossless fold, as before.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: N/A, the guard is internal and no docs describe its
formats.
- `Unit tests pass` and `New and existing unit tests pass locally` are
unchecked because the full suite is not clean on this Windows machine
for main either. Across the 28 test files that reference the fold, the
guard, the detector or the mixed-content splitter, the branch run fails
the same 8 tests as main (7 in `test_image_compressor.py`, 1 in
`test_garbled_compression_fixes.py`) and nothing else; the full-suite
comparison is in Test Output.
- Ruby's default `Logger` format (`E, [2026-09-02T14:30:00.123456 #4321]
ERROR -- : ...`) folds the same way (`I, [2026-09-02T14` header). It is
not in #4900, so I left it out; it is one more alternative if you want
it here.
- Logs with ANSI color codes before the timestamp still fold under the
`search` kind, for every format including ISO. That predates this
change.
- An access log whose client is an IPv6 address with an all-digit second
group (`2001:4860:4860::8888 - - [...]`) splits on the address itself,
not the timestamp, and folds under a `2001` header on main and on this
branch. Left for a separate change.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4901"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:20:01 -06:00
0d203db6b8 fix(copilot): keep new VS Code chats on the Local harness so they reach the proxy (#4899)
## Description

VS Code can run a Copilot chat on its "Copilot harness" (Agent Host, the
bundled Copilot SDK runtime) instead of the Local harness. That runtime
ignores `github.copilot.advanced.debug.overrideCapiUrl` /
`overrideProxyUrl`, so the chat goes straight to GitHub. Copilot's small
background calls (title, intent) still reach the proxy, so the proxy
looks connected while the main conversation never arrives. The harness
default (`chat.defaultToCopilotHarness`,
`chat.editor.preferCopilotHarness`) is tagged `experiment: startup` in
VS Code, so it can be flipped per user by rollout.

`headroom wrap vscode` now pins new chats to the Local harness.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/copilot/vscode.py`: the managed block also writes
`"chat.defaultToCopilotHarness": false` and
`"chat.editor.preferCopilotHarness": false`. Because they sit inside the
marker block, `headroom unwrap vscode` removes them. A harness setting
the user already set outside the block is left alone (no duplicate key).
New helper `copilot_harness_settings_enabled(path)` reports a user value
of `true`.
- `headroom/cli/wrap.py`: one line saying new chats stay on the Local
harness and existing chats keep their engine; a warning instead when the
user's own setting is `true`. `--no-configure` prints the two extra
settings.
- `docs/content/docs/vscode-copilot.mdx`: documents the harness settings
and the troubleshooting case.
- Blocks written by older versions (including ones with
`overrideAuthType`) are replaced by marker as before.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
tests/test_provider_copilot_vscode_config.py + tests/test_cli/test_wrap_vscode.py: 32 passed.
Against origin/main source: 7 of those fail (block contents, old-block upgrade, user-set key [True]/[False], wrap output, --no-configure output).
ruff 0.17.4 check + format --check on the 5 changed Python files: clean.
mypy on the changed source files: 8 errors, all in headroom/cache/prefix_tracker.py and headroom/ccr/mcp_server.py, identical on origin/main; none in changed files.
```

## Real Behavior Proof

- Environment: macOS 27.2.1, Python 3.12.13, VS Code 1.145.2 with GitHub
Copilot signed in, core run from this branch, real GitHub Copilot
upstream. Real user `settings.json` (backed up and restored afterwards).
- Exact command / steps:
  1. `headroom wrap vscode -p 8795` from this branch.
2. `code chat -n -m agent "Reply with the single word OK and nothing
else."` (opens a new VS Code window and sends one Agent-mode chat).
3. Read the proxy's `/stats`, VS Code's `agenthost.log`, then `headroom
unwrap vscode`.
- Observed result:
  ```text
VS Code Copilot proxy settings updated: ~/Library/Application
Support/Code/User/settings.json
Keep using Copilot's normal model picker; the selected model is
preserved.
New chats stay on VS Code's Local harness; a chat keeps the engine it
started with, so start a new chat.

  settings.json after wrap:
  {
  	// --- Headroom Copilot proxy ---
"github.copilot.advanced.debug.overrideProxyUrl":
"http://127.0.0.1:8795/p/win-local",
"github.copilot.advanced.debug.overrideCapiUrl":
"http://127.0.0.1:8795/p/win-local",
  	"chat.defaultToCopilotHarness": false,
  	"chat.editor.preferCopilotHarness": false
  	// --- end Headroom Copilot proxy ---
  }

  proxy requests before chat: 0 {}
proxy requests after one Agent-mode chat in a new VS Code window: 4
{'gpt-4o-mini-2024-07-18': 3, 'claude-opus-5.5': 1}
  agent host turns during the chat: 0

Removed Headroom Copilot proxy settings from ~/Library/Application
Support/Code/User/settings.json
  settings.json after unwrap:
  {
  }
  ```
Separately, on the same machine with the settings edited by hand (same
VS Code, same proxy, new window each time):
`"chat.defaultToCopilotHarness": true` -> the Agent-mode chat ran in the
Agent Host (provider `copilotcli`) and 0 requests reached the proxy;
`"chat.defaultToCopilotHarness": false` +
`"chat.editor.preferCopilotHarness": false` -> Local harness, 4 requests
reached the proxy including the main `/v1/messages` call.
`"chat.temporaryDefaultToLocalHarness": true` as a user setting had no
effect, so it is not written.
- Not tested: a VS Code where Microsoft's rollout has flipped the
default (the explicit user value is expected to win, but that could not
be reproduced); an organization that enforces managed sandbox settings,
where VS Code can require the Copilot harness regardless; VS Code
profiles, workspace-level settings, Insiders/VSCodium; Windows and
Linux; the persistent `headroom start --persistent --vscode` path in the
Enterprise client.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: `headroom wrap vscode` writes two
more VS Code user settings inside its managed block. No proxy behaviour
changes.
- Kill switch / disable path: `headroom unwrap vscode` removes them; a
user who sets either key themselves keeps their value.
- Unsafe override required: no.
- Qualification impact: none expected.
- Rollback path: revert this commit; `unwrap` removes blocks of either
format.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- This keeps users off the Copilot harness; it does not route that
harness through Headroom. Supporting it is separate work.
- Only the configured settings file is checked for a user value. A
`true` in workspace settings, another profile, or policy is not
detected.
- After `unwrap` the keys return to VS Code's default, which a rollout
may move to the Copilot harness.
- Existing chats keep the harness they started with; the wrap output
says to start a new chat.
- `e2e/copilot_live.py` and `e2e/wrap/run.py` were not run.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4899"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:19:43 -06:00
Yuchuan HeandJD Davis b560cba927 fix(memory): parse frontmatter delimiters on complete lines (#4886)
## Description

A memory beginning with `Use foo---bar as the separator` writes that
text into the generated name/description. The reader takes the embedded
`---` as the closing delimiter, mixes frontmatter into the body and
changes the content hash. The next sync imports a polluted duplicate.

Recognize opening and closing frontmatter delimiters only on complete
lines, including LF and CRLF input. Preserve the original text if the
header is malformed or has no closing delimiter.

Related to #4832's full-store sync work; this fixes a separate contract
on the current main snapshot.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Require complete delimiter lines; preserve bodies containing embedded
dashes and malformed headers.
- Add regressions against the unmodified base
`5023c84e527e28281dd73d1ba04184fe738aac0f` and real file/component
coverage.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook with httpx
installed)
- [x] New tests added
- [ ] Manual testing performed by the human author

### Test Output

7 new regressions; sync, CLI index and supersession suites: 77 passed.
The existing 35 sync cases are included. Existing datetime deprecation
warnings remain.

Original pre-commit and commit-msg hooks passed. Original full pre-push
`make ci-precheck` passed; native extension rebuilt/import-verified.
Full gate summaries:

```text
cargo fmt / clippy --workspace: passed
Rust workspace: 1755 passed, 3 ignored
Native extension: rebuilt and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing development
environment; base `5023c84e527e28281dd73d1ba04184fe738aac0f`, patch
`1fda45a711b4ce38848e8615148c74dc9c1bd6ab`. Real LocalBackend factory,
SQLite/FTS/vector store, agent files and Ollama HTTP adapter; the
loopback service returns controlled 384D vectors.
- Exact command / steps: save the following script as `proof.py`. Run
`python proof.py frontmatter <fresh-output-directory> before` from the
base checkout, then `python proof.py frontmatter
<another-fresh-directory> after` from the patch. The identical script
ran outside pytest.
- Observed result: first_exported=1 on both; round_trip_equal:
main=false, patch=true; repeat_imported/repeat_exported: main=1/1,
patch=0/0; primary_count: main=2, patch=1. The patch uses one actual
embedding HTTP request; main uses a second request to embed the polluted
duplicate.
- Not tested: trained-model inference or answer quality, paid/provider
APIs, non-local backends, production workloads, other operating systems
or the entire Python suite. Local focused, broader and original-gate
results are reported separately.

<details>
<summary>Standalone runtime reproduction</summary>

```python
"""Real LocalBackend/Ollama HTTP/SQLite/agent-file before-after replay."""
import asyncio
import json
import os
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
import sys
import threading

from headroom.memory.backends.local import LocalBackend, LocalBackendConfig
from headroom.memory.sync import sync
from headroom.memory.sync_adapters.codex_agent import CodexAdapter
from headroom.memory.sync_adapters.claude_code import ClaudeCodeAdapter

case, folder, expected = sys.argv[1:]
out = Path(folder).resolve()
out.mkdir(parents=True, exist_ok=False)
os.environ.update(HEADROOM_STATELESS="0", HEADROOM_WORKSPACE_DIR=str(out/'workspace'), HEADROOM_OFFLINE="0")

class Endpoint(BaseHTTPRequestHandler):
    requests = []
    def log_message(self, *_): pass
    def do_POST(self):
        payload=json.loads(self.rfile.read(int(self.headers['content-length'])))
        self.requests.append({'path':self.path,'body':payload})
        assert self.path=='/api/embeddings'
        body=json.dumps({'embedding':[1.0]+[0.0]*383}).encode()
        self.send_response(200)
        self.send_header('Content-Type','application/json')
        self.send_header('Content-Length',str(len(body)))
        self.end_headers()
        self.wfile.write(body)

server=ThreadingHTTPServer(('127.0.0.1',0),Endpoint)
threading.Thread(target=server.serve_forever,daemon=True).start()

async def run():
    backend=LocalBackend(LocalBackendConfig(db_path=str(out/'memory.db'),embedder_backend='ollama',embedder_model='controlled-vector',vector_dimension=384,ollama_base_url=f'http://127.0.0.1:{server.server_port}'))
    state=out/'sync-state.json'
    try:
        if case=='fingerprint':
            saved=await backend.save_memory('Original orchard fact',user_id='alice')
            adapter=CodexAdapter(out/'AGENTS.md')
            first=await sync(backend,adapter,'alice',state)
            unchanged=await sync(backend,adapter,'alice',state)
            await backend._hierarchical_memory.update(saved.id,content='Corrected orchard fact')
            result=await sync(backend,adapter,'alice',state)
            text=adapter._path.read_text(encoding='utf-8')
            data={'first_exported':first.exported,'unchanged_exported':unchanged.exported,'edited_exported':result.exported,'corrected_in_file':'Corrected orchard fact' in text,'edited_imported':result.imported}
            assert data['first_exported']==1 and data['unchanged_exported']==0
            assert data['corrected_in_file']==(expected=='after'),data
            assert result.exported==(1 if expected=='after' else 0),data
        else:
            content='Use foo---bar as the separator\nKeep the second line intact.'
            saved=await backend.save_memory(content,user_id='alice')
            adapter=ClaudeCodeAdapter(out/'claude-memory')
            first=await sync(backend,adapter,'alice',state,force=True)
            readback=await adapter.read_memories()
            repeated=await sync(backend,adapter,'alice',state,force=True)
            rows=await backend.get_user_memories('alice')
            data={'first_exported':first.exported,'round_trip_equal':readback[0].content==content,'reread_content':readback[0].content,'repeat_imported':repeated.imported,'repeat_exported':repeated.exported,'primary_count':len(rows)}
            assert data['round_trip_equal']==(expected=='after'),data
            assert data['repeat_imported']==(0 if expected=='after' else 1),data
        data['requests']=Endpoint.requests
        data['source_file']=__import__('headroom').__file__
        (out/'result.json').write_text(json.dumps(data,indent=2),encoding='utf-8',newline='\n')
        print(json.dumps(data,ensure_ascii=False))
    finally: await backend.close()
try: asyncio.run(run())
finally: server.shutdown();server.server_close()

```
</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing memory sync.
- Minimum rollout channel: existing memory sync users.
- Stable/default behavior changed: metadata containing triple dashes
remains separate from the memory body.
- Kill switch / disable path: stop invoking memory sync; no new toggle.
- Unsafe override required: no.
- Qualification impact: memory synchronization correctness; no
dependency, native or workflow change.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented the non-obvious behavior
- [x] I have added regression tests
- [x] I did not edit CHANGELOG.md

## Additional Notes

Implementation and local test execution were completed with Codex
assistance. The author personally reviewed the two patches and the
before/after evidence; this does not claim the author personally ran the
tests.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4886"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:17:10 -06:00
Yuchuan HeandJD Davis 00b38736d9 fix(memory): detect content changes in incremental sync (#4885)
## Description

Incremental sync hashes only the row count and the first eight
characters of the newest five IDs. An in-place content edit keeps those
values unchanged, so sync returns early and never exports the corrected
fact.

Hash the content, complete IDs and fields used for export/lineage across
the already-fetched snapshot. Unchanged snapshots still take the no-op
path. The persisted fingerprint is opaque; existing states receive one
normal sync after this change.

Related to #4832's full-store sync work; this fixes a separate contract
on the current main snapshot.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Include full IDs, content and export/lineage fields from every
returned row; exclude volatile access counters.
- Add regressions against the unmodified base
`5023c84e527e28281dd73d1ba04184fe738aac0f` and real file/component
coverage.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook with httpx
installed)
- [x] New tests added
- [ ] Manual testing performed by the human author

### Test Output

6 new regressions; focused sync suite: 41 passed. Broader memory run:
675 passed, 82 skipped, 8 failed. The same 8 failures reproduce on the
exact unmodified base (275 passed in the two affected files): six
Windows SQLite temporary-directory cleanup failures and two POSIX-path
assumptions. No old fixture changed.

Original pre-commit and commit-msg hooks passed. Original full pre-push
`make ci-precheck` passed; native extension rebuilt/import-verified.
Full gate summaries:

```text
cargo fmt / clippy --workspace: passed
Rust workspace: 1755 passed, 3 ignored
Native extension: rebuilt and import-verified
Critical Python: 177 passed, 8 skipped
commitlint: passed
```

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing development
environment; base `5023c84e527e28281dd73d1ba04184fe738aac0f`, patch
`c4ffa08665c26491b2379af0f4d953ceb203ee6b`. Real LocalBackend factory,
SQLite/FTS/vector store, agent files and Ollama HTTP adapter; the
loopback service returns controlled 384D vectors.
- Exact command / steps: save the following script as `proof.py`. Run
`python proof.py fingerprint <fresh-output-directory> before` from the
base checkout, then `python proof.py fingerprint
<another-fresh-directory> after` from the patch. The identical script
ran outside pytest.
- Observed result: first_exported=1; unchanged_exported=0;
edited_exported: main=0, patch=1; corrected_in_file: main=false,
patch=true. The existing additive bidirectional semantics also reimport
the old file fact (edited_imported=1 on the patch); this change does not
implement deletion propagation or conflict resolution.
- Not tested: trained-model inference or answer quality, paid/provider
APIs, non-local backends, production workloads, other operating systems
or the entire Python suite. Local focused, broader and original-gate
results are reported separately.

<details>
<summary>Standalone runtime reproduction</summary>

```python
"""Real LocalBackend/Ollama HTTP/SQLite/agent-file before-after replay."""
import asyncio
import json
import os
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
import sys
import threading

from headroom.memory.backends.local import LocalBackend, LocalBackendConfig
from headroom.memory.sync import sync
from headroom.memory.sync_adapters.codex_agent import CodexAdapter
from headroom.memory.sync_adapters.claude_code import ClaudeCodeAdapter

case, folder, expected = sys.argv[1:]
out = Path(folder).resolve()
out.mkdir(parents=True, exist_ok=False)
os.environ.update(HEADROOM_STATELESS="0", HEADROOM_WORKSPACE_DIR=str(out/'workspace'), HEADROOM_OFFLINE="0")

class Endpoint(BaseHTTPRequestHandler):
    requests = []
    def log_message(self, *_): pass
    def do_POST(self):
        payload=json.loads(self.rfile.read(int(self.headers['content-length'])))
        self.requests.append({'path':self.path,'body':payload})
        assert self.path=='/api/embeddings'
        body=json.dumps({'embedding':[1.0]+[0.0]*383}).encode()
        self.send_response(200)
        self.send_header('Content-Type','application/json')
        self.send_header('Content-Length',str(len(body)))
        self.end_headers()
        self.wfile.write(body)

server=ThreadingHTTPServer(('127.0.0.1',0),Endpoint)
threading.Thread(target=server.serve_forever,daemon=True).start()

async def run():
    backend=LocalBackend(LocalBackendConfig(db_path=str(out/'memory.db'),embedder_backend='ollama',embedder_model='controlled-vector',vector_dimension=384,ollama_base_url=f'http://127.0.0.1:{server.server_port}'))
    state=out/'sync-state.json'
    try:
        if case=='fingerprint':
            saved=await backend.save_memory('Original orchard fact',user_id='alice')
            adapter=CodexAdapter(out/'AGENTS.md')
            first=await sync(backend,adapter,'alice',state)
            unchanged=await sync(backend,adapter,'alice',state)
            await backend._hierarchical_memory.update(saved.id,content='Corrected orchard fact')
            result=await sync(backend,adapter,'alice',state)
            text=adapter._path.read_text(encoding='utf-8')
            data={'first_exported':first.exported,'unchanged_exported':unchanged.exported,'edited_exported':result.exported,'corrected_in_file':'Corrected orchard fact' in text,'edited_imported':result.imported}
            assert data['first_exported']==1 and data['unchanged_exported']==0
            assert data['corrected_in_file']==(expected=='after'),data
            assert result.exported==(1 if expected=='after' else 0),data
        else:
            content='Use foo---bar as the separator\nKeep the second line intact.'
            saved=await backend.save_memory(content,user_id='alice')
            adapter=ClaudeCodeAdapter(out/'claude-memory')
            first=await sync(backend,adapter,'alice',state,force=True)
            readback=await adapter.read_memories()
            repeated=await sync(backend,adapter,'alice',state,force=True)
            rows=await backend.get_user_memories('alice')
            data={'first_exported':first.exported,'round_trip_equal':readback[0].content==content,'reread_content':readback[0].content,'repeat_imported':repeated.imported,'repeat_exported':repeated.exported,'primary_count':len(rows)}
            assert data['round_trip_equal']==(expected=='after'),data
            assert data['repeat_imported']==(0 if expected=='after' else 1),data
        data['requests']=Endpoint.requests
        data['source_file']=__import__('headroom').__file__
        (out/'result.json').write_text(json.dumps(data,indent=2),encoding='utf-8',newline='\n')
        print(json.dumps(data,ensure_ascii=False))
    finally: await backend.close()
try: asyncio.run(run())
finally: server.shutdown();server.server_close()

```
</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): none; existing memory sync.
- Minimum rollout channel: existing memory sync users.
- Stable/default behavior changed: edited sync snapshots trigger
comparison/export instead of a false no-op.
- Kill switch / disable path: stop invoking memory sync; no new toggle.
- Unsafe override required: no.
- Qualification impact: memory synchronization correctness; no
dependency, native or workflow change.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented the non-obvious behavior
- [x] I have added regression tests
- [x] I did not edit CHANGELOG.md

## Additional Notes

Implementation and local test execution were completed with Codex
assistance. The author personally reviewed the two patches and the
before/after evidence; this does not claim the author personally ran the
tests.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4885"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 23:16:44 -06:00
Yuchuan HeandJerrettDavis 0d9155019f fix(memory): dispatch the advertised memory_list tool (#4830)
## Description

`MemorySystem.get_tools()` advertises `memory_list`, but
`process_tool_call()` has no handler for it. An agent can see the tool
and still receive `Unknown tool: memory_list` when trying to browse
stored memories or obtain an ID for an update.

Dispatch the existing tool to the backend's scoped listing API. A
session-bound MemorySystem lists only that user's current session; an
unscoped one retains user-wide browsing. Return recent memories with
their IDs and readable fields, without embeddings or a semantic query.
Backends without the requested listing scope return an explicit
unsupported-operation error rather than falling back to a wider scope.

Refs #4362, item 1 only.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Add the `memory_list` dispatcher and public handler.
- Keep the schema's default limit of 10 and bounds of 1–100. Filter by
user and, when bound, session before applying the limit, using the
existing HierarchicalMemory.get_session_memories API through
LocalBackend.
- Cover normal and optimized tool inventories, empty results, bounds,
unsupported backends and backend failures.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook)
- [x] Regression tests added
- [x] Standalone runtime validation completed

### Test Output

```text
Initial regressions against original production code: 8 failed
Session regressions against initial PR head: 4 failed, 9 passed
Final list regressions on the corrected head: 13 passed
pytest tests/test_memory_list_dispatch.py tests/test_memory_system.py tests/test_memory_tool_project_scope.py -q:
  113 passed, 13 skipped
Original pre-commit and commit-msg hooks: passed
Whole-tree Ruff check and format --check: passed
Original pre-push make ci-precheck: passed
  Rust workspace: 1749 passed, 3 ignored
  Native extension: rebuilt, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

The owning-suite skips require Hugging Face model files unavailable
offline. Existing datetime/optional-model deprecation warnings remain
visible in the logs.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing dev venv; clean
production baseline `9f2cc017863f540c61f8747b3c744a97776ebaf2`, patch
`42aca035dde6734d1a9d7a0b87f1b99cf8b27e69`. Explicit checkout on
PYTHONPATH; no paid/provider endpoint. The loopback service returns
controlled vectors, not trained-model inference.
- Exact command / steps: Save the script below as `proof.py`. From each
checkout run `python proof.py list <fresh-output-directory> before` on
main, or `python proof.py list <fresh-output-directory> after` on the
patch. Identical scripts ran outside pytest. The script initializes a
real LocalBackend, saves three Alice memories and one Bob memory through
MemorySystem, then calls the advertised listing tool with limit=2. The
Ollama HTTP adapter sends actual requests to a bound loopback receiver
returning controlled 384-dimensional vectors; SQLite, FTS5 and the
configured local vector index are real.

<details>
<summary>Standalone reproduction script</summary>

```python
"""Standalone real SQLite/vector/FTS/HTTP and agent-file boundary reproductions.

Usage: python memory_http_proof.py {list,vectors,sync} FRESH_OUTPUT {before,after}
The loopback embedding service returns controlled vectors, not model inference.
"""

import asyncio
from datetime import datetime, timedelta, timezone
import hashlib
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
from pathlib import Path
import platform
import sqlite3
import sys
import threading

import headroom
import headroom._core
from headroom.memory.backends.local import LocalBackend, LocalBackendConfig
from headroom.memory.config import EmbedderBackend, MemoryConfig, VectorBackend
from headroom.memory.core import HierarchicalMemory
from headroom.memory.models import Memory
from headroom.memory.sync import sync, sync_import
from headroom.memory.sync_adapters.codex_agent import CodexAdapter
from headroom.memory.system import MemorySystem

sys.stdout.reconfigure(encoding="utf-8")
case, directory, expected = sys.argv[1:]
assert case == "list"
assert expected in ("before", "after")
out = Path(directory).resolve()
out.mkdir(parents=True, exist_ok=False)
os.environ["HEADROOM_STATELESS"] = "0"
os.environ["HEADROOM_WORKSPACE_DIR"] = str(out / "workspace")
os.environ["HEADROOM_CCR_BACKEND"] = "memory"
os.environ["HEADROOM_OFFLINE"] = "0"  # Only the ephemeral loopback service is called.


class Endpoint(BaseHTTPRequestHandler):
    dimension = 384
    requests = []

    def log_message(self, *_):
        pass

    def do_POST(self):
        payload = json.loads(self.rfile.read(int(self.headers["content-length"])))
        Endpoint.requests.append({"path": self.path, "body": payload, "dimension": self.dimension})
        assert self.path == "/api/embeddings", self.path
        body = json.dumps({"embedding": [1.0] + [0.0] * (self.dimension - 1)}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)


server = ThreadingHTTPServer(("127.0.0.1", 0), Endpoint)
worker = threading.Thread(target=server.serve_forever, daemon=True)
worker.start()
url = f"http://127.0.0.1:{server.server_port}"


def backend_config(name):
    return LocalBackendConfig(
        db_path=str(out / (name + ".db")), embedder_backend="ollama",
        embedder_model="probe-fixed", ollama_base_url=url, vector_dimension=384,
    )


async def list_case():
    backend = LocalBackend(backend_config("list"))
    try:
        system = MemorySystem(backend, user_id="alice")
        ids = []
        for content in ("First scoped fact", "Second scoped fact", "Latest scoped fact"):
            response = await system.process_tool_call("memory_save", {"content": content, "importance": 0.4})
            assert response["success"], response
            ids.append(response["memory_id"])
        response = await MemorySystem(backend, "bob").process_tool_call("memory_save", {"content": "Bob's private fact", "importance": 0.4})
        assert response["success"], response
        before_requests = len(Endpoint.requests)
        result = await system.process_tool_call("memory_list", {"limit": 2})
        result["embedding_requests_for_listing"] = len(Endpoint.requests) - before_requests
        result["stored_alice_count"] = len(await backend.get_user_memories("alice"))
        if expected == "after":
            assert result["success"] and result["count"] == 2, result
            assert [m["memory_id"] for m in result["memories"]] == [ids[2], ids[1]]
            assert all("Bob" not in m["content"] for m in result["memories"])
        else:
            assert result["success"] is False and result["error"] == "Unknown tool: memory_list", result
        assert result["stored_alice_count"] == 3 and result["embedding_requests_for_listing"] == 0
        return result
    finally:
        await backend.close()


try:
    result = asyncio.run(list_case())
    result.update({
        "case": case, "expected": expected, "python": platform.python_version(),
        "source": str(Path(headroom.__file__).resolve()),
        "native": str(Path(headroom._core.__file__).resolve()),
        "http_requests": Endpoint.requests,
        "not_tested": "Trained model inference, paid APIs, provider answer quality, non-Windows runtime, non-local memory backends.",
    })
    (out / "result.json").write_text(json.dumps(result, indent=2), encoding="utf-8", newline="\n")
    print(json.dumps(result, indent=2))
finally:
    server.shutdown()
    server.server_close()
    worker.join(timeout=5)
```

</details>

- Observed result: On main the tool returns Unknown tool: memory_list
even though Alice has three stored rows. With the patch it returns
Alice's two newest memories, in order, with the saved IDs. Bob's memory
is absent, and listing makes zero additional embedding requests.


Session-scope follow-up: run the script below as `python
session_proof.py <fresh-output-directory> before` on the initial PR head
`908762a8705dea9a878f28dca4cb25b3fc5136ee`, or with `after` on the
corrected head. It saves real memories for Alice/session-a,
Alice/session-b and Bob/session-a through LocalBackend and its actual
loopback Ollama HTTP adapter. With limit=1, the initial head returns the
newer Alice/session-b fact to session-a; the correction returns
session-a's fact. A missing session returns zero results, unscoped
browsing still returns both Alice facts, Bob remains excluded, and
listing sends no embedding request.

<details>
<summary>Session isolation reproduction</summary>

```python
"""Real LocalBackend listing must filter user and session before LIMIT."""
import asyncio
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
from pathlib import Path
import sys
import threading

from headroom.memory.backends.local import LocalBackend, LocalBackendConfig
from headroom.memory.system import MemorySystem

sys.stdout.reconfigure(encoding="utf-8")
directory, expected = sys.argv[1:]
assert expected in ("before", "after")
out = Path(directory).resolve()
out.mkdir(parents=True, exist_ok=False)
os.environ.update(HEADROOM_OFFLINE="0", HEADROOM_STATELESS="0", HEADROOM_CCR_BACKEND="memory")


class Endpoint(BaseHTTPRequestHandler):
    calls = []

    def log_message(self, *_):
        pass

    def do_POST(self):
        request = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        assert self.path == "/api/embeddings"
        Endpoint.calls.append(request["prompt"])
        response = json.dumps({"embedding": [1.0] + [0.0] * 383}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(response)))
        self.end_headers()
        self.wfile.write(response)


server = ThreadingHTTPServer(("127.0.0.1", 0), Endpoint)
worker = threading.Thread(target=server.serve_forever, daemon=True)
worker.start()


async def main():
    backend = LocalBackend(LocalBackendConfig(
        db_path=str(out / "memory.db"), embedder_backend="ollama", embedder_model="probe-fixed",
        ollama_base_url=f"http://127.0.0.1:{server.server_port}", vector_dimension=384,
    ))
    try:
        current = MemorySystem(backend, "alice", session_id="session-a")
        other_session = MemorySystem(backend, "alice", session_id="session-b")
        other_user = MemorySystem(backend, "bob", session_id="session-a")
        saved = []
        for system, content in ((current,"Current-session fact"),(other_session,"Newer other-session fact"),(other_user,"Newest other-user fact")):
            result = await system.process_tool_call("memory_save", {"content":content,"importance":0.4})
            assert result["success"],result
            saved.append(result["memory_id"])
        count = len(Endpoint.calls)
        listed = await current.process_tool_call("memory_list", {"limit":1})
        assert listed["success"] and listed["count"] == 1,listed
        scope_preserved = listed["memories"][0]["memory_id"] == saved[0]
        assert scope_preserved == (expected == "after"),listed
        assert listed["memories"][0]["memory_id"] != saved[2]
        missing = await MemorySystem(backend,"alice",session_id="missing").process_tool_call("memory_list", {"limit":1})
        assert missing["count"] == (0 if expected == "after" else 1),missing
        unscoped = await MemorySystem(backend,"alice").process_tool_call("memory_list", {"limit":2})
        assert [m["memory_id"] for m in unscoped["memories"]] == [saved[1],saved[0]]
        assert len(Endpoint.calls) == count
        result = {"session_isolated":scope_preserved,"scoped_content":listed["memories"][0]["content"],"missing_session_count":missing["count"],"unscoped_count":unscoped["count"],"other_user_excluded":True,"embedding_calls_for_listing":len(Endpoint.calls)-count,"save_http_calls":count,"expected":expected,"source":str(Path(__import__('headroom').__file__).resolve())}
        (out/"result.json").write_text(json.dumps(result,indent=2),encoding="utf-8",newline="\n")
        print(json.dumps(result,indent=2))
    finally:
        await backend.close()


try:
    asyncio.run(main())
finally:
    server.shutdown()
    server.server_close()
    worker.join(timeout=5)
```

</details>

- Not tested: Remote memory backends, trained embedding inference, paid
APIs, model answer quality, non-Windows runtime, or the complete Python
suite.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing memory behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: The advertised tool can enumerate
recent memories on backends that expose get_user_memories. Other memory
operations retain their dispatch paths.
- Kill switch / disable path: Use the existing memory-disable path, or
revert this commit.
- Unsafe override required: No.
- Qualification impact: Related tests and original local gates passed;
hosted CI is tracked separately.
- Rollback path: Revert this commit; no dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Listing is supported through LocalBackend's user and session APIs; the
session adapter delegates to an existing core query that applies both
filters before LIMIT. This patch does not add an enumeration API to
DirectMem0. The author's review covered the initial patch and evidence;
the session-isolation review correction was additionally reviewed and
validated with Codex.

Developed and validated with Codex assistance.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4830"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:09:26 -06:00
Yuchuan HeandJerrettDavis cd6f438fec fix(memory): validate embedding dimensions before writes (#4831)
## Description

A mismatched embedding reaches the vector index after the primary memory
row has already been saved. A rejected add leaves a half-written row; a
rejected supersession also ends the old version and removes its index
entry before failing. This occurs with a vector_dimension that differs
from the embedder's actual output.

Validate each newly produced embedding's shape before the first
primary-store mutation. Apply this to add, the whole add_batch, content
updates and supersession. An invalid replacement leaves the old row,
cached object and version chain intact.

Refs #4362, item 2's half-write boundary. The configured vector
dimension remains explicit; this does not automatically resize or
migrate an index.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Reject non-vector and wrong-width embeddings before saving rows.
- Validate an entire batch before saving any member.
- Validate a content update before mutating a store-returned object,
preserving update()'s existing object identity and metadata-only refresh
behavior.
- Validate supersession before ending the old version or removing its
indexes.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook)
- [x] Regression tests added
- [x] Standalone runtime validation completed

### Test Output

```text
Final regressions against original production code: 7 failed, 1 passed
pytest tests/test_memory_embedding_validation.py tests/test_memory/test_core_operations.py tests/test_memory/test_hierarchical.py tests/test_memory/test_core_metadata_index_sync.py tests/test_memory/test_clear_scope_consistency.py -q:
  65 passed, 51 skipped
Original pre-commit and commit-msg hooks: passed
Whole-tree Ruff check and format --check: passed
Original pre-push make ci-precheck: passed
  Rust workspace: 1749 passed, 3 ignored
  Native extension: rebuilt, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

The owning-suite skips require Hugging Face model files unavailable
offline. Existing datetime/optional-model deprecation warnings remain
visible in the logs.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing dev venv; clean
production baseline `9f2cc017863f540c61f8747b3c744a97776ebaf2`, patch
`50d126b1b8379390e72d3e47216519f9d297b4a1`. Explicit checkout on
PYTHONPATH; no paid/provider endpoint. The loopback service returns
controlled vectors, not trained-model inference.
- Exact command / steps: Save the script below as `proof.py`. From each
checkout run `python proof.py vectors <fresh-output-directory> before`
on main, or `python proof.py vectors <fresh-output-directory> after` on
the patch. Identical scripts ran outside pytest. The script creates real
HierarchicalMemory with SQLite, FTS5, HNSW(dimension=384) and the Ollama
HTTP adapter. A bound loopback server first returns a 768-dimensional
embedding for a rejected add, then 384 dimensions for a valid original,
then 768 dimensions for a rejected supersession. It inspects primary
rows, the original version and the live vector index.

<details>
<summary>Standalone reproduction script</summary>

```python
"""Standalone real SQLite/vector/FTS/HTTP and agent-file boundary reproductions.

Usage: python memory_http_proof.py {list,vectors,sync} FRESH_OUTPUT {before,after}
The loopback embedding service returns controlled vectors, not model inference.
"""

import asyncio
from datetime import datetime, timedelta, timezone
import hashlib
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
from pathlib import Path
import platform
import sqlite3
import sys
import threading

import headroom
import headroom._core
from headroom.memory.backends.local import LocalBackend, LocalBackendConfig
from headroom.memory.config import EmbedderBackend, MemoryConfig, VectorBackend
from headroom.memory.core import HierarchicalMemory
from headroom.memory.models import Memory
from headroom.memory.sync import sync, sync_import
from headroom.memory.sync_adapters.codex_agent import CodexAdapter
from headroom.memory.system import MemorySystem

sys.stdout.reconfigure(encoding="utf-8")
case, directory, expected = sys.argv[1:]
assert case == "vectors"
assert expected in ("before", "after")
out = Path(directory).resolve()
out.mkdir(parents=True, exist_ok=False)
os.environ["HEADROOM_STATELESS"] = "0"
os.environ["HEADROOM_WORKSPACE_DIR"] = str(out / "workspace")
os.environ["HEADROOM_CCR_BACKEND"] = "memory"
os.environ["HEADROOM_OFFLINE"] = "0"  # Only the ephemeral loopback service is called.


class Endpoint(BaseHTTPRequestHandler):
    dimension = 384
    requests = []

    def log_message(self, *_):
        pass

    def do_POST(self):
        payload = json.loads(self.rfile.read(int(self.headers["content-length"])))
        Endpoint.requests.append({"path": self.path, "body": payload, "dimension": self.dimension})
        assert self.path == "/api/embeddings", self.path
        body = json.dumps({"embedding": [1.0] + [0.0] * (self.dimension - 1)}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)


server = ThreadingHTTPServer(("127.0.0.1", 0), Endpoint)
worker = threading.Thread(target=server.serve_forever, daemon=True)
worker.start()
url = f"http://127.0.0.1:{server.server_port}"


async def new_memory(name):
    return await HierarchicalMemory.create(MemoryConfig(
        db_path=out / (name + ".db"), embedder_backend=EmbedderBackend.OLLAMA,
        embedder_model="nomic-embed-text", ollama_base_url=url,
        vector_backend=VectorBackend.HNSW, vector_dimension=384, auto_bubble=False,
    ))


async def vectors_case():
    memory = await new_memory("vectors")
    Endpoint.dimension = 768
    result = {}
    try:
        try:
            await memory.add("Rejected incompatible embedding", user_id="alice")
            raise AssertionError("mismatch was accepted")
        except ValueError as exc:
            result["add_error"] = str(exc)
        result["rows_after_rejected_add"] = len(await memory.get_user_memories("alice"))
        Endpoint.dimension = 384
        original = await memory.add("Valid original fact", user_id="alice")
        Endpoint.dimension = 768
        try:
            await memory.supersede(original.id, "Rejected replacement fact")
            raise AssertionError("replacement mismatch was accepted")
        except ValueError as exc:
            result["supersede_error"] = str(exc)
        stored = await memory._store.get(original.id)
        result["original_still_active"] = stored.valid_until is None and stored.superseded_by is None
        result["vector_entries_after_rejected_supersede"] = memory._vector_index.size
        with sqlite3.connect(out / "vectors.db") as db:
            result["all_primary_rows"] = db.execute("SELECT COUNT(*) FROM memories WHERE user_id = 'alice'").fetchone()[0]
        assert result["rows_after_rejected_add"] == (0 if expected == "after" else 1), result
        assert result["original_still_active"] == (expected == "after"), result
        assert result["vector_entries_after_rejected_supersede"] == (1 if expected == "after" else 0), result
        assert result["all_primary_rows"] == (1 if expected == "after" else 3), result
        return result
    finally:
        await memory.close()


try:
    result = asyncio.run(vectors_case())
    result.update({
        "case": case, "expected": expected, "python": platform.python_version(),
        "source": str(Path(headroom.__file__).resolve()),
        "native": str(Path(headroom._core.__file__).resolve()),
        "http_requests": Endpoint.requests,
        "not_tested": "Trained model inference, paid APIs, provider answer quality, non-Windows runtime, non-local memory backends.",
    })
    (out / "result.json").write_text(json.dumps(result, indent=2), encoding="utf-8", newline="\n")
    print(json.dumps(result, indent=2))
finally:
    server.shutdown()
    server.server_close()
    worker.join(timeout=5)
```

</details>

- Observed result: Main leaves one row after the rejected add; after the
rejected replacement it has three primary rows, the original is no
longer active and its vector was removed. The patch leaves zero rows
after the rejected add, then retains exactly the valid original row and
its one vector after the rejected replacement.
- Not tested: General cross-component transactions for arbitrary
storage/index I/O failures, automatic dimension discovery or index
migration, remote vector backends, trained model inference, paid APIs,
non-Windows runtime, or the complete Python suite.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing memory behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Embedding shape errors are reported
before primary writes. Matching vectors and explicit auto_embed=False
operations retain their existing behavior.
- Kill switch / disable path: Use the existing memory-disable path, or
revert this commit.
- Unsafe override required: No.
- Qualification impact: Related tests and original local gates passed;
hosted CI is tracked separately.
- Rollback path: Revert this commit; no dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Broader regression exposed update()'s object-identity contract; the
final fix preserves it without changing the existing fixture. The
author's review covered the initial patch and evidence; this
compatibility adjustment was additionally reviewed and tested with
Codex.

Developed and validated with Codex assistance. The author reviewed the
changes and before/after evidence.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4831"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:08:59 -06:00
Yuchuan HeandJerrettDavis a6a825c55d fix(memory): include older memories in agent sync (#4832)
## Description

The memory sync engine builds its deduplication and export sets from
only the latest 500 user memories. Older rows are omitted from agent
files, and an existing older fact can be imported again as a duplicate.

Use the user's full snapshot for synchronization, including the refetch
after an import and the standalone import/export entry points. Carry the
existing MemoryFilter limit=None semantics through LocalBackend and
HierarchicalMemory; ordinary queries keep their default limit of 100.

Refs #4362, item 9 only.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Request the full user-scoped snapshot at each sync entry point.
- Document and type the existing unbounded query option without changing
default query limits.
- Cover a 501-row store, older-fact deduplication, import-triggered
refetch, forced/ordinary sync, idempotence and user isolation.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook)
- [x] Regression tests added
- [x] Standalone runtime validation completed

### Test Output

```text
Final regressions against original production code: 5 failed, 2 passed
pytest tests/test_memory_sync_full_store.py tests/test_memory_sync.py tests/test_memory/test_core_operations.py tests/test_memory/test_clear_scope_consistency.py -q:
  57 passed, 48 skipped
Original pre-commit and commit-msg hooks: passed
Whole-tree Ruff check and format --check: passed
Original pre-push make ci-precheck: passed
  Rust workspace: 1749 passed, 3 ignored
  Native extension: rebuilt, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

The owning-suite skips require Hugging Face model files unavailable
offline. Existing datetime/optional-model deprecation warnings remain
visible in the logs.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing dev venv; clean
production baseline `9f2cc017863f540c61f8747b3c744a97776ebaf2`, patch
`64e537fa2459d3540522bf934a32ee2e8e089ea7`. Explicit checkout on
PYTHONPATH; no paid/provider endpoint. The loopback service returns
controlled vectors, not trained-model inference.
- Exact command / steps: Save the script below as `proof.py`. From each
checkout run `python proof.py sync <fresh-output-directory> before` on
main, or `python proof.py sync <fresh-output-directory> after` on the
patch. Identical scripts ran outside pytest. The script initializes a
real LocalBackend and SQLite store, seeds 501 chronologically ordered
Alice rows plus a Bob row, and runs sync into a real CodexAdapter
AGENTS.md file. It then presents the oldest existing fact from a second
agent file and calls sync_import. The unpatched duplicate path makes an
actual Ollama HTTP request to a loopback embedding service; the patched
deduplication path makes none.

<details>
<summary>Standalone reproduction script</summary>

```python
"""Standalone real SQLite/vector/FTS/HTTP and agent-file boundary reproductions.

Usage: python memory_http_proof.py {list,vectors,sync} FRESH_OUTPUT {before,after}
The loopback embedding service returns controlled vectors, not model inference.
"""

import asyncio
from datetime import datetime, timedelta, timezone
import hashlib
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
from pathlib import Path
import platform
import sqlite3
import sys
import threading

import headroom
import headroom._core
from headroom.memory.backends.local import LocalBackend, LocalBackendConfig
from headroom.memory.config import EmbedderBackend, MemoryConfig, VectorBackend
from headroom.memory.core import HierarchicalMemory
from headroom.memory.models import Memory
from headroom.memory.sync import sync, sync_import
from headroom.memory.sync_adapters.codex_agent import CodexAdapter
from headroom.memory.system import MemorySystem

sys.stdout.reconfigure(encoding="utf-8")
case, directory, expected = sys.argv[1:]
assert case == "sync"
assert expected in ("before", "after")
out = Path(directory).resolve()
out.mkdir(parents=True, exist_ok=False)
os.environ["HEADROOM_STATELESS"] = "0"
os.environ["HEADROOM_WORKSPACE_DIR"] = str(out / "workspace")
os.environ["HEADROOM_CCR_BACKEND"] = "memory"
os.environ["HEADROOM_OFFLINE"] = "0"  # Only the ephemeral loopback service is called.


class Endpoint(BaseHTTPRequestHandler):
    dimension = 384
    requests = []

    def log_message(self, *_):
        pass

    def do_POST(self):
        payload = json.loads(self.rfile.read(int(self.headers["content-length"])))
        Endpoint.requests.append({"path": self.path, "body": payload, "dimension": self.dimension})
        assert self.path == "/api/embeddings", self.path
        body = json.dumps({"embedding": [1.0] + [0.0] * (self.dimension - 1)}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)


server = ThreadingHTTPServer(("127.0.0.1", 0), Endpoint)
worker = threading.Thread(target=server.serve_forever, daemon=True)
worker.start()
url = f"http://127.0.0.1:{server.server_port}"


def backend_config(name):
    return LocalBackendConfig(
        db_path=str(out / (name + ".db")), embedder_backend="ollama",
        embedder_model="probe-fixed", ollama_base_url=url, vector_dimension=384,
    )


async def sync_case():
    backend = LocalBackend(backend_config("sync"))
    try:
        await backend._ensure_initialized()
        now = datetime.now(timezone.utc)
        memories = [Memory(
            id=f"alice-{i:04}", content=f"Distinct memory {i}", user_id="alice",
            created_at=now - timedelta(seconds=i),
        ) for i in range(501)]
        memories.append(Memory(content="Bob's private fact", user_id="bob"))
        await backend._hierarchical_memory._store.save_batch(memories)
        adapter = CodexAdapter(out / "AGENTS.md")
        first = await sync(backend, adapter, "alice", state_path=out / "sync-state.json", force=True)
        exported = {m.content for m in await adapter.read_memories()}
        older_adapter = CodexAdapter(out / "older-agent.md")
        await older_adapter.write_memories([{"content": "Distinct memory 500"}])
        imported = await sync_import(backend, older_adapter, "alice")
        result = {
            "first_exported": first.exported,
            "oldest_exported": "Distinct memory 500" in exported,
            "other_user_exported": "Bob's private fact" in exported,
            "reimported_existing_oldest": imported,
            "agent_file_sha256": hashlib.sha256((out / "AGENTS.md").read_bytes()).hexdigest(),
        }
        with sqlite3.connect(out / "sync.db") as db:
            result["alice_primary_rows"] = db.execute("SELECT COUNT(*) FROM memories WHERE user_id = 'alice'").fetchone()[0]
        assert result["first_exported"] == (501 if expected == "after" else 500), result
        assert result["oldest_exported"] == (expected == "after"), result
        assert result["reimported_existing_oldest"] == (0 if expected == "after" else 1), result
        assert result["alice_primary_rows"] == (501 if expected == "after" else 502), result
        assert result["other_user_exported"] is False
        return result
    finally:
        await backend.close()


try:
    result = asyncio.run(sync_case())
    result.update({
        "case": case, "expected": expected, "python": platform.python_version(),
        "source": str(Path(headroom.__file__).resolve()),
        "native": str(Path(headroom._core.__file__).resolve()),
        "http_requests": Endpoint.requests,
        "not_tested": "Trained model inference, paid APIs, provider answer quality, non-Windows runtime, non-local memory backends.",
    })
    (out / "result.json").write_text(json.dumps(result, indent=2), encoding="utf-8", newline="\n")
    print(json.dumps(result, indent=2))
finally:
    server.shutdown()
    server.server_close()
    worker.join(timeout=5)
```

</details>

- Observed result: Main exports 500 memories, omits the oldest and
imports that existing fact as a new row (502 Alice rows). The patch
exports all 501, includes the oldest, imports zero duplicates and keeps
501 Alice rows. Bob's row is excluded in both runs.
- Not tested: Very large-store throughput or peak memory, concurrent
sync snapshot consistency, changes to the existing fingerprint sampling
policy, remote memory backends, trained model inference, paid APIs,
non-Windows runtime, or the complete Python suite.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing memory behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Agent sync covers all user rows;
ordinary get_user_memories calls keep the default 100-row limit and
explicit limits still apply.
- Kill switch / disable path: Use the existing memory-disable path, or
revert this commit.
- Unsafe override required: No.
- Qualification impact: Related tests and original local gates passed;
hosted CI is tracked separately.
- Rollback path: Revert this commit; no dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Synchronization already materializes a snapshot and hash set. Covering
the full store makes their size scale with the user's retained memories;
this patch does not introduce a paging protocol or alter the existing
no-op fingerprint algorithm.

Developed and validated with Codex assistance. The author reviewed the
changes and before/after evidence.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4832"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:08:11 -06:00
99021af20a fix(proxy): make the Rust front's native Bedrock and Vertex routes opt-in (#4851)
## Description

Two routes in the Rust front (`crates/headroom-proxy`) attach the host's
own cloud credentials to whatever request reaches them, without checking
the caller:

- **Bedrock native** (`/model/{id}/invoke`, `/converse`,
`/invoke-with-response-stream`, `/converse-stream`) is on by default and
signs every request with SigV4 using the AWS credentials found on the
host.
- **Vertex** (`…/publishers/anthropic/models/{m}:rawPredict` /
`:streamRawPredict`) is mounted unconditionally, replaces the caller's
`Authorization` with a bearer from the host's Google ADC, and forwards
to the generic `--upstream`, whatever host that is. So with a non-Google
upstream (the Python proxy, another provider) the host's Google token is
sent there. `--vertex-adc-scope` was parsed but never used.

This PR makes both routes explicit opt-ins, only sends the ADC bearer to
a Google API host, and uses the configured scope.

**Dependency:** these routes still do not authenticate callers
themselves. Callers are expected to be authenticated by the inbound
`HEADROOM_PROXY_TOKEN` gate added in #4840 (which also moves the default
listen address to loopback). This PR is based on `main` and does not
require #4840 to merge first, but operators who enable either route on a
non-loopback address need #4840 to restrict who can use them.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `config.rs`: `--enable-bedrock-native` /
`HEADROOM_PROXY_ENABLE_BEDROCK_NATIVE` now defaults to `false`. New
`--enable-vertex-native` / `HEADROOM_PROXY_ENABLE_VERTEX_NATIVE`
(default `false`) and `--vertex-allow-non-google-upstream` /
`HEADROOM_PROXY_VERTEX_ALLOW_NON_GOOGLE_UPSTREAM` (default `false`).
`Config::for_test` keeps both routes on so the existing route tests run
unchanged.
- `proxy.rs`: the Vertex route is mounted only when enabled; with it
off, Vertex paths go through the catch-all with the caller's own headers
(as the Python proxy's Vertex passthrough expects). The ADC token source
is built with `--vertex-adc-scope`. The "Bedrock route not mounted"
startup line is now `info`, since off is the default.
- `vertex/raw_predict.rs`: before fetching the ADC bearer, the handler
refuses (502 `vertex upstream is not a Google API host`,
`vertex_upstream_not_google` error log) unless `--upstream` is
`https://` on a `*.googleapis.com` host (regional, global and
`*.p.googleapis.com` endpoints), or the operator set
`--vertex-allow-non-google-upstream=true` (local emulators, gateways).
No token is fetched on that path.
- `RUST_DEV.md`: the three flags in the configuration table, with what
enabling them means.

**Behaviour change for operators:** deployments that used the Rust
front's native Bedrock or Vertex signing must now pass
`--enable-bedrock-native true` / `--enable-vertex-native true` (or the
env vars). Vertex with a non-Google `--upstream` additionally needs
`--vertex-allow-non-google-upstream true`. Without the flags, those
paths are forwarded to `--upstream` unsigned with the client's own
credentials (Claude Code in Vertex mode keeps sending its own ADC token,
which is what `docs/content/docs/claude-code-vertex.mdx` describes).
`scripts/qualify_cargo_dependencies.py` already passes
`--enable-bedrock-native true`.

## Testing

- [x] `cargo test -p headroom-proxy` (45 test binaries, 596 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes

New tests in `tests/integration_cloud_routes_opt_in.rs` build the config
from the real CLI defaults (`CliArgs` → `Config::from_cli`), give the
app host AWS credentials and a host ADC bearer, and record what a
`wiremock` upstream receives:
- `default_config_does_not_sign_bedrock_requests`: no `AWS4-HMAC-SHA256`
Authorization on `invoke`, `invoke-with-response-stream`, `converse`.
- `default_config_never_attaches_host_adc_bearer`: `rawPredict` and
`streamRawPredict` reach the upstream without the host bearer.
- `vertex_bearer_is_not_sent_to_a_non_google_upstream`: with the route
enabled and an `http://127.0.0.1` upstream, 502 and the upstream sees
nothing.
- `vertex_opt_in_still_signs_when_non_google_upstream_is_allowed`: the
explicit escape hatch still works.
- `configured_adc_scope_reaches_token_source`.
- Unit test `vertex::raw_predict::tests::google_api_host_check`
(`http://`, `googleapis.com.example.net`, `example-googleapis.com`,
loopback are rejected).

### Test Output

Before (the three tests that compile against `main`, run with this PR's
`src/` changes stashed):

```text
test configured_adc_scope_reaches_token_source ... FAILED
test default_config_never_attaches_host_adc_bearer ... FAILED
test default_config_does_not_sign_bedrock_requests ... FAILED
test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.77s
```

After:

```text
$ cargo test -p headroom-proxy --test integration_cloud_routes_opt_in
test configured_adc_scope_reaches_token_source ... ok
test vertex_opt_in_still_signs_when_non_google_upstream_is_allowed ... ok
test vertex_bearer_is_not_sent_to_a_non_google_upstream ... ok
test default_config_never_attaches_host_adc_bearer ... ok
test default_config_does_not_sign_bedrock_requests ... ok
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.69s
```

## Real Behavior Proof

- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (c21b0813) and this branch. AWS credentials supplied
to the proxy through `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (fake
values), IMDS disabled. Upstream and Bedrock endpoint are a small Python
recorder on `127.0.0.1:18788` that prints the `Authorization` it
receives.
- Exact command / steps: `headroom-proxy --listen 127.0.0.1:18787
--upstream http://127.0.0.1:18788 --bedrock-endpoint
http://127.0.0.1:18788` (no Bedrock/Vertex flags), then `curl -d
'<anthropic body>'
http://127.0.0.1:18787/model/anthropic.claude-3-haiku-20240307-v1:0/invoke`
with no client credentials. Repeated on this branch with
`--enable-bedrock-native true`.
- Observed result (key id and signature redacted):

```text
== origin/main (no flags)
POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke, no client credentials -> 200
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": "AWS4-HMAC-SHA256 Credential=<redacted>/20261113/us-east-1/bedrock/aws4_request, SignedHeaders=..., Signature=<sig>"}

== this branch (no flags)
POST /model/anthropic.claude-3-haiku-20240307-v1:0/invoke, no client credentials -> 200
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": null}

== this branch, --enable-bedrock-native true
UPSTREAM SAW {"path": "/model/...:0/invoke", "authorization": "AWS4-HMAC-SHA256 Credential=<redacted>/20261113/us-east-1/bedrock/aws4_request, SignedHeaders=..., Signature=<sig>"}
```

- Not tested: Vertex against real Google ADC and a real
`*.googleapis.com` upstream (covered by the integration tests with a
static token source; no GCP credentials on the test machine).

## Runtime Rollout Safety

- Rollout-managed feature(s): `native_bedrock` (existing). The CLI flag
already maps `false` to a rollout disable, so with the new default the
feature reports disabled unless `--enable-bedrock-native true` is
passed. Vertex is not rollout-managed.
- Minimum rollout channel: unchanged
- Stable/default behavior changed: yes, both native routes are off by
default
- Kill switch / disable path: the routes are now off unless enabled
- Unsafe override required: no
- Qualification impact: none, the qualification script already passes
`--enable-bedrock-native true`
- Rollback path: pass `--enable-bedrock-native true` /
`--enable-vertex-native true` (plus `--vertex-allow-non-google-upstream
true` if the upstream is not a Google host), or revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Fix adapted from a proposal by HetCreep.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4851"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:07:49 -06:00
123XieCand123XieC d077263012 fix(openclaw): preserve assistant text blocks when tool arguments change (#4829)
## Description

OpenClaw history can contain text on both sides of an assistant tool
call. If the compression response changes the call arguments while
leaving the wire text intact, restoring that message currently moves
both pieces of text ahead of the call and joins them into a single
block.

## Type of Change

- [x] Bug fix

## Changes Made

- Compare the returned assistant text with the text sent for that
message. Keep the original blocks when the text matches, then apply the
changed arguments to the matching tool call.
- Extend the assemble regression to check the order and content of both
text blocks, the signed thinking block, and the updated tool call.

## Testing

- [x] Regression and existing OpenClaw tests pass.
- [x] Manual testing performed.

### Test Output

```text
cd plugins/openclaw
npm test          # 11 files, 162 tests passed
npm run typecheck # passed
npm run build     # passed
git diff --check  # passed
```

The repository commit hooks and Rust formatting check passed. The full
local pre-push suite stopped at Rust dependency resolution after the
registry fetch stalled. I used the hook's documented push override for
this TypeScript-only change. The verification above covers the changed
plugin.

## Real Behavior Proof

- Environment: macOS arm64, Node 22.22.2, built OpenClaw plugin with the
pinned JavaScript SDK, and a Node HTTP server listening on loopback. The
payload was synthetic and did not need a model.
- Exact command / steps: Run `npm run build` in `plugins/openclaw` on
base `9f2cc01` and the patched tree. In Node, import
`HeadroomContextEngine` from the built `dist/index.js`. Start a loopback
server that answers `GET /v1/retrieve/stats` with `{"store":{}}`. For
`POST /v1/compress`, echo the submitted messages and set the first tool
call's argument string to `{"id":7}`. Return `tokens_before: 100`,
`tokens_after: 25`, and `tokens_saved: 75`. Create the engine with that
`proxyUrl`, `autoStart: false`, and `minContextChars: 0`. Await
`ensureProxyUrl()`, then call `assemble()` with one assistant message
whose blocks are signed thinking, text `Checking.`, a `lookup` call with
ID `lookup-1` and arguments `{id: 7, notes: "fixture ".repeat(80)}`, and
text `Awaiting result.`.
- Observed result: the SDK sent one actual compression request in both
runs. Base returned `thinking, text("Checking.Awaiting result."),
toolCall({id:7})`. The patch returned `thinking, text("Checking."),
toolCall({id:7}), text("Awaiting result.")`. The signature and call ID
were preserved, and the engine reported 25 tokens in both runs.
- Not tested: a remote proxy, model calls, or an OpenClaw host process.
The runtime proof exercised the built plugin and SDK over HTTP with a
deterministic compression response.

## Runtime Rollout Safety

- Rollout-managed feature(s): OpenClaw context restoration.
- Minimum rollout channel: existing plugin channel.
- Stable/default behavior changed: text block placement is preserved
when only assistant tool arguments change.
- Kill switch / disable path: existing OpenClaw plugin enablement.
- Unsafe override required: none.
- Qualification impact: OpenClaw plugin suite and loopback verification
above.
- Rollback path: revert the commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

This is a small correction to the restoration path introduced by #3826.
AI assistance was used to investigate, implement, and verify the change.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4829"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: 123XieC <323077714+123XieC@users.noreply.github.com>
2026-11-16 23:05:24 -06:00
Yuchuan HeandJerrettDavis d58b3ef19b fix(cache): clear older skips when storing successful results (#4819)
## Description

Concurrent compression requests can finish with different verdicts for
the same content key. If a no-op marks the key as skipped before another
request stores a successful result, the old put leaves that skip in
place. The next request returns the original even though the successful
compression is present in memory and SQLite.

Remove the older skip under the existing memory lock when admitting a
successful result. First-sight mark_skip and explicit move_to_skip
retain their original behavior. The initial PR also routed mark_skip
through disk invalidation; review identified the unnecessary DELETEs
that added to new non-compressible blocks, so that part has been
removed.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Clear an older skip when a successful result is admitted.
- Cover both existing skip APIs with memory-only and SQLite-backed
caches.
- Trace real SQLite statements to preserve the memory-only first-sight
skip path.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook in WSL)
- [x] Regression tests added
- [x] Standalone runtime validation completed

### Test Output

```text
Four successful-result regressions failed on original main.
The added first-sight SQL regression failed on the initial PR and passes after review revision.
pytest tests/test_router_result_cache.py tests/test_router_cache_policy.py tests/test_transforms/test_content_router.py tests/test_content_router_concurrency.py tests/test_compression_determinism.py tests/test_cache tests/test_cold_prefix.py:
  561 passed, 3 skipped
Original pre-commit and commit-msg hooks: passed
Whole-tree Ruff check and format --check: passed
Original pre-push make ci-precheck: passed
  Rust workspace: 1749 passed, 3 ignored
  Native extension: rebuilt, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

Optional-model skips and the existing token-count approximation warning
remain visible in the gate logs.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing dev venv; clean
baseline `c05fbdddc873078dd97cb9ede841e75ef7ed0bef`, patch
`b7f8872ac59c41abd12e2274364bcfecba67ceb7`. Explicit checkout on
PYTHONPATH; no provider credentials. The proof enables loopback HTTP;
model downloads remain disabled.
- Exact command / steps: Save the following script as `proof.py` and run
`python proof.py` from either checkout with `PYTHONPATH=.`. Identical
before/after scripts ran outside pytest.

```python
"""Two actual router requests with real remote HTTP, then SQLite replay."""
from concurrent.futures import ThreadPoolExecutor
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
from pathlib import Path
import tempfile
import threading
from headroom.cache.result_store import ResultStore
from headroom.tokenizers.estimator import CharacterCounter
from headroom.transforms.content_router import CompressionCache, ContentRouter, ContentRouterConfig

os.environ['HEADROOM_CCR_BACKEND']='memory'
os.environ['HEADROOM_OFFLINE']='0'  # This script binds and calls loopback only.
os.environ['HEADROOM_FREEZE_BLOCK_DECISION']='0'
first_seen, release_first = threading.Event(), threading.Event()
lock=threading.Lock();calls=[]
short='The deployment is healthy and the requested records are present.'
class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        payload=json.loads(self.rfile.read(int(self.headers['Content-Length'])))
        with lock:
            index=len(calls);calls.append(payload)
        if index==0:
            first_seen.set()
            if not release_first.wait(15): raise RuntimeError('controller did not release response')
            text=short
        else:
            text=payload['content']
        data=json.dumps({'compressed':text,'original_tokens':len(payload['content'].split()),'compressed_tokens':len(text.split()),'compression_ratio':0.1 if index==0 else 1.0}).encode()
        self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
    def log_message(self,*args):pass

server=ThreadingHTTPServer(('127.0.0.1',0),Handler)
thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()
os.environ['HEADROOM_KOMPRESS_ENDPOINT']=f'http://127.0.0.1:{server.server_port}'
payload=[{'role':'tool','tool_call_id':'t1','content':'The service contains useful deployment observations for the current investigation. '*80}]
tokenizer=CharacterCounter()
try:
    with tempfile.TemporaryDirectory(prefix='headroom-tier-proof-') as d:
        router=ContentRouter(ContentRouterConfig(ccr_inject_marker=False, result_cache_disk=False))
        disk=ResultStore(Path(d)/'cache.db',1800)
        router._cache=CompressionCache(store=disk,namespace='proof:')
        def apply():return router.apply(payload,tokenizer,force_kompress=True).messages[0]['content']
        # The real HTTP client is built once before sharing the router.
        router._get_kompress()
        with ThreadPoolExecutor(max_workers=2) as pool:
            earlier=pool.submit(apply)
            assert first_seen.wait(10),'first router call did not reach HTTP'
            later=pool.submit(apply)
            noop=later.result(timeout=10)
            release_first.set()
            compressed=earlier.result(timeout=10)
        replay=apply()
        reopened=ContentRouter(ContentRouterConfig(ccr_inject_marker=False,result_cache_disk=False))
        reopened._cache=CompressionCache(store=ResultStore(Path(d)/'cache.db',1800),namespace='proof:')
        disk_replay=reopened.apply(payload,tokenizer,force_kompress=True).messages[0]['content']
        print(json.dumps({'noop_is_original':noop==payload[0]['content'],'successful_request_is_compressed':compressed==short,'next_request_reuses_success':replay==short,'reopened_sqlite_reuses_success':disk_replay==short,'skip_entries':router._cache.skip_size,'http_calls':len(calls)},indent=2))
        reopened._cache._store._db.close()
        disk._db.close()
        router._get_kompress()._client.close()
finally:
    release_first.set();server.shutdown();server.server_close();thread.join(5)
```

The first-sight SQLite trace was also run outside pytest with this
script:

```python
"""Trace actual SQLite statements for first-sight negative cache entries."""
import json
from pathlib import Path
import tempfile
from headroom.cache.result_store import ResultStore
from headroom.transforms.content_router import CompressionCache

with tempfile.TemporaryDirectory(prefix='headroom-skip-io-') as directory:
    store=ResultStore(Path(directory)/'cache.db',1800)
    cache=CompressionCache(store=store)
    statements=[]
    store._db.set_trace_callback(statements.append)
    for i in range(100):cache.mark_skip(f'new-{i}')
    print(json.dumps({'new_skip_entries':cache.skip_size,'sqlite_statements':len(statements),'deletes':sum(s.startswith('DELETE ') for s in statements)}))
    store._db.close()
```

- Observed result: Two real ContentRouter.apply calls reach a bound
loopback RemoteKompress endpoint. The controller lets the no-op finish
first, then releases the successful compression. On clean main,
next_request_reuses_success=false and skip_entries=1, while reopening
SQLite reads the successful result. With the patch both the next request
and reopened cache reuse that result, skip_entries=0, and the endpoint
still receives only the original two calls. The trace script records 100
DELETEs for 100 new skips on the initial PR, versus zero SQL statements
after this revision.
- Not tested: Trained-model quality, paid endpoints, production
throughput, cross-process in-memory coherence, network filesystems, or
the complete Python suite. HTTP responses are deterministic synthetic
fixtures; actual router, HTTP client and SQLite operations are used.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing cache behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: A successful store supersedes an
older skip. First-sight negative classification keeps its existing
memory-only path.
- Kill switch / disable path: Revert this commit to restore the prior
successful-store behavior; no new configuration is introduced.
- Unsafe override required: No.
- Qualification impact: Related tests and original local gates passed;
hosted CI is tracked separately.
- Rollback path: Revert this commit; no dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Developed and validated with Codex assistance. I reviewed the initial
changes and runtime evidence; the review follow-up was checked with
Codex.



<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4819"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:04:48 -06:00
48f650ad04 test: make the trust_remote_code scan see every truthy value and run without git (#4785)
## Description

`tests/test_no_trust_remote_code.py` scans the tracked tree so that
nothing re-enables HuggingFace remote code execution. Its regex captured
the assigned value only when it started with a letter
(`([A-Za-z_][\w.]*)`), but the library tests the flag for truthiness, so
these also enable it and were not seen: `trust_remote_code=1`,
`trust_remote_code=(True)`, `trust_remote_code="yes"`, `setattr(cfg,
"trust_remote_code", True)` and `kwargs.setdefault("trust_remote_code",
True)`. The test also called `pytest.skip` when `git ls-files` failed
(for example in an exported tarball), so the guard silently stopped
running.

This PR widens the value rule (a quoted string, or a name or number,
with an optional leading `(`, treating anything except the exact name
`False` as enabling the flag), adds a second pattern for the `"name",
value` form, matches across a line break, and falls back to scanning
`headroom/**/*.py` when git is unavailable instead of skipping. The scan
of the current tree finds no offender with the wider rules.

Part of #4783

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `tests/test_no_trust_remote_code.py`: `_VALUE`, a wider `ASSIGNMENT`,
a new `NAMED_ARGUMENT` limited to
`setattr`/`getattr`/`setdefault`/`get`/`pop` calls, a `_values()` helper
that reports `(line, value)`, a git-less fallback in `_tracked_files()`,
and 17 new cases: calls that do not set the flag (`log.info("...",
True)`) must be ignored, every truthy shape must be flagged, a value on
the next line must be flagged, and an explicit `False` (including
`(False)`, `kwargs["..."] = False` and `setattr(..., False)`) must be
accepted.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`mypy headroom` was not run (no change under `headroom/`). `ruff check`
and `ruff format` (0.17.3) are clean.

### Test Output

The old regex on the shapes the old test never saw (`re.findall` with
the old pattern):

```text
old regex matches=True  trust_remote_code=True
old regex matches=False trust_remote_code=1
old regex matches=False trust_remote_code=(True)
old regex matches=False trust_remote_code="yes"
old regex matches=False setattr(cfg, "trust_remote_code", True)
old regex matches=False kwargs.setdefault("trust_remote_code", True)
```

The new tests with the old scan rule (value must start with a letter, no
`"name", value` pattern):

```text
6 failed, 10 passed   (the =1, ="yes", mapping =1, setattr, setdefault and setattr(..., False) cases fail)
```

The new tests with this branch's scan:

```text
tests/test_no_trust_remote_code.py   19 passed   (the tree scan finds no offender)
same file run from a copy of the tree without .git   19 passed, 0 skipped
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, pytest 9.2.0, `main` at
`3c66c91e`.
- Exact command / steps: `python -m pytest
tests/test_no_trust_remote_code.py`; then the same file from a copy of
the repository without `.git`; then the new tests with the scan logic
swapped for the old rule.
- Observed result: as above. The tree scan passes on the current tree
with the wider rules, so no existing code needs to change.
- Not tested: CI, and a deliberately injected `trust_remote_code=1` in a
real module (the shapes are covered by the unit cases instead).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; test only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for the library and proxy. The
scan is stricter and no longer skips without git.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (this
file; the whole suite was not run)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- A value such as `"False"` (a string) is truthy in Python, so the scan
flags it; I think that is right for this guard.
- The git-less fallback scans only `headroom/`: an exported tarball has
no other tracked-file list.

AI assistance: this change, its tests and the runs above were done with
Claude (Sonnet 5.5), operated by me (HetCreep).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4785"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:02:31 -06:00
dc94c46a1f test: remove the developer's GitHub and Copilot tokens from every test's environment (#4790)
## Description

Any test that enters the proxy lifespan starts the Copilot quota poller
(`headroom/subscription/copilot_quota.py`), which reads
`GITHUB_COPILOT_GITHUB_TOKEN`, `GITHUB_TOKEN`, `COPILOT_GITHUB_TOKEN` or
`GITHUB_COPILOT_API_TOKEN` from the environment and sends it as a
`Bearer` header to `api.github.com`. `tests/conftest.py` scrubs only
`HEADROOM_*`, so a developer with `GITHUB_TOKEN` in the shell has that
token used by the test run.
`tests/test_subscription_poll_credential_policy.py:1-8` already treats
using a credential nobody meant for this purpose as a defect.

This PR adds an autouse fixture that removes the nine Copilot/GitHub
token variables that `copilot_quota.py` and `copilot_auth.py` read.
Tests that need one still set it with `monkeypatch.setenv`, which runs
after the fixture.

Part of #4789

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `tests/conftest.py`: `_COPILOT_TOKEN_ENV_VARS` and the autouse fixture
`_scrub_developer_copilot_tokens`, placed before
`_scrub_developer_headroom_env`.
- `tests/test_conftest_scrubs_copilot_tokens.py`: five names must be
absent from `os.environ`, `discover_github_token()` finds nothing, and a
test can still set its own token. No product code changes.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`mypy headroom` was not run (no change under `headroom/`). `ruff check`
and `ruff format --check` (0.17.3) are clean on the two files.

### Test Output

With `GITHUB_TOKEN` and `GH_TOKEN` set to dummy values in the shell, the
new file:

```text
this branch:            7 passed
conftest from main:     3 failed, 4 passed
  (GITHUB_TOKEN and GH_TOKEN still in the environment, and discover_github_token() returns the dummy token)
```

The Copilot and GitHub-token related test files, with the same dummy
variables set:

```text
test_cli/test_wrap_copilot.py, test_cli/test_wrap_opencode.py, test_copilot_auth.py,
test_copilot_quota.py, test_offline_egress_chokepoint.py
  this branch:        1 failed, 435 passed, 4 skipped
  conftest from main: 1 failed, 435 passed, 4 skipped
test_release_workflows.py, test_subscription_poll_credential_policy.py,
test_copilot_subscription_smoke.py
  this branch:        86 passed
```

The one failure is
`tests/test_offline_egress_chokepoint.py::TestUpdateCheckOffline::test_explicit_update_check_never_constructs_request`
(`AttributeError: module 'headroom.cli.main' has no attribute 'name'`).
It fails the same way with `main`'s `conftest.py` and passes when run
alone with either, so it depends on test order and is not caused by this
change; I did not investigate it.

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, pytest 9.2.0, the compiled
`headroom._core`, `main` at `3c66c91e`.
- Exact command / steps: `GITHUB_TOKEN=ghp_dummy_not_real
GH_TOKEN=dummy2 python -m pytest
tests/test_conftest_scrubs_copilot_tokens.py`, once with this branch's
`conftest.py` and once with `main`'s; then the two groups of files above
with the same variables.
- Observed result: as in Test Output. Only dummy values were used, never
a real token (the runs of the other files with the dummy variables may
have sent them to GitHub, where they are rejected).
- Not tested: CI, Linux and macOS, the whole suite, and a network
capture of the poller (the 53 lookups in #4789 come from an instrumented
run, not from this PR's proof).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; test only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (the
files listed above; the whole suite was not run, and one order-dependent
failure exists on `main`)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Not in this PR: the other items of #4789 (credential readers in
`test_copilot_auth.py`, writes to `~/.headroom`, tests that reach the
network).
- If a CI job relies on a token reaching a test, that test would now
need `monkeypatch.setenv`; I found none in the files above.

AI assistance: this change, its tests and the runs above were done with
Claude (Sonnet 5.5), operated by me (HetCreep).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4790"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:01:59 -06:00
ParideboyandJerrettDavis 327d92414f fix(cache): keep literal cache_control fields in the response-cache key (#4799)
## Description

The proxy response-cache key policy strips `cache_control` recursively
at every dict depth, with a single carve-out for JSON Schema
`properties` names (#327). But `cache_control` is overloaded: on a
content block, a `system` entry or a tool definition it is a transport
prompt-cache directive, while inside tool arguments and JSON Schema
literals it is ordinary application data the model actually sees.

So two requests that differ only in that data get the same key. With
`cache_enabled` on by default, the second caller is replayed the first
caller's response for the full TTL.

Reproduced on `main` (`e995eb9f`) against the real `SemanticCache` API,
no mock:

```python
msgs = [{"role": "assistant", "content": [{"type": "tool_use", "id": "c1",
         "name": "configure", "input": {"cache_control": "first"}}]}]
changed = copy.deepcopy(msgs); changed[0]["content"][0]["input"]["cache_control"] = "second"
c = SemanticCache()
await c.set(msgs, "m", b'{"id":"first"}', {}, partition="caller")
print("replayed:", (await c.get(changed, "m", partition="caller")) is not None)
```

Before (this branch's parent, `e995eb9f`):

```text
changed tool payload replayed: True
changed schema replayed: True
```

Closes #4272

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/semantic_cache_key_policy.py`: `strip_cache_control`
no longer recurses into fields whose value is opaque payload — `input`,
`arguments`, `input_schema`, `parameters`, `schema`, `properties`,
`const`, `default`, `enum`, `example`, `examples`, `text` — so a
`cache_control`-named field there stays in the key. Everywhere else
(content blocks, `system`, tool definitions) it is still dropped, so a
moved transport breakpoint still does not fragment the key. The narrower
`properties`-only carve-out falls out of the general rule.
- `tests/test_proxy_semantic_cache_key_policy.py`: covers the
mandatory-partition precondition (`ValueError` when no partition is
passed), which was the one uncovered line Codecov flagged in this
change.
- `tests/test_semantic_cache_literal_payloads.py`: new regression suite
covering tool `input`, tool `input_schema` `default`, and
`response_format` schema `const`, each through the real `SemanticCache`
(changed request misses, original still hits), plus a guard for the
decoded-dict `function.arguments` shape Headroom accepts
(`handlers/openai.py` parses either), a guard that OpenAI's
JSON-*string* `arguments` is passed through verbatim, and guards that
transport `cache_control` is still ignored and nested transport
annotations are still dropped.

One pure module changed; no provider, no native rebuild.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`mypy` is unchecked because the pre-commit hook fails on 11 pre-existing
Windows-only `attr-defined` errors in files this PR does not touch
(`headroom/fileperms.py`, `headroom/ccr/mcp_server.py`,
`headroom/proxy/interceptors/astgrep.py`: `os.fchmod`, `os.O_DIRECTORY`,
`fcntl`); neither changed file produces a mypy error.

### Test Output

Regression tests fail before the fix. With
`headroom/proxy/semantic_cache_key_policy.py` reverted and the new tests
kept:

```text
$ git show HEAD~1:headroom/proxy/semantic_cache_key_policy.py > headroom/proxy/semantic_cache_key_policy.py
$ uv run pytest tests/test_semantic_cache_literal_payloads.py -q
FAILED tests/test_semantic_cache_literal_payloads.py::test_literal_cache_control_does_not_alias_cached_responses[tool_input]
FAILED ... [tool_arguments_dict]
FAILED ... [tool_schema]
FAILED ... [response_schema]
FAILED tests/test_semantic_cache_literal_payloads.py::test_literal_cache_control_changes_the_key[tool_input]
FAILED ... [tool_arguments_dict]
FAILED ... [tool_schema]
FAILED ... [response_schema]
FAILED tests/test_semantic_cache_literal_payloads.py::test_strip_cache_control_still_drops_nested_transport_annotations
9 failed, 3 passed in 0.64s
```

The three that pass against the old policy are the guards: the OpenAI
string-`arguments` case (a string is never rewritten), the
schema-property-name
case (#327), and the moved-transport-breakpoint case.

After the fix:

```text
$ uv run pytest tests/test_semantic_cache_literal_payloads.py tests/test_proxy_semantic_cache_key_policy.py \
    tests/test_semantic_cache_key_policy.py tests/test_proxy_response_cache_partition.py \
    tests/test_proxy_semantic_cache_key.py tests/test_proxy_semantic_cache_key_integration.py \
    tests/test_openai_response_cache_key.py tests/test_proxy_openai_cache_key_integration.py -q
94 passed, 1 warning in 28.39s

$ uv run pytest <the four policy/partition files> -q \
    --cov=headroom.proxy.semantic_cache_key_policy --cov-report=term-missing --cov-branch
Name                                          Stmts   Miss Branch BrPart  Cover
headroom\proxy\semantic_cache_key_policy.py      54      0     16      0   100%
```

$ uv run pytest tests/test_cache/test_semantic.py
tests/test_openai_response_cache_key.py \
tests/test_proxy_openai_cache_key_integration.py
tests/test_proxy_response_cache_store_guard.py \
tests/test_proxy_semantic_cache_key.py
tests/test_proxy_semantic_cache_key_integration.py \
    tests/test_shared_context.py -q
133 passed, 1 warning in 34.11s

$ uv run pytest tests/test_cache_control_move_bust.py
tests/test_anthropic_fork_lineage_cache.py \
tests/test_cross_turn_cache_safety.py
tests/test_cache/test_prefix_tracker.py -q
113 passed, 1 warning in 6.21s

$ uv run ruff check headroom/proxy/semantic_cache_key_policy.py
tests/test_semantic_cache_literal_payloads.py
All checks passed!
$ uv run ruff format --check <both files>
1 file already formatted
```

## Real Behavior Proof

- Environment: Windows 11, `uv run` against the project venv (Python 3.13.11), branch `fix/4272-literal-payload-cache-keys` cut from upstream `main` `e995eb9f`. No mocked cache and no provider: the reproducer calls the real `SemanticCache.set`/`get` in `headroom/proxy/semantic_cache.py`.
- Exact command / steps: run the reproducer above as a script under `uv run python` — store a response under a tool input containing `{"cache_control": "first"}`, deep-copy the messages, change that value to `"second"`, then `get` with the changed messages; repeat for a tool `input_schema` `default` and a `response_format` schema `const`.
- Observed result: before the fix the changed request hit the cache in both cases (`changed tool payload replayed: True`, `changed schema replayed: True`). After the fix both print `False`, and the original request still hits (asserted in each regression test), so the key was not merely widened.
- Not tested: the real OpenAI HTTP path with a decoded-dict `arguments` field, which is a nonstandard client encoding (`arguments` is normally a JSON string, covered by `test_openai_arguments_string_is_not_rewritten`); the full `make ci-precheck` (Rust/native rebuild) and the end-to-end proxy HTTP path; the change is confined to the pure key policy, which the tests above exercise directly through the real `SemanticCache`. `headroom/cache/prefix_tracker.py` has the same blunt recursive strip, but it serves prefix-content equality rather than response replay and is left as a separate follow-up.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; the proxy response cache (`cache_enabled`) is not a rollout-gated feature.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, deliberately — for requests whose tool data or JSON Schema values contain a `cache_control`-named field, two distinct requests that used to share one key now get distinct keys. Requests without such a field produce the same key as before, so hit rate there is unchanged.
- Kill switch / disable path: `cache_enabled = false`, or the existing proxy config for the response cache; no new switch.
- Unsafe override required: no
- Qualification impact: none; no rollout-managed surface touched.
- Rollback path: revert this commit. Cache keys are process-local (`SemanticCache` is in-memory), so a revert needs no cache migration or invalidation.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — not needed: the key policy's docstrings describe the new rule, and no user-facing docs claim `cache_control` is stripped from payloads.
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this)

## Additional Notes

`headroom/cache/prefix_tracker.py:165` has the same blunt recursive strip, used for prefix-content equality rather than response replay. Worth a follow-up, deliberately out of scope here so the response-replay fix stays a one-module change.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:01:23 -06:00
Yuchuan HeandJerrettDavis 973c12d2af fix(cache): replay every parallel tool result (#4818)
## Description

An Anthropic user message can hold several parallel `tool_result`
blocks. The session compression cache recorded and replayed only the
first one, but a hit for that result could freeze the whole message.
Later results then reached the provider in their raw form instead of the
previously compressed form.

Visit every result when recording, replaying and marking stable content.
Freeze a message only if every result is replayable or textless. Reuse
the existing single-result shape guard, so images, tool references,
multiblock boundaries and text-block metadata retain their established
protection. Cache seeding checks the paired tool-call IDs before
associating original and compressed text.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Cache and replay each parallel result independently without mutating
caller input.
- Stop the frozen prefix on a later miss or unsupported text layout.
- Cover a textless first result, later text-block metadata, changed IDs
and partial replay. Update the existing later-result limitation test to
assert actual replay.

## Testing

- [x] Relevant unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Formatting passes (`ruff format --check .`)
- [x] Type checking passes (original full-package mypy hook)
- [x] Regression tests added
- [x] Standalone runtime validation completed

### Test Output

```text
New regressions against original production code: 5 failed, 1 passed
pytest tests/test_compression_cache.py tests/test_session_engine.py tests/test_cache_replay_shapes.py tests/test_token_headroom_mode.py tests/test_cache_prefix_safety.py tests/test_proxy_anthropic_cache_stability.py tests/test_prefix_tracker_restart.py:
  189 passed
Original pre-commit and commit-msg hooks: passed
Whole-tree Ruff check and format --check: passed
Original pre-push make ci-precheck: passed
  Rust workspace: 1749 passed, 3 ignored
  Native extension: rebuilt, installed and import-verified
  Critical Python: 177 passed, 8 skipped
  commitlint: passed
```

Optional-model skips and the existing token-count approximation warning
remain visible in the gate logs. The restart tests also report the
environment's existing Starlette/httpx deprecation warning.

## Real Behavior Proof

- Environment: Windows 11, CPython 3.13.9, existing dev venv; clean
baseline `c05fbdddc873078dd97cb9ede841e75ef7ed0bef`, patch
`a09ad47dc8ceb04b42858a67eecb83a7ee246485`. Explicit checkout on
PYTHONPATH; no provider credentials. Headroom runs offline; the parallel
proof uses a real SDK to send only to its bound loopback receiver.
- Exact command / steps: Save the following script as `proof.py` and run
`python proof.py` from either checkout with `PYTHONPATH=.`. Identical
before/after scripts ran outside pytest.

```python
"""Production cache/session APIs followed by real Anthropic SDK loopback HTTP."""
import copy
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import threading
from anthropic import Anthropic
from headroom.cache.compression_cache import CompressionCache
from headroom.proxy.session_engine import FREEZE_POLICY_REPLAYABLE, prepare_turn

captured = []
class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        captured.append(json.loads(self.rfile.read(int(self.headers['Content-Length']))))
        data = json.dumps({'id':'local','type':'message','role':'assistant','model':'local-fixture','content':[{'type':'text','text':'received'}],'stop_reason':'end_turn','stop_sequence':None,'usage':{'input_tokens':1,'output_tokens':1}}).encode()
        self.send_response(200);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
    def log_message(self,*args): pass

def message(a,b):
    return {'role':'user','content':[{'type':'tool_result','tool_use_id':'t1','content':a},{'type':'tool_result','tool_use_id':'t2','content':b}]}

server = ThreadingHTTPServer(('127.0.0.1',0),Handler)
thread = threading.Thread(target=server.serve_forever,daemon=True);thread.start()
raw = message('first original '*20,[{'type':'text','text':'second original '*20,'cache_control':{'type':'ephemeral'}}])
short = message('first compressed',[{'type':'text','text':'second compressed','cache_control':{'type':'ephemeral'}}])
messages=[{'role':'assistant','content':[{'type':'tool_use','id':'t1','name':'inspect','input':{}},{'type':'tool_use','id':'t2','name':'inspect','input':{}}]},raw,{'role':'user','content':'continue'}]
before = copy.deepcopy(messages)
cache = CompressionCache();cache.update_from_result([raw],[short])
prep = prepare_turn(cache,messages,policy=FREEZE_POLICY_REPLAYABLE)
try:
    with Anthropic(api_key='local-fixture',base_url=f'http://127.0.0.1:{server.server_port}',max_retries=0) as client:
        client.messages.create(model='local-fixture',max_tokens=16,messages=prep.pipeline_input)
    wire = captured[-1]['messages'][1]
    print(json.dumps({'all_results_replayed':wire==short,'second_result':wire['content'][1]['content'],'frozen_count':prep.frozen_message_count,'caller_unchanged':messages==before,'requests':len(captured)},indent=2))
finally:
    server.shutdown();server.server_close();thread.join(5)
```

- Observed result: With the same original/compressed message pair, clean
main freezes the two-message prefix but the real Anthropic SDK sends the
second result as its original text. With the patch, the HTTP receiver
gets both compressed results, including the second text block's
cache_control; frozen_count remains 2 and caller_unchanged remains true.
- Not tested: Live Anthropic service acceptance, model inference,
billing, task accuracy, or the complete Python suite. The receiver
records actual SDK JSON and returns a synthetic response.

## Runtime Rollout Safety

- Rollout-managed feature(s): Existing cache behavior; no new feature
flag or channel expansion.
- Minimum rollout channel: Normal release after maintainer review and CI
qualification.
- Stable/default behavior changed: Cached replay covers every tool
result in a message; a later cache miss or unsupported shape keeps that
message mutable.
- Kill switch / disable path: Revert this commit to restore the prior
replay behavior; no new configuration is introduced.
- Unsafe override required: No.
- Qualification impact: Related tests and original local gates passed;
hosted CI is tracked separately.
- Rollback path: Revert this commit; no dependency or storage-schema
change.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix
- [x] Relevant new and existing tests pass locally
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Developed and validated with Codex assistance. I reviewed the changes
and runtime evidence.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4818"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 23:00:39 -06:00
d7a2fcf092 test: make the loopback-gating tests able to tell the guard from a missing key (#4784)
## Description

`tests/test_proxy_loopback_gating.py` says non-loopback callers get a
404 on the gated routes, but `test_non_loopback_caller_gets_404` used a
default `TestClient`, whose peer is the string `"testclient"` (not an
IP), and asserted only the 404 status. Two of the twelve gated routes,
`/v1/telemetry/tools/{hash}` and `/v1/toin/pattern/{prefix}`, answer 404
on their own when the key is absent, and `test_loopback_caller_allowed`
accepts `{200, 404, 422}`. So removing the guard from those routes, or
making it refuse every parseable IP, did not turn the test red.

The guard's denial body is `{"detail": "Not Found"}` (what FastAPI
answers for an unknown path); a detail route answers with its own
message. This PR (1) uses a routable remote peer (`203.0.113.9`,
TEST-NET-3) with a loopback `Host`, so only the client-IP check can
refuse it, (2) asserts the guard's exact denial body, and (3) asserts
that a loopback caller never receives that body.

Part of #4783

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `tests/test_proxy_loopback_gating.py`: a `GUARD_DENIAL` constant with
a comment saying why; `test_non_loopback_caller_gets_404` uses
`client=("203.0.113.9", 4242)` and `base_url="http://127.0.0.1"` and
asserts `resp.json() == GUARD_DENIAL`; `test_loopback_caller_allowed`
also asserts `resp.json() != GUARD_DENIAL`. No product code changes.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`mypy headroom` was not run (no change under `headroom/`). `ruff check`
and `ruff format --check` (0.17.3) are clean.

### Test Output

Whole file, product unchanged:

```text
pytest tests/test_proxy_loopback_gating.py   147 passed
```

The same file run against two mutations of the product (each restored
afterwards), with the old tests (`origin/main`) and the new ones:

```text
Mutation A: loopback_guard.py  `return address.is_loopback`  ->  `return True`
  old tests: 20 failed, 127 passed
  new tests: 32 failed, 115 passed      (the 12 non-loopback cases now fail too)

Mutation B: remove `dependencies=[Depends(_require_loopback)]` from
            /v1/telemetry/tools/{signature_hash} and /v1/toin/pattern/{hash_prefix:path}
  old tests: 147 passed                 (nothing noticed)
  new tests: 2 failed, 145 passed       (the two routes)
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, pytest 9.2.0, the compiled
`headroom._core`, `main` at `3c66c91e`.
- Exact command / steps: `create_app(ProxyConfig(...))` and three
`TestClient`s (default peer, `client=("203.0.113.9", 4242)`,
`client=("127.0.0.1", 12345)`, the last two with
`base_url="http://127.0.0.1"`); `GET` each of
`/v1/telemetry/tools/example`, `/v1/toin/pattern/example`,
`/v1/feedback/example`; then the two mutations above.
- Observed result: both non-loopback peers got `404 {"detail":"Not
Found"}` on all three; the loopback peer got `404 {"detail":"No
telemetry found for signature: example"}`, `404 {"detail":"No TOIN
pattern found with hash starting with: example"}` and `200`
respectively. The mutation results are in Test Output.
- Not tested: CI, other platforms, and the two non-GET gated routes
under mutation B (they are covered by the new body assertion but I did
not mutate them).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; test only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (this
file; the whole suite was not run)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- The vanilla `TestClient` (peer `"testclient"`) is also refused by the
guard; I did not keep a test for it because it is not a
loopback-versus-remote distinction.

AI assistance: this change, its tests and the runs above were done with
Claude (Sonnet 5.5), operated by me (HetCreep).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4784"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 22:59:20 -06:00
854a696544 fix(proxy): retry a TLS error only while the request is being written (#4776)
## Description

A corrupt TLS record after the handshake surfaces as a raw
`ssl.SSLError` (for example `[SSL: RECORD_LAYER_FAILURE] record layer
failure`, or the `SSLV3_ALERT_BAD_RECORD_MAC` bursts in #1615). httpcore
maps `ssl.SSLError` to `ConnectError` only in `start_tls`; its
`read`/`write` maps cover only anyio's resource errors and timeouts, and
anyio's `TLSStream` re-raises a non-EOF `SSLError` as it is. So the
error is not an `httpx.TransportError`, and the #1639 retry in
`_retry_request` and `_stream_response` lets it through.

That error alone does not say whether upstream received the request.
This PR retries it only when httpcore reports that writing the request
failed: its documented `trace` request extension emits
`send_request_headers.failed` or `send_request_body.failed`. Upstream
then never had a complete request to run. A TLS error raised after the
request was written propagates, as on main, so a POST upstream may have
accepted is never sent twice. In the streaming path it still ends as the
SSE 502, now with `upstream_tls_error` instead of a generic `api_error`.

What this no longer fixes: the case the first version of this PR
targeted, a corrupt record in the reply on a reused HTTP/1.1 keep-alive
connection, happens after the request was written. It is ambiguous, so
it still fails the request (502), as on main. What it does fix: a TLS
failure while the request is still being written, for example on a
shared HTTP/2 connection where a large body waits for flow-control
window and the read for that window hits the corrupt record.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `helpers.py`: `RequestWriteTrace`, an httpcore `trace` callback that
records whether `send_request_headers` or `send_request_body` failed.
`forbids_retry(exc)` is true for a raw `ssl.SSLError` raised after a
complete write.
- `server.py` (`_retry_request`) and `handlers/streaming.py`
(`_stream_response`): each attempt attaches a fresh trace, and a raw
`ssl.SSLError` is retried only when that trace shows the write failed.
Every other retry is unchanged. The streaming exhaustion handler still
catches `ssl.SSLError`, for the 502 `upstream_tls_error` SSE body.
- `tests/test_h2_stream_reset_retry.py`: the SSL cases now run through
real httpcore over a fake socket. An accepted POST sends one POST and
propagates, in the buffered and the streaming path. A failed write is
retried, in both paths. The review's harness (an `httpx.MockTransport`
that records the POST, then raises `ssl.SSLError`) sends one POST.
- Three test fakes of `http_client.post` accept the new `extensions`
kwarg.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_h2_stream_reset_retry.py -q
11 passed in 0.39s

# source reverted to the previous head's retry (240680df), tests kept:
FAILED tests/test_h2_stream_reset_retry.py::test_retry_request_does_not_replay_a_post_upstream_received   (assert 2 == 1)
FAILED tests/test_h2_stream_reset_retry.py::test_stream_does_not_replay_a_post_upstream_received          (assert 2 == 1)
FAILED tests/test_h2_stream_reset_retry.py::test_ssl_error_without_write_evidence_is_not_retried          (assert 2 == 1)
3 failed, 8 passed

# source as on upstream/main (no SSL retry), tests kept:
FAILED tests/test_h2_stream_reset_retry.py::test_retry_request_retries_ssl_error_while_writing_the_request
FAILED tests/test_h2_stream_reset_retry.py::test_stream_does_not_replay_a_post_upstream_received   (raw SSLError escapes; no upstream_tls_error 502)
FAILED tests/test_h2_stream_reset_retry.py::test_stream_retries_ssl_error_while_writing_the_request
3 failed, 8 passed

$ uv run --frozen --extra dev pytest <134 files: the tests this PR and main changed since 200c838c, every test importing handlers.streaming, and every test calling _retry_request, _stream_response or a fake http_client> -q
2457 passed, 2 failed, 22 skipped in 178s
# both failures are timing assertions in test_read_maturation_handler_nobust.py, which replaces
# _retry_request with a mock. Run alone 3 times, that file fails 1 test in 3 of 3 runs on this head
# and in 2 of 3 runs with server.py, streaming.py and helpers.py as on upstream/main.

$ uvx ruff@0.17.3 check <touched files>             All checks passed!
$ uvx ruff@0.17.3 format --check <touched files>    already formatted
$ mypy headroom/proxy/helpers.py headroom/proxy/server.py headroom/proxy/handlers/streaming.py --ignore-missing-imports   Success: no issues found in 3 source files
```

## Real Behavior Proof

- Environment: macOS 27.2 (Apple Silicon), Python 3.13, httpx 0.28.1,
httpcore 1.0.9, anyio 4.14.2. `headroom proxy --port 6811
--no-rate-limit --anthropic-api-url https://127.0.0.1:8445` (plus
`--no-http2` in run 1), a clean temp HOME per run, `HEADROOM_CA_BUNDLE`
pointing at a local test CA. Each run starts its own stub and proxy and
stops both.
- Exact command / steps: Controlled upstream: a local TLS stub, not a
real provider. Run 1 (HTTP/1.1, accepted POST): the stub reads the whole
request and counts it, then writes a corrupt TLS application-data record
instead of a reply and closes; a byte-identical replay of a body it has
seen gets a normal 200. One buffered and one streaming `POST
/v1/messages` through the proxy, on upstream/main 9f2cc017, on the
previous head 240680df and on this head. Run 2 (HTTP/2, write failure):
on the first connection the stub grants no flow-control window beyond
the default 65,535 bytes, so a 418,702-byte streaming POST is still
being written when the stub writes the corrupt record; later connections
are served normally. On main and on this head.
- Observed result: Run 1: main and this head send each POST once and
return 502; 240680df sends each POST twice, byte-identical, and returns
the second reply as 200. Run 2: main returns 502 `api_error`; this head
retries on a fresh connection and returns 200, and the stub received one
complete POST.
- Not tested: A real provider sending a corrupt record, and a real
HTTP/2 connection shared by several streams when one of them hits it.

```text
Run 1: controlled upstream, HTTP/1.1, TLS broken after the whole POST was received
version         request     POSTs the stub received    client got
upstream/main   buffered    1                          HTTP 502 api_error
upstream/main   streaming   1                          HTTP 502 api_error
240680df        buffered    2 (byte-identical)         HTTP 200 (reply to the replay)
240680df        streaming   2 (byte-identical)         HTTP 200 (reply to the replay)
this head       buffered    1                          HTTP 502 api_error
this head       streaming   1                          HTTP 502, SSE error event with "code": "upstream_tls_error"

Run 2: controlled upstream, HTTP/2, TLS broken at 65,535 of 418,702 body bytes
upstream/main   streaming   1 incomplete               HTTP 502 api_error
this head       streaming   1 incomplete, 1 complete   HTTP 200 (message_start ... message_stop)
  WARNING [hr_1794561952_000001] Connection error to upstream API (attempt 1/3): SSLError(1, '[SSL: RECORD_LAYER_FAILURE] record layer failure (_ssl.c:2660)'); retrying in 899ms
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes. A raw TLS error raised while the
request is being written is retried on a fresh connection, like other
transport errors. A raw TLS error after the request was written is not
retried, as on main; in the streaming path it now ends as a 502
`upstream_tls_error` SSE error instead of a 502 `api_error`.
- Kill switch / disable path: `--retry-max-attempts 1`
(`HEADROOM_RETRY_MAX_ATTEMPTS=1`) disables the retry, as for every
transport error.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

- Changed after review: the first version retried any raw `ssl.SSLError`
before the first byte reached the client. JerrettDavis and Devin showed
that replays a POST upstream may already have accepted. This version
retries only a failed write, and the title now says so.
- No idempotency key is added. The retry rests only on httpcore
reporting that the request write failed.
- Not changed here: the existing #1639 retries of `httpx.TransportError`
(for example a `ReadError` or `ReadTimeout` after the request was
written) carry the same after-the-write ambiguity. This PR leaves them
as they are.
- Companion PR #4775 is independent of this change. The
cancellation-stress numbers quoted in the earlier version of this body
were measured with the broader retry, so I removed them.
- Documentation: no doc describes the retry classes, so there is nothing
to update.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 22:56:52 -06:00
998d71080c fix(proxy): keep Copilot requests on Copilot when an extension advises another provider (#4894)
## Description

A proxy in front of GitHub Copilot (started by `headroom wrap vscode` /
`wrap copilot`) holds only a Copilot token. When a proxy extension
published cross-provider route advice (`request.state.headroom_route`),
`BackendResolver.for_request` built a translating LiteLLM backend for it
without looking at the upstream. A GPT-family Copilot call advised onto
a Claude model was sent to Anthropic directly and failed:

```
LiteLLM backend initialized (provider=anthropic, region=None)
route advice: claude-sonnet-5-5 via anthropic (...)
/stats -> failed_by_provider: litellm-anthropic
```

The resolver now takes the request's upstream and ignores cross-provider
advice when that upstream is Copilot.

Companion change: headroomlabs-ai/routemegood#49 stops the routing
extension publishing such advice in shadow mode and for holdouts. This
PR is the safety property for apply mode.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/route_advice.py`: `for_request` takes `upstream_url`.
When `is_copilot_upstream_url(upstream_url)` it returns the default
backend and leaves `body["model"]` alone, with one info log line (model
id and provider only).
- `headroom/proxy/handlers/openai.py`, `handlers/anthropic.py`: all five
call sites pass the request's upstream. The Anthropic sites are included
because `wrap vscode` points the Anthropic target at Copilot as well.
- `handlers/openai.py`: chat-stream CCR eligibility asks the resolver
whether a backend will serve the request, instead of treating any advice
as "routed to backend". Ignored advice no longer disables CCR tool
injection.
- `tests/test_route_advice_native_providers.py`: a resolver-level test
(GHE Copilot host) and an end-to-end test through `create_app`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
CI on this PR: 34 checks passed, 12 skipped, 0 failed (lint, commitlint, test shards 1-4, test-macos).
Locally: ruff 0.17.4 check + format --check clean on the 4 changed files; mypy clean on the 3 source files.
tests/test_route_advice_native_providers.py: 8 passed.
With the Copilot check disabled, test_copilot_chat_stays_on_copilot_despite_cross_provider_advice fails with: assert ['anthropic'] == []
```

## Real Behavior Proof

- Environment: macOS 27.2.1, Python 3.12.13, core run from source (this
branch vs `origin/main` 5023c84e), `routemegood` from its unfixed
`origin/main` (f53bcb5) loaded as the only proxy extension with a valid
licence, default (shadow) mode. Upstream: the real GitHub Copilot API on
a signed-in Copilot account, proxy started by `headroom wrap vscode -p
8795 --settings-file <temp file>`.
- Exact command / steps: start the wrap proxy, then send one
Copilot-shaped call, and read `/stats` and the proxy log:
  ```
  curl -s http://127.0.0.1:8795/chat/completions \
-H "Authorization: Bearer placeholder" -H "Content-Type:
application/json" \
-H "Copilot-Integration-Id: vscode-chat" -H "Editor-Version:
vscode/1.145.2" \
-d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Reply
with the single word OK"}],"max_tokens":16,"stream":false}'
  ```
- Observed result:

**Before (`origin/main` 5023c84e)** — the request leaves Copilot and
fails at Anthropic:
  ```text
  HTTP 401
error: litellm.AuthenticationError: AnthropicException -
{"type":"error","error":{"type":"authentication_error","message":"invalid
x-api-key"}, ...}
/stats: {'total': 0, 'failed': 0, 'by_model': {}, 'by_provider': {},
'failed_by_provider': {'unknown': 0}}
09:33:54,229 headroom.backends.litellm - INFO - LiteLLM backend
initialized (provider=anthropic, region=None)
09:33:54,234 headroom.proxy.route_advice - INFO - route advice:
claude-sonnet-5-5 via anthropic (cannot cache (prefix 7 < min 512); ...
09:33:54,438 headroom.backends.litellm - ERROR - LiteLLM OpenAI error:
litellm.AuthenticationError: AnthropicException - ... "invalid
x-api-key" ...
09:33:54,438 headroom.proxy - INFO - event=proxy_inbound_response ...
method=POST path=/chat/completions status=401
  ```

**After (this branch, fcfa965b)** — same extension, same request, served
by Copilot:
  ```text
  HTTP 200
  served model: gpt-4o-mini-2024-07-18 | reply: OK
/stats: {'total': 1, 'failed': 0, 'by_model': {'gpt-4o-mini': 1},
'by_provider': {'copilot': 1}, 'failed_by_provider': {'unknown': 0}}
09:34:11,511 headroom.proxy.route_advice - INFO - route advice ignored:
claude-sonnet-5-5 via anthropic; the upstream is GitHub Copilot
09:34:13,044 headroom.proxy - INFO - event=proxy_inbound_response ...
method=POST path=/chat/completions status=200
  ```
- Not tested: a request sent by the VS Code UI itself (the call above is
the same shape, sent with curl); a streaming request; the
Anthropic-handler (`/v1/messages`) call sites end to end; a custom
`GITHUB_COPILOT_PROXY_URL` host; Windows.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: only for proxies whose upstream is a
GitHub Copilot host, and only when an extension publishes cross-provider
route advice. Those requests now stay on Copilot instead of being sent
to another provider.
- Kill switch / disable path: none added. Not loading the extension that
publishes advice restores the old path trivially.
- Unsafe override required: no.
- Qualification impact: none expected.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation: no user-facing docs describe route advice, so none were
changed.
- A custom `GITHUB_COPILOT_PROXY_URL` host is recognised only when that
variable is set, since recognition goes through the existing helper.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-17 09:50:56 +05:30
ac932eb7c4 feat(telemetry): split beacon sessions by harness, model and day (#4881)
## Description

After backfilling the Beacon corpus, six questions about Oct 19 – Nov 14
could not be answered from what the client sends:

1. **Which harness, for a quarter of sessions.** 228,441 of 933,044
schema-v2 sessions (36.2M turns) carry no client. 30.2M of those turns
came through `/v1/compress`.
2. **Tokens and savings per harness.** `clients` has turn counts only;
34,553 sessions mix harnesses and cannot be divided.
3. **Harness version.** Only the Headroom version is reported.
4. **Tokens and savings per model.** `models` is a list, so savings
cannot be priced.
5. **Real installs versus churned ids.** Linux proxies show 19,916
installs for about the traffic of 4,258 on macOS.
6. **Per-day figures.** Always-on proxies report one session that grows
for weeks (34% of savings sat in multi-day sessions).

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

All client changes are in `headroom/telemetry/session.py`, from data the
outcome funnel and the HTTP middleware already have. No handler changes.
`schema_version` stays 2; the fields are additive.

| Field | Content |
|---|---|
| `client_usage` | Per-harness rows of `n, original, saved, tool_saved,
output, cache_read, cache_write, cache_write_inferred, cache_write_5m,
cache_write_1h, uncached`. Covers every turn: unclaimed turns are
`via_mcp`, `via_compress_api` or `unidentified`. |
| `model_usage` | The same columns per public model id; anything else
pools into `other`. |
| `client_versions` | `(client, version, n)` from the User-Agent, e.g.
`claude_code 2.1.311`. Request counts, not turns. |
| `install` | `age_days` (id file mtime), `container`, `ci`. |
| rollover | A session at least 12h old is closed by the first request
on a later UTC day (`ended: "rollover"`). Sessions that merely run past
midnight are untouched. |

- `clients` and `models` are unchanged.
- `deploy/beacon/worker.js` learns the new keys and their nested shapes.
- `headroom/proxy/auth_policy.py` gains
`client_version_from_user_agent`; `headroom/proxy/server.py` calls it
for every POST.

**Pricing a usage row (review feedback).** Every column is the raw
per-turn counter and sums to its session total. Where a provider reports
no write count, Headroom infers one as input minus cached; those are the
same tokens as `uncached`. `cache_write_inferred` is that part, so
billed input is `uncached + cache_read + (cache_write -
cache_write_inferred)`, the rule `CostTracker.record` applies, and
`cache_write_5m` / `cache_write_1h` give the TTL split.
`test_a_usage_row_prices_inferred_cache_writes_the_way_the_cost_tracker_does`
covers the 1000-not-2000 case.

**Privacy.**
- Harness names in the new fields are checked against a closed list
(`_KNOWN_CLIENTS`), because `classify_client` returns the caller's
`X-Client` header as typed. Unknown names become `other`. The existing
`clients` field has the same gap and is not changed here.
- A version is sent only if it directly follows a known harness prefix
and is one to three numeric parts.
- The Worker holds the new string leaves to a slug or version pattern;
its test asserts injected text, hostnames and wrong types are dropped.

## Testing

- [x] Unit tests pass (`pytest`) — the suites listed below, not the
whole repository
- [x] Linting passes (`ruff check .`) — on the changed files
- [ ] Type checking passes (`mypy headroom`) — 8 errors in
`headroom/cache/prefix_tracker.py` and `headroom/ccr/mcp_server.py`,
identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_beacon_schema_v2.py tests/test_telemetry_session_selfcheck.py \
    tests/test_telemetry.py tests/test_beacon_gzip.py tests/test_auth_policy.py tests/test_header_isolation.py -q
============================= 165 passed in 11.75s =============================
$ node deploy/beacon/test-rollup.mjs
ok
$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
```

Not run: the full Python suite. 3 tests in
`tests/test_proxy_telemetry_env.py` fail the same way on `main` without
this change.

## Real Behavior Proof

- Environment: macOS (darwin arm64), Python 3.12.13, this branch's proxy
(`service.version 0.45.0-dev`) with an isolated
`HEADROOM_WORKSPACE_DIR`, `HEADROOM_BEACON=on`,
`HEADROOM_TELEMETRY_ENDPOINT=http://127.0.0.1:8791/v1/logs`. Receiver:
this branch's `deploy/beacon/worker.js` `fetch()` served by a 30-line
Node HTTP listener with the R2 binding replaced by a directory.
Providers: Anthropic `claude-haiku-4-5-20251001` and OpenAI
`gpt-4o-mini`, live.
- Exact command / steps:
  1. `node receiver.mjs deploy/beacon/worker.js ./r2 8791`
  2. `headroom proxy --port 8799` (from this branch)
3. Four POSTs through the proxy: two `/v1/messages` with `User-Agent:
claude-cli/2.1.311 (external, cli)`, one `/v1/messages` with
`User-Agent: curl/8.4.0`, one `/v1/chat/completions` with `User-Agent:
codex-cli/0.45.0`.
  4. `kill -INT` the proxy so it sends its closing report.
- Observed result: the three Anthropic calls returned 200 (the second
and third were served by Headroom's response cache); the OpenAI call
returned 429 from the provider (account quota). The receiver logged
`POST /v1/logs -> 204` and stored one object,
`sessions/dt=2026-11-15/hh=15/<uuid>.json`. Its new fields, as stored
after the allowlist:

```json
"session": {"seq": 0, "duration_s": 5, "turns": 4, "ended": "shutdown", "final": true}
"tokens": {"original": 28, "input": 28, "output": 4, "saved": 0, "cache_read": 0, "cache_write": 14, "uncached": 28}
"clients": [{"client": "claude", "n": 1}, {"client": "claude_code", "n": 2}, {"client": "codex", "n": 1}]
"client_usage": [
  {"client": "claude", "n": 1, "original": 0, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 0},
  {"client": "claude_code", "n": 2, "original": 14, "saved": 0, "tool_saved": 0, "output": 4, "cache_read": 0, "cache_write": 0, "cache_write_inferred": 0, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14},
  {"client": "codex", "n": 1, "original": 14, "saved": 0, "tool_saved": 0, "output": 0, "cache_read": 0, "cache_write": 14, "cache_write_inferred": 14, "cache_write_5m": 0, "cache_write_1h": 0, "uncached": 14}]
"model_usage": [
  {"model": "claude-haiku-4-5-20251001", "n": 3, "original": 14, "output": 4, "cache_write": 0, "cache_write_inferred": 0, "uncached": 14, "...": "other columns 0"},
  {"model": "gpt-4o-mini", "n": 1, "original": 14, "output": 0, "cache_write": 14, "cache_write_inferred": 14, "uncached": 14, "...": "other columns 0"}]
"client_versions": [{"client": "claude_code", "version": "2.1.311", "n": 2}, {"client": "codex", "version": "0.45.0", "n": 1}]
"install": {"age_days": 0, "container": false, "ci": false}
```

Both tables' `n` sum to `session.turns` (4) and `original` to
`tokens.original` (28). The OpenAI row is the reviewed case:
`cache_write` 14 is entirely inferred, and `uncached + cache_read +
(cache_write - cache_write_inferred)` is 14, not 28. Nothing from the
User-Agent beyond harness and version reached the stored object.
- Not tested:
- The Worker on Cloudflare itself (Workers runtime, real R2).
`worker.js` ran under Node.
- A successful OpenAI turn, a Responses API turn with a reported write,
and Anthropic turns with cache reads or 5m/1h writes (covered by unit
tests only).
- `via_compress_api`, `via_mcp` and `unidentified` labels, a session
mixing harnesses on one model, and the 12-hour rollover (unit tests
only).
- Codex over WebSocket, which never passes the HTTP middleware and so
reports no version.
- gzip transport (`HEADROOM_BEACON_GZIP=1`) and a heartbeat (`ended:
"active"`); only the closing report was observed.
  - Linux, Windows, containers and CI.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; the beacon is not behind a rollout
channel.
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, for installs with the beacon on
(the default). The session report carries four more keys (about 5 KB
more per report for a typical session with gzip off), and a session at
least 12 hours old now ends at the first request on a later UTC day, so
always-on installs report about one session per day instead of one that
grows for weeks. No request-path behaviour changes.
- Kill switch / disable path: `HEADROOM_BEACON=off`,
`HEADROOM_TELEMETRY=off`, `DO_NOT_TRACK=1` or offline mode disable all
of it, as before. Server side, removing a key from `ALLOWED_KEYS` in
`deploy/beacon/worker.js` drops that field for every install without a
client release.
- Unsafe override required: no
- Qualification impact: none known.
- Rollback path: revert this PR. Rows already stored keep the extra
keys; readers use `union_by_name`.
- Deploy order: the Worker must be deployed before a client release that
sends these fields, or they are dropped on arrival. Production is still
the 2026-10-08 Worker build.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — none
made; `docs/content/docs/security-model.mdx` and `api-reference.mdx`
describe beacon fields and have not been updated
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — for the
suites listed under Testing
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- `client_versions` misses WebSocket sessions.
- `/v1/compress` traffic is now visible but still not attributed to the
integration that called it.
- Tool-call kinds, tool-definition size, compaction and subagent turns
are not included: they need the request body in each handler and a
privacy design for tool names.
- An earlier review suggested splitting rollover into its own PR since
it changes what a session row means; it is kept here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-17 09:50:49 +05:30
00436a8b11 feat(savings): book modelled output savings per request, as a token and dollar range (#4878)
## Description

Benchmark factors registered with `register_modelled_factors` reached
`/stats` only as a percentage. They never reached the per-request
rollup, so modelled output savings had no tokens or dollars in the
lifetime totals that the local dashboard and the control-plane heartbeat
read. This PR books each shaped request's output saving as a low–high
range of tokens and dollars, from the same tier the dashboard reports.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue) — an unshaped
treatment request was booked a baseline saving
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `SavingsRecorder.request_savings_range(labels, output_tokens) -> (low,
high)` replaces `estimate_request_savings` in the outcome funnel.
- It counts only requests that were actually shaped, marked by the
`output_shaper:verbosity:L<n>` label. The treatment stratum label is
attached before shaping is attempted, so it proves nothing.
  - A holdout (control) request books nothing.
  - A response with no reported output usage (output 0) books nothing.
- The tier comes from `SavingsLedger.select_tier(level)`, the function
`best_estimate` now delegates to. Lifetime dollars therefore always come
from the tier that the dashboard's percentage comes from:
- **measured:** the request's own stratum's control mean minus its
output, with low = high. It applies only if that stratum qualifies at
the cluster gate the measurement used. A stratum outside the measurement
books 0, because the measured headline excludes it too.
    - **estimated:** the request's stratum baseline, with low = high.
    - **modelled:** the factor band, `observed × r/(1−r)` at each end.
- `SavingsTracker`:
- Books the low end everywhere it booked before (history, rollups,
OTel).
- Adds `lifetime.output_tokens_saved_high` and
`output_savings_usd_high`, priced per model at output rates.
  - Both state builders write the new keys, so a fresh tracker has them.
- Dashboard: the Output Tokens Saved tile adds "≈ $low–$high saved on
output · low–high tokens, all time", read through a component getter.

OSS is unchanged. It ships an empty factor table, so the modelled tier
never appears.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): clean on the touched files
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ git log --oneline -1
1600ecae Merge branch 'main' into feat/output-savings-dashboard-range
$ pytest -q $(rg -l "output_savings|output_tokens_saved|best_estimate" tests) tests/test_request_outcome.py
421 passed in 23.29s
$ ruff check headroom
All checks passed!
```

Hosted CI:
`test_upstream_guard.py::test_openai_routes_refuse_a_private_override_instead_of_using_the_default`
and
`test_grok_shared_proxy_call_sites.py::test_the_base_url_override_is_resolved_once_per_request`
also fail on `main` at `d6df127d`.

New tests:
- the factor band;
- one control observation without a baseline keeps the modelled tier,
returning (200, 800);
- a measured holdout in stratum A credits A's requests from A's control
mean, while a request in stratum B (no control) books 0;
- a response without output usage books nothing;
- a baseline outranks the factors;
- an unshaped treatment request books nothing;
- no factors registered;
- the lifetime range survives a restart;
- a fresh tracker has the keys;
- end to end: outcome funnel → `PrometheusMetrics.record_request` →
tracker lifetime.

## Real Behavior Proof

- **Environment:**
  - macOS (darwin), Python 3.12.
- The real `headroom proxy`, run from a scratch merge of this PR with
#4877 (version `0.45.0-dev`).
  - An isolated `HEADROOM_WORKSPACE_DIR` per scenario.
- A local fake Anthropic upstream that returns `usage.output_tokens=800`
and records each request body.
- With an extension: a licensed extension that registers factors for its
levels (values not shown here).
- **Exact command / steps:**
- Start the proxy: `HEADROOM_OUTPUT_SHAPER=1 headroom proxy --port 18799
--anthropic-api-url http://127.0.0.1:18798`, with and without the
extension.
- Send 3 `POST /v1/messages` requests, each with a different user
prompt. Identical prompts are served from the proxy's response cache and
never reach the upstream.
- Read `GET /stats` → `tokens.output_reduction`, and the tracker's
persisted `proxy_savings.json` → `lifetime`.
- **Observed result** (3 requests × 800 output tokens each, all 3
reached the upstream):

| Scenario | `output_reduction` | lifetime saved tokens and USD |
|---|---|---|
| Core only (no extension) | `available: false` | 0–0 |
| Extension loaded | `modelled`, with a low–high band | a low–high
range, low < high; each end equals 3 × 800 × r/(1−r) for that end of the
registered band, and USD is tokens × the model's output price |
| Extension, `HEADROOM_VERBOSITY_LEVEL=3` pinned | `modelled`, the L3
band | a smaller low–high range, by the same arithmetic |
| Extension requested, no licence | `available: false` | 0–0 |

- **Not tested:**
- The dashboard tile rendered in a browser. Its Alpine expression was
evaluated in Node and renders "≈ $1.50–$4.20 saved on output · 1000–2600
tokens, all time" for a range, and a single figure for a point estimate.
  - A real upstream provider.
- Streaming responses. The funnel is shared, but no streaming request
was sent.
- The measured and estimated tiers on live holdout or baseline traffic.
These are covered by unit tests, not a run.

## Runtime Rollout Safety

- **Rollout-managed feature(s):** `proxy_output_shaper`. Nothing is
booked unless steering is on and factors are registered.
- **Minimum rollout channel:** unchanged.
- **Stable/default behavior changed:** none in OSS, because no factors
are registered. With an extension, two lifetime keys are added, and the
per-request saving follows the dashboard's tier.
- **Kill switch / disable path:** `HEADROOM_OUTPUT_SHAPER=0`, or don't
load the extension.
- **Unsafe override required:** none.
- **Qualification impact:** each shaped request runs one `select_tier`
inside the recorder lock, off the event loop in the existing
`asyncio.to_thread` block.
- **Rollback path:** revert the PR. Older cores drop the two `_high`
keys when they load the savings file, and nothing else reads them.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (no docs
describe these lifetime keys)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- Known gap: history checkpoints carry only the low end. If a reload
repairs lifetime from history, the band narrows for that window.
- Companion PR: #4877 adds the extension seams.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-17 09:49:13 +05:30
1f4c66f50a fix(beacon): roll up each hour in sixteen parallel queue slices (#4880)
## Description

The hourly rollup (`deploy/beacon/worker.js`) read every heartbeat
object of an hour inside one cron invocation.

- A Worker may have 6 requests in flight, so reads ran at about 12
objects/s regardless of `FANOUT = 100`.
- A cron invocation is stopped at 15 minutes, so one run could read
about 11k objects.
- Hours passed that size on 2026-08-19. A stopped run wrote nothing.

Result: 1,388 of 2,490 clock hours (23.8M of 33.3M heartbeat objects)
never got a rollup. Raw `sessions/` is complete; only `rollup/` had
gaps. The stats dashboard reads only rollups. The missing hours have
since been backfilled outside the Worker, so this PR is about not
falling behind again.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `scheduled()` no longer reads heartbeats. It lists hours with no
rollup and enqueues 16 messages per hour to a Cloudflare Queue, one per
first hex character of the object name (names are `crypto.randomUUID()`,
so slices are even and this works on objects already stored).
- `queue()` handles one slice per invocation: `rollupSlice()` reads it
(6 in flight) and writes `rollup-parts/<hour>/<slice>`; `finalizeHour()`
merges the 16 parts into `rollup/<hour>/data.ndjson` once all are
present, then deletes the parts.
- A failed slice is retried alone (60 s delay, 5 tries, then dead-letter
queue). An hour still unfinished after 3 hours is enqueued again.
`scheduled()` sweeps anything a finished hour left under
`rollup-parts/`.
- Rollup output is unchanged: same key, same winner per (install,
session).
- `wrangler.toml` gains the queue producer and consumer bindings.

## Testing

- [ ] Unit tests pass (`pytest`) — not applicable, no Python changed
- [ ] Linting passes (`ruff check .`) — not applicable
- [ ] Type checking passes (`mypy headroom`) — not applicable
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ cd deploy/beacon && node test-rollup.mjs
rollup dt=2026-08-06/hh=14: skipped 1 unparseable record(s)
gzip: 4079 B -> 104 B, bomb refused
rate limit: 429 over, fail-open on outage and when unbound
nested allowlist: client fields kept, unknown nested keys and free text dropped
ok
```

New cases: cross-slice dedupe, a failed slice leaving the hour
unpublished, enqueue / skip / re-enqueue in `scheduled()`, retry with
delay, reuse of an existing part, and the sweep.

## Real Behavior Proof

- Environment: macOS 27.2.1 (arm64), Node v26.7.0, DuckDB v1.5.5 with
`httpfs`, this branch at `811e6cf1`. Input is production data read from
the R2 bucket `headroom-telemetry` with a read-only key. The Worker
module ran under Node, not on Cloudflare; the R2 binding was an
in-memory map that pages `list()` at 1000 keys like R2.
- Exact command / steps (both scripts are in full below):
1. `./fetch_hour.sh 2026-08-10 03 h0810` and `./fetch_hour.sh 2026-10-14
12 h1014` — download every raw object of the hour, and the rollup object
stored in R2 for it.
2. `node compare.mjs deploy/beacon/worker.js h0810 dt=2026-08-10/hh=03`
(and likewise for `h1014`) — deliver 16 slice messages concurrently to
this branch's `queue()` handler, then compare what it published with the
stored rollup, row by row.
3. `node deploy/beacon/test-rollup.mjs h0810/raw` — the repo's own
real-corpus check.
- Observed result:

```text
$ ./fetch_hour.sh 2026-08-10 03 h0810
3666 raw objects written to h0810/raw
$ ./fetch_hour.sh 2026-10-14 12 h1014
24850 raw objects written to h1014/raw

$ node compare.mjs deploy/beacon/worker.js h0810 dt=2026-08-10/hh=03
{ "part": "dt=2026-08-10/hh=03", "raw_objects": 3666, "raw_gets": 3666, "acked": 16, "retried": 0,
  "sessions_published": 945, "sessions_in_reference": 945, "same_session_and_seq": 945,
  "identical_rows": 945, "parts_left_behind": 0, "wall_ms": 39, "peak_rss_mb": 103 }

$ node compare.mjs deploy/beacon/worker.js h1014 dt=2026-10-14/hh=12
{ "part": "dt=2026-10-14/hh=12", "raw_objects": 24850, "raw_gets": 24850, "acked": 16, "retried": 0,
  "sessions_published": 6903, "sessions_in_reference": 6903, "same_session_and_seq": 6903,
  "identical_rows": 6903, "parts_left_behind": 0, "wall_ms": 706, "peak_rss_mb": 432 }

$ node deploy/beacon/test-rollup.mjs h0810/raw
real corpus: 3666 objects -> 945 sessions in 1 object
ok
```

- **`dt=2026-08-10/hh=03`** (3,666 objects): the reference is the rollup
the current production Worker wrote. All 945 rows are identical after
key-order normalisation.
- **`dt=2026-10-14/hh=12`** (24,850 objects, among the busiest hours):
the old Worker never completed this hour. The reference is the rollup
built by the offline backfill, a separate implementation, and uploaded
to R2 on 2026-11-15. All 6,903 rows are identical. The published object
is 15.7 MB.
- In both runs every raw object was read exactly once, all 16 messages
were acknowledged, none retried, and no parts were left behind.
- Not tested:
- **Anything on Cloudflare.** Not deployed, not run under `wrangler
dev`. Queue delivery, consumer concurrency and ramp-up, the 6-connection
limit, the 15-minute limit, retries, the dead-letter queue and real R2
latency are unobserved.
- **Memory on the Workers runtime.** `peak_rss_mb` above is the Node
process including all raw objects held by the stub, so it says nothing
about the 128 MB isolate. `finalizeHour` holds the merged hour in memory
and builds the output as one string; for the 15.7 MB hour that is a few
multiples of 15.7 MB, by estimate only.
- **`scheduled()` against a real bucket**, including the listing of
`rollup/` and `rollup-parts/` at production size and enqueueing a
backlog.
- Throughput per slice. The measured ~12 objects/s per invocation comes
from the old Worker's write timestamps, not from this code.

<details><summary><code>fetch_hour.sh</code></summary>

```bash
#!/usr/bin/env bash
# Download one hour of raw beacon objects from R2 into <out>/<object name>, plus
# the rollup object stored for that hour.  ./fetch_hour.sh 2026-08-10 03 <out>
# Needs R2_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY (read-only is enough).
set -euo pipefail
DAY="$1"; HH="$2"; OUT="$3"; mkdir -p "$OUT/raw"
duckdb >/dev/null <<SQL
LOAD httpfs; SET threads=48; SET http_retries=8; SET http_timeout=120;
CREATE SECRET (TYPE r2, KEY_ID '${R2_ACCESS_KEY_ID}', SECRET '${R2_SECRET_ACCESS_KEY}', ACCOUNT_ID '${R2_ACCOUNT_ID}');
COPY (SELECT filename, json::VARCHAR line FROM read_ndjson_objects('r2://headroom-telemetry/sessions/dt=${DAY}/hh=${HH}/*.json', filename=true))
  TO '${OUT}/raw.jsonl' (FORMAT json);
COPY (SELECT json::VARCHAR line FROM read_ndjson_objects('r2://headroom-telemetry/rollup/dt=${DAY}/hh=${HH}/data.ndjson', maximum_object_size=40000000))
  TO '${OUT}/stored_rollup.ndjson' (FORMAT csv, HEADER false, QUOTE '', ESCAPE '', DELIMITER '\x1f');
SQL
python3 - "$OUT" <<'PY'
import json, os, sys, collections
out = sys.argv[1]; files = collections.defaultdict(list)
for l in open(f"{out}/raw.jsonl"):
    r = json.loads(l); files[os.path.basename(r["filename"])].append(r["line"])
for name, lines in files.items():
    open(f"{out}/raw/{name}", "w").write("\n".join(lines))
print(f"{len(files)} raw objects written to {out}/raw")
PY
```

</details>

<details><summary><code>compare.mjs</code></summary>

```js
// Run this branch's queue consumer over one real hour of raw objects and compare
// what it publishes with a reference rollup, row by row.
//   node compare.mjs <worker.js> <dir with raw/ and stored_rollup.ndjson> <dt=…/hh=…>
import { readFileSync, readdirSync } from 'node:fs';
import { resolve } from 'node:path';
import { pathToFileURL } from 'node:url';

const [workerPath, dir, part] = process.argv.slice(2);
const worker = (await import(pathToFileURL(resolve(workerPath)).href)).default;

// The slice of the R2 binding the Worker uses, over a Map. list() pages at 1000
// like R2 does.
const files = new Map();
for (const name of readdirSync(`${dir}/raw`)) {
  files.set(`sessions/${part}/${name}`, readFileSync(`${dir}/raw/${name}`, 'utf8'));
}
const raw = files.size;
let gets = 0;
const CORPUS = {
  list: async ({ prefix, cursor, limit = 1000 }) => {
    const keys = [...files.keys()].filter((k) => k.startsWith(prefix)).sort();
    const start = cursor ? keys.indexOf(cursor) : 0;
    const page = keys.slice(start, start + limit);
    const next = keys[start + limit];
    return { objects: page.map((key) => ({ key, uploaded: new Date(0) })), truncated: !!next, cursor: next };
  },
  get: async (key) => { gets++; return files.has(key) ? { text: async () => files.get(key) } : null; },
  head: async (key) => (files.has(key) ? { key } : null),
  put: async (key, body) => { files.set(key, body); },
  delete: async (keys) => { for (const key of keys) files.delete(key); },
};

const started = Date.now();
const outcome = { ack: 0, retry: 0 };
// Sixteen messages, one per slice, delivered concurrently as the queue would.
await Promise.all([...'0123456789abcdef'].map((slice) =>
  worker.queue({ messages: [{ body: { part, slice }, ack: () => outcome.ack++, retry: () => outcome.retry++ }] }, { CORPUS })));
const ms = Date.now() - started;

const canon = (v) => Array.isArray(v) ? v.map(canon)
  : v && typeof v === 'object' ? Object.fromEntries(Object.keys(v).sort().map((k) => [k, canon(v[k])])) : v;
const index = (text) => new Map(text.split('\n').filter(Boolean).map((line) => {
  const rec = JSON.parse(line);
  return [`${rec.resource?.['headroom.install_id']} ${rec.session?.id}`, { seq: rec.session?.seq, row: JSON.stringify(canon(rec)) }];
}));
const mine = index(files.get(`rollup/${part}/data.ndjson`) ?? '');
const reference = index(readFileSync(`${dir}/stored_rollup.ndjson`, 'utf8'));
let sameSeq = 0, sameRow = 0;
for (const [key, value] of mine) {
  const other = reference.get(key);
  if (other?.seq === value.seq) sameSeq++;
  if (other?.row === value.row) sameRow++;
}
console.log(JSON.stringify({
  part, raw_objects: raw, raw_gets: gets - 16, acked: outcome.ack, retried: outcome.retry,
  sessions_published: mine.size, sessions_in_reference: reference.size,
  same_session_and_seq: sameSeq, identical_rows: sameRow,
  parts_left_behind: [...files.keys()].filter((k) => k.startsWith('rollup-parts/')).length,
  wall_ms: ms, peak_rss_mb: Math.round(process.resourceUsage().maxRSS / 1024),
}, null, 1));
```

</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is the telemetry receiver's
cron, not the proxy; no client or request-path behaviour changes.
- Minimum rollout channel: n/a
- Stable/default behavior changed: only on the Worker. Rollups are built
by queue consumers instead of inside the cron run; the rollup objects
readers consume are unchanged. A new `rollup-parts/` prefix appears in
the bucket while an hour is in flight.
- Kill switch / disable path: remove the cron trigger or the queue
consumer in `wrangler.toml` and redeploy. Raw `sessions/` is never
modified, so any rollup can be rebuilt.
- Unsafe override required: no
- Qualification impact: none known.
- Rollback path: redeploy the previous Worker build. Hours it cannot
finish can be filled by the offline backfill, as was done for the
existing gap.
- Deploy steps (not done by this PR):

```
npx wrangler queues create beacon-rollup
npx wrangler queues create beacon-rollup-dlq
npx wrangler deploy
```

Queues need the Workers Paid plan, which this Worker already requires.
Deploying also ships the two merged-but-undeployed Worker changes (#4008
rate limit, #4691 nested allowlist); production is still the 2026-10-08
build.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — none
needed that I know of
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes — `node
deploy/beacon/test-rollup.mjs`
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Nothing alerts on the dead-letter queue or on hours that stay
unrolled.
- The first run on Cloudflare should be watched: check that `rollup/`
gains the previous hour within a few minutes of :05 and that
`rollup-parts/` empties.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-17 09:48:51 +05:30
d9ae7ad721 fix(proxy): stop a "*" origin setting from waiving the WebSocket Origin check (#4846)
## Description

The WebSocket routes (`/v1/responses`, `/v1/live`) check the browser
`Origin` against `HEADROOM_WS_ORIGINS`, falling back to
`HEADROOM_CORS_ORIGINS`. A `*` entry in either made every Origin
acceptable. `HEADROOM_CORS_ORIGINS=*` is a documented value for CORS
(which pages may *read* a response), so loosening CORS for a remote
dashboard also let any page open in the user's browser open these
sockets, and the proxy then adds its own credentials upstream.

`*` now matches nothing in the Origin check. A list that holds only `*`
keeps the loopback default; explicit origins in either variable keep
working. CORS itself is unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/handlers/openai.py`: `_allowed_ws_origins_from_env`
drops `*` entries (empty result means the loopback default);
`_is_allowed_websocket_origin` no longer short-circuits to `True` on
`*`. `providers/codex/live.py` uses the same helper, so `/v1/live` is
covered too.
- `tests/test_openai_codex_routing.py`: policy test for `*` via either
variable and mixed with an explicit origin, plus a handler test that a
foreign Origin is closed with 1008 under `HEADROOM_CORS_ORIGINS=*`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# new tests on origin/main (before the fix)
$ pytest tests/test_openai_codex_routing.py -k wildcard
FAILED tests/test_openai_codex_routing.py::test_websocket_origin_policy_ignores_wildcard[None-*]
FAILED tests/test_openai_codex_routing.py::test_websocket_origin_policy_ignores_wildcard[*-None]
FAILED tests/test_openai_codex_routing.py::test_websocket_origin_policy_ignores_wildcard[None-*, https://dash.example.com]
FAILED tests/test_openai_codex_routing.py::test_handle_openai_responses_ws_closes_foreign_origin_under_wildcard_cors
======================= 4 failed, 43 deselected in 1.33s =======================

# with the fix
$ pytest tests/test_openai_codex_routing.py tests/test_codex_live.py
======================= 57 passed, 7 warnings in 16.31s ========================

$ ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_routing.py
All checks passed!
$ mypy headroom/proxy/handlers/openai.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, `headroom proxy --host 127.0.0.1
--no-optimize` started with `HEADROOM_CORS_ORIGINS='*'`, one run on
origin/main and one on this branch.
- Exact command / steps: open `ws://127.0.0.1:<port>/v1/responses` with
the `websockets` client, once with `Origin: https://page.example` and
once with `Origin: http://127.0.0.1:<port>`.
- Observed result:
  ```text
  == origin/main (HEADROOM_CORS_ORIGINS=*)
  Origin=https://page.example -> handshake accepted, socket open
  Origin=http://127.0.0.1:18941 -> handshake accepted, socket open
  == this branch (HEADROOM_CORS_ORIGINS=*)
  Origin=https://page.example -> handshake refused HTTP 403
  Origin=http://127.0.0.1:18942 -> handshake accepted, socket open
  ```
- Not tested: `/v1/live` end to end (same helper, covered by the unit
tests' shared code path).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for operators who set `*` in
`HEADROOM_WS_ORIGINS` or `HEADROOM_CORS_ORIGINS`. Native clients (Codex,
SDKs) send no Origin and are unaffected; loopback pages still connect.
- Kill switch / disable path: list the browser origins that need the
WebSocket routes explicitly in `HEADROOM_WS_ORIGINS`.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

`HEADROOM_WS_ORIGINS` is not documented anywhere, so there was no doc to
update. A companion PR applies the same Origin rule to the HTTP routes;
once both land, `*` is ignored there too because they share
`_is_allowed_websocket_origin`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4846"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-17 09:48:30 +05:30
bae62bc87b fix(compression): bound the JSON-block and truncation-banner scans (#4845)
## Description

Two scans over tool output ran in quadratic time:

1. **Mixed-content JSON detection** (`_has_valid_json_block_with_text`,
default-on through `ContentRouter` → `is_mixed_content`) and the section
splitter try every line that starts with `{` or `[` as a JSON block. A
candidate that never balances walks to the end of the content, so N such
lines cost N²/2 line steps (20,000 lines of `{` took about a minute).
The existing per-line memo removes the character work but not the line
walk.
2. **ast-grep truncation-banner detection** (`_detect_truncation`): the
banner regex has unbounded runs on both sides of the `showing lines A-B
of T total` phrase, so an unclosed banner was rescanned once per phrase.

Fixes:

1. `_JsonBlockScanner` answers every candidate of one pass with exactly
the result of walking it. Candidates are walked directly until they have
walked as many lines as the content has. After that, all candidates are
answered together:
- Every walk is a chain of (line, string-state) nodes with one successor
each, so all walks form one forest, and each node is scanned once.
- With `D(v)` the bracket and brace sums from node `v` to the end of its
chain, a walk from `u` balances at `w` exactly when `D(next(w)) >= D(u)`
in both sums. So each candidate ends just before its nearest ancestor
that satisfies this.
- One depth-first pass with an undoable tree of (brace sum, depth)
stacks, keyed by bracket sum, finds that ancestor in O(log² n). There is
no budget.
2. The banner is found in two linear steps: locate `[truncated: partial
view`, take the span to the next bracket (it must be `]`), then match
the phrase inside that span. Same matches as the old regex (checked
against it on 3,000 random layouts).

Fix adapted from a proposal by HetCreep.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/transforms/mixed_content.py`: `_JsonBlockScanner` (direct
walks, then a forest solve over all candidates), used by
`_has_valid_json_block_with_text` and `split_into_sections`.
`_extract_json_block` is unchanged from main.
- `headroom/proxy/interceptors/astgrep.py`: `_TRUNCATION_SIGNATURES` now
holds only the phrase pattern; `_iter_banners` finds the `[truncated:
partial view … ]` span and searches the phrase inside it.
- `tests/test_tool_output_scan_bounds.py`: timing tests for both shapes,
an oracle test against the previous banner regex, and checks that a real
banner and a 5,000-line JSON block inside text are still found. A JSON
object that follows 150 or 20,000 unclosed `{` lines, or a prefix whose
bracket and brace counts each reach zero but never on the same line
(once, and repeated 130 times), still gets its own JSON section, as does
one after 100 or 10,000 pairs of lines ending inside an escaped string.
The solve is checked against `_extract_json_block` on 3,000 random
inputs.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests on `origin/main` code:

```text
FAILED tests/test_tool_output_scan_bounds.py::test_truncation_scan_is_linear_on_unclosed_banner
FAILED tests/test_tool_output_scan_bounds.py::test_json_block_scans_are_bounded_on_unclosed_braces
    assert time.perf_counter() - start < 10.0
E   assert (63611.161382791 - 63549.279589625) < 10.0
==================== 2 failed, 3 passed in 75.96s (0:01:15) ====================
```

With the fix:

```text
$ pytest -q tests/test_tool_output_scan_bounds.py --durations=3   # (first version; now 10 tests, 10 passed in 0.88s)
0.34s call     tests/test_tool_output_scan_bounds.py::test_json_block_scans_are_bounded_on_unclosed_braces
0.02s call     tests/test_tool_output_scan_bounds.py::test_truncation_banner_matches_previous_regex
============================== 5 passed in 1.09s ===============================

$ pytest -q tests/test_tool_output_scan_bounds.py tests/test_mixed_content_scan_cache.py tests/test_mixed_content_sections.py tests/test_content_router_mixed_fence_placeholder.py
============================= 300 passed in 5.90s ==============================

$ pytest -q tests/test_tool_result_interceptors.py tests/test_proxy_compression_executor.py tests/test_subprocess_encoding.py tests/test_runtime_env.py tests/test_content_router*.py
============================= 247 passed in 32.58s =============================

$ ruff check / ruff format --check (0.17.3) on changed files: All checks passed
$ mypy headroom/proxy/interceptors/astgrep.py headroom/transforms/mixed_content.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS (Apple silicon), Python 3.12 venv, this branch vs
`origin/main`.
- Exact command / steps: called `_has_valid_json_block_with_text` +
`split_into_sections` on 20,000 lines of `{`, and `_detect_truncation`
on `"[truncated: partial view " + "showing lines 1-10 of 50 total " *
12000`.
- Observed result: JSON scans about 62 s on main, 0.34 s here; banner
scan about 13 s on main, milliseconds here. The existing scan-cache
oracle tests (which compare against the original extractor) still pass.
- Not tested: end-to-end through the proxy; both functions are pure and
covered directly.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no change in results (exact); only
time and memory bounds change
- Kill switch / disable path: none needed
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — n/a
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Timings with the fix: 20,000 unclosed `{` lines take 0.13 s, the
crossed-count prefix repeated 130 times takes 0.23 s, 20,000 alternating
`{` / `}[` lines take 0.29 s, and 10,000 escaped-string line pairs take
0.14 s. With direct walking, the first takes about 62 s. Separately, on
main a balanced block nested about 10,000 deep makes `json.loads` raise
`RecursionError` in the gate; that is unchanged here.

Note: `tests/test_transforms_content_router.py` has 3 failures
(`test_json_in_html_preserves_records_and_markup`,
`test_embedded_json_mixed_tool_output_preserves_every_record[minify-*]`)
that also fail on unmodified `origin/main` in my environment, so they
are not caused by this change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-17 09:48:08 +05:30
f5c4217648 fix(proxy): enforce HEADROOM_PROXY_TOKEN in the Rust front and listen on loopback by default (#4840)
## Description

The Rust front (`crates/headroom-proxy`) relays every request to the
Python proxy from `127.0.0.1`. The Python proxy treats a loopback caller
as local: it skips the `HEADROOM_PROXY_TOKEN` check and serves `/admin`,
`/debug` and `/settings` to it. The front had no inbound check of its
own, listened on `0.0.0.0:8787` by default, and dropped
`x-headroom-proxy-token` (as an `x-headroom-*` header), so a token set
on the Python side neither protected the stack nor could be presented
through the front.

This PR makes the Rust front apply the same token model as the Python
proxy (`read_proxy_token` / `scrub_proxy_token_headers` in
`headroom/proxy/server.py`, `headroom/proxy/bind_policy.py`) before
anything is forwarded, and makes its default listen address loopback.

Fix adapted from a proposal by HetCreep.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- New `crates/headroom-proxy/src/inbound_auth.rs`, installed as a
router-wide layer (all routes, the catch-all and WebSocket upgrades)
when `HEADROOM_PROXY_TOKEN` / `--proxy-token` is set:
- a non-loopback peer (or one with no peer address) must send the token
as `X-Headroom-Proxy-Token` or `Authorization: Bearer`; the explicit
header wins, as in Python. Constant-time compare. Otherwise `401
{"error":"unauthorized"}` and no upstream contact; a
`proxy_auth_rejected` warning is logged (no credential values).
- loopback peers are exempt (same as Python); `GET /healthz`,
`/healthz/upstream`, `/health`, `/livez`, `/readyz` stay open;
`GET`/`HEAD /api/hello` (also under `/p/<project>`) is answered locally
with an empty 200, as Python does, instead of being relayed.
- the token is removed before forwarding: `x-headroom-proxy-token`
always, `Authorization` only when it carries the token (a provider
credential is left alone).
  - the token is redacted from `Debug` output of `Config`.
- `main.rs`: startup bind policy, same rule and variable as Python: a
non-loopback `--listen` with no token exits with code 2 and a message
naming both remedies, unless `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`
(then a warning is logged).
- `config.rs`: `--listen` / `HEADROOM_PROXY_LISTEN` default changes from
`0.0.0.0:8787` to `127.0.0.1:8787`; new `--proxy-token` (env
`HEADROOM_PROXY_TOKEN`, value hidden from `--help`).
- `RUST_DEV.md`: runbook exports a token for the `0.0.0.0` cutover, flag
table updated, new "Inbound authentication" section.

**Behaviour change for operators:** a Rust front started with `--listen
0.0.0.0:...` (or relying on the old default to be reachable from other
hosts) now needs `HEADROOM_PROXY_TOKEN` set, and clients on other hosts
must send it. Setups that front the port with something else that
restricts access can set `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`.
Loopback clients (Claude Code, Codex, SDKs on the same host) are
unaffected. The Docker image's entrypoint is the Python proxy, so it is
unaffected.

## Testing

- [x] `cargo test -p headroom-proxy` (46 test binaries, 608 tests pass)
- [x] `cargo clippy -p headroom-proxy --all-targets -- -D warnings` and
`cargo fmt --all -- --check` clean
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] `pytest` / `ruff` / `mypy`: N/A, no Python changes

New tests:
- `tests/integration_inbound_auth.rs` (11 tests, `oneshot` with an
injected `ConnectInfo` so a non-loopback peer can be simulated,
`wiremock` upstream): missing/wrong token is 401 and the upstream sees
nothing; correct token via header or Bearer is forwarded without the
token; provider `Authorization` kept when the header carries the token;
loopback and IPv4-mapped loopback peers pass; missing peer address fails
closed; health probes open but `POST /readyz` is 401; `/api/hello`
answered locally; no token configured keeps passthrough.
- `tests/integration_bind_policy.rs` (3 tests, real binary): `0.0.0.0`
with no token refuses to start; loopback starts; default `--listen` is
loopback.
- unit tests in `inbound_auth.rs` for token parsing, constant-time
compare, bind policy and exemptions.

### Test Output

Before (the bind-policy test file copied onto a clean `origin/main`
worktree; `integration_inbound_auth.rs` does not compile there since
main has no token setting at all):

```text
test default_listen_address_is_loopback ... FAILED
test loopback_listen_without_token_still_starts ... ok
test non_loopback_listen_without_token_refuses_to_start ... FAILED
test result: FAILED. 1 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20.01s
```

After:

```text
$ cargo test -p headroom-proxy --test integration_inbound_auth --test integration_bind_policy
test default_listen_address_is_loopback ... ok
test non_loopback_listen_without_token_refuses_to_start ... ok
test loopback_listen_without_token_still_starts ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.01s
test missing_peer_address_fails_closed ... ok
test remote_peer_with_bearer_token_is_forwarded_without_it ... ok
test remote_peer_without_token_gets_401_and_upstream_is_not_contacted ... ok
test token_header_wins_over_a_provider_bearer ... ok
test no_token_configured_keeps_todays_passthrough ... ok
test loopback_peer_without_token_is_still_forwarded ... ok
test ipv4_mapped_loopback_peer_counts_as_loopback ... ok
test remote_peer_with_token_header_is_forwarded_without_it ... ok
test hello_probe_is_answered_locally_without_token ... ok
test remote_peer_with_wrong_token_gets_401 ... ok
test health_probes_stay_reachable_without_token ... ok
test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.74s
```

## Real Behavior Proof

- Environment: macOS (Darwin 27.2), debug builds of `headroom-proxy`
from `origin/main` (c21b0813) and from this branch. Upstream is a small
Python HTTP recorder on `127.0.0.1:18788` that prints each request it
receives. Requests are sent with curl to the machine's LAN address
(non-loopback peer) and to `127.0.0.1`.
- Exact command / steps: `HEADROOM_PROXY_TOKEN=s3cret
HEADROOM_PROXY_STATS=false headroom-proxy --listen 0.0.0.0:18787
--upstream http://127.0.0.1:18788`, then `curl
http://<lan-ip>:18787/admin/upstream` with no token, a wrong token and
the right token in `x-headroom-proxy-token`, `GET /readyz` with no token
from the LAN address, and `GET /v1/models` from `127.0.0.1` with no
token.
- Observed result:

```text
== origin/main
GET /admin/upstream (no token)                     -> 200
GET /admin/upstream -H x-headroom-proxy-token:wrong -> 200
GET /admin/upstream -H x-headroom-proxy-token:s3cret -> 200
GET /readyz (no token)                             -> 200
GET /v1/models from 127.0.0.1 (no token)           -> 200
UPSTREAM SAW {"path": "/admin/upstream", "peer": "127.0.0.1", "token_hdr": null, "authorization": null}   (x3)
UPSTREAM SAW {"path": "/readyz", ...}
UPSTREAM SAW {"path": "/v1/models", ...}

== this branch
GET /admin/upstream (no token)                     -> 401
GET /admin/upstream -H x-headroom-proxy-token:wrong -> 401
GET /admin/upstream -H x-headroom-proxy-token:s3cret -> 200
GET /readyz (no token)                             -> 200
GET /v1/models from 127.0.0.1 (no token)           -> 200
UPSTREAM SAW {"path": "/admin/upstream", "peer": "127.0.0.1", "token_hdr": null, "authorization": null}   (only the authenticated one)
UPSTREAM SAW {"path": "/readyz", ...}
UPSTREAM SAW {"path": "/v1/models", ...}

$ headroom-proxy --listen 0.0.0.0:18787 --upstream http://127.0.0.1:18788   # no token
headroom-proxy: Refusing to listen on 0.0.0.0:18787: with no HEADROOM_PROXY_TOKEN this proxy would relay every network peer to the upstream Headroom proxy from 127.0.0.1, where it is treated as a trusted local caller. Either set HEADROOM_PROXY_TOKEN ... or ... set HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1 to acknowledge the open bind.
(exit 2)
```

- Not tested: a full chain with the real Python proxy behind the Rust
front (the recorder stands in for it); WebSocket upgrades through the
gate are covered only by the layer wrapping the catch-all, not by a
dedicated test.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: yes, default listen address is now
loopback, and a non-loopback listen needs a token or the acknowledgement
variable
- Kill switch / disable path: `HEADROOM_ALLOW_UNAUTHENTICATED_BIND=1`
restores starting on a non-loopback address without a token; `--listen
0.0.0.0:8787` restores the old address
- Unsafe override required: no
- Qualification impact: none (qualification scripts already pass
`--listen 127.0.0.1:<port>`)
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

A companion PR makes the Rust front's native Bedrock and Vertex routes
opt-in; it relies on this token gate for callers on those routes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-17 09:47:48 +05:30
JD Davis b0a74a5540 Merge branch 'main' into feat/https-compose-caddy 2026-11-16 20:41:34 -06:00
cfdeb1c466 fix(learn): render non-string Bash commands without raising (#4826)
## Description

Fixes a crash in `headroom learn` when a scanned session stores a
non-string Bash `command`.

`ToolCall.input_summary` sliced `input_data["command"]` directly,
assuming a string. A real opencode v2 database contained eight Bash
calls with `{"command": true}`, which raised:

```
TypeError: object of type 'bool' has no len()
```

Both the digest formatter (`analyzer._format_tool_call`) and the loop
sampler (`loops.detect_loops`) read `input_summary`, so a single
malformed call aborted the entire `headroom learn --all --apply` run.

No linked issue: this was found while validating the opencode v2 scanner
change in #4199 (now merged). The crash is in shared `ToolCall` code, so
the fix is submitted separately.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `ToolCall.input_summary` now coerces a non-string `command` with
`str()` instead of slicing the raw value; `None` renders as an empty
string.
- `input_summary` treats a non-mapping `input_data` as `{}` before
calling `.get()`, matching the defensive shape already used in
`loops._identity_input`.
- Added `TestInputSummaryToleratesMalformedInput` with six cases:
bool/int/`None`/list commands, a non-mapping input, and an end-to-end
`detect_loops` + `_build_digest` case.
- No behavior change for valid string commands; the other summary
branches already use `str()`/f-strings.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New tests added for new functionality
- [x] Manual testing performed
- [ ] Type checking passes (`mypy headroom`) — not run; the touched
module was not type-checked as part of this change.

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_learn/test_loop_weighting.py -k TestInputSummaryToleratesMalformedInput -q
6 passed, 89 deselected, 1 warning in 0.78s

$ .venv/bin/python -m pytest tests/test_learn/ tests/test_cli_learn.py -q
455 passed, 7 skipped, 2 warnings in 32.88s

$ uvx ruff@0.16.8 check headroom/learn/models.py tests/test_learn/test_loop_weighting.py
All checks passed!

$ uvx ruff@0.16.8 format --check headroom/learn/models.py tests/test_learn/test_loop_weighting.py
2 files already formatted
```

All six new tests fail against the unpatched `models.py`.

## Real Behavior Proof

- Environment: Linux x86_64; Python 3.14.8; `uv sync --frozen`; PR head
`22e6a6f8fb05f79dcc82ea57a24bfbd6615305b8`. No provider credentials,
network, or private data required.
- Exact command / steps: save the script below as
`proof_malformed_opencode.py` at the repo root, then run
`.venv/bin/python proof_malformed_opencode.py`. It creates a temporary
opencode v2 SQLite database (`project` / `session_v2` /
`session_message`) containing three Bash calls whose input is
`{"command": true}`, then runs `OpenCodePlugin.scan_project` →
`detect_loops` → `_build_digest`.
- Observed result: with this PR, `sessions: 1 calls: 3`, `summaries:
['True', 'True', 'True']`, `loops: [('Bash', 3, 'True')]`, `digest
chars: 684`, `digest contains malformed call: True`. With
`headroom/learn/models.py` reverted to `origin/main`, the same script
exits with `TypeError: object of type 'bool' has no len()` at
`models.py:67` (`return cmd[:100] + "..." if len(cmd) > 100 else cmd`).
The original private database scan (28 sessions / 7,882 calls, 8
malformed commands, 29 loops, 319,885-char digest) is additional
evidence but is not reproducible outside that machine.
- Not tested: a real LLM analysis call (requires provider credentials),
Windows path handling, other agents' databases, and `mypy`. The scanner
and analyzer logic are unchanged; only the shared display property is
guarded.

<details>
<summary>proof_malformed_opencode.py</summary>

```python
"""Reproduce the malformed-command crash with a synthetic opencode v2 DB."""

import json
import sqlite3
import tempfile
from pathlib import Path

from headroom.learn.analyzer import _build_digest
from headroom.learn.loops import detect_loops
from headroom.learn.plugins.opencode import OpenCodePlugin

with tempfile.TemporaryDirectory() as tmp:
    tmp_path = Path(tmp)
    project_path = tmp_path / "project"
    project_path.mkdir()
    db_path = tmp_path / "opencode.db"

    conn = sqlite3.connect(db_path)
    conn.executescript(
        """
        CREATE TABLE project (id TEXT PRIMARY KEY, worktree TEXT NOT NULL, name TEXT);
        CREATE TABLE session_v2 (
            id TEXT PRIMARY KEY, project_id TEXT NOT NULL, time_created INTEGER NOT NULL
        );
        CREATE TABLE session_message (
            id TEXT PRIMARY KEY, session_id TEXT NOT NULL, type TEXT NOT NULL,
            seq INTEGER NOT NULL, time_created INTEGER NOT NULL,
            time_updated INTEGER NOT NULL, data TEXT NOT NULL
        );
        """
    )
    conn.execute("INSERT INTO project VALUES (?, ?, ?)", ("p", str(project_path), "proof"))
    conn.execute("INSERT INTO session_v2 VALUES (?, ?, ?)", ("s", "p", 1))
    content = [
        {
            "type": "tool",
            "id": f"call-{i}",
            "name": "bash",
            "state": {
                "status": "completed",
                "input": {"command": True},
                "content": [{"type": "text", "text": "exit code 0"}],
            },
        }
        for i in range(3)
    ]
    conn.execute(
        "INSERT INTO session_message VALUES (?, ?, ?, ?, ?, ?, ?)",
        (
            "m",
            "s",
            "assistant",
            1,
            1,
            1,
            json.dumps({"content": content}, separators=(",", ":")),
        ),
    )
    conn.commit()
    conn.close()

    plugin = OpenCodePlugin(db_path=db_path)
    project = plugin.discover_projects()[0]
    sessions = plugin.scan_project(project)
    print("sessions:", len(sessions), "calls:", sum(len(s.tool_calls) for s in sessions))
    print("summaries:", [c.input_summary for c in sessions[0].tool_calls])

    loops = detect_loops(sessions)
    print("loops:", [(lp.tool, lp.count, lp.sample_input) for lp in loops])

    digest = _build_digest(project, sessions, loops=loops)
    print("digest chars:", len(digest))
    print("digest contains malformed call:", "Bash: True" in digest)
```

</details>

## Runtime Rollout Safety

- Rollout-managed feature(s): `headroom learn` local session scanning
and digest formatting only; no proxy, provider, or runtime request path.
- Minimum rollout channel: N/A — library bug fix, not gated by a rollout
channel.
- Stable/default behavior changed: No. Valid string commands render
exactly as before; only malformed inputs that previously raised now
render.
- Kill switch / disable path: N/A — revert the commit to restore the
prior behavior.
- Unsafe override required: No.
- Qualification impact: None; no network, model, or rollout behavior is
touched.
- Rollback path: revert this commit/PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation — N/A, no
public API or docs change
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — no UI change.

## Additional Notes

The malformed value is real session data, not a synthetic edge case:
session `ses_e6bcaf904fd4blosS7bPwIvQ1f` in a local opencode v2 DB
stores `{"command": true}` for 8 Bash calls. The fix only affects
display/summary rendering, so loop identity, failure classification, and
recommendations are unchanged.

Co-authored-by: JavierLeon9966 <javier@lumineproxy.org>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 20:22:17 -06:00
93331f2874 fix(proxy): retain final backend streaming usage (#4814)
## Description

Translated OpenAI streams (`/v1/chat/completions` through a backend such
as `anyllm` or LiteLLM) recorded the **first nonzero** usage snapshot
instead of the last one.

`_absorb` in `_stream_openai_via_backend` wrote a counter only when the
accumulator was still falsy:

```python
if key in usage and not stream_state.get(key):
    stream_state[key] = usage[key]
```

A provider that reports climbing cumulative snapshots on successive
content chunks therefore had every later frame thrown away. Replaying
the two frames from the issue through the real proxy path, `/stats`
recorded 10 input / 1 output tokens and `/metrics` recorded 2 cache
reads — while the client received both frames intact. The bug is in
accounting after delivery, not a dropped upstream event.

Closes #4263

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `_parse_sse_usage_from_buffer` (openai branch): absent counters no
longer default to `0`. A frame that omits a field was previously
indistinguishable from one reporting an explicit zero, which is a
prerequisite for "absent keeps the last value".
- `_absorb`: assigns unconditionally, so the latest cumulative snapshot
wins. An absent field keeps its earlier value; an explicit zero still
replaces it.
- Removed `_parse_completion_tokens_from_sse_chunk`. It existed to catch
completion-only frames, but the buffered SSE parser already handles
those, and its `not stream_state["output_tokens"]` guard latched the
output count a second time.
- Replaced that helper's unit tests with eight regressions driving the
real `_stream_openai_via_backend` path through stream finalization, in
Chat and Responses shapes.
- `docs/content/docs/proxy.mdx`: one paragraph on the `POST
/v1/chat/completions` page documenting the accounting rule, in the
published docs site.

Net: -100/+113 lines, of which most deletions are the removed helper and
its implementation-only tests.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_streaming_usage_parser.py -q
........                                                                 [100%]
8 passed in 1.01s

$ ruff check headroom/proxy/handlers/streaming.py tests/test_streaming_usage_parser.py
All checks passed!

$ ruff format --check headroom/proxy/handlers/streaming.py tests/test_streaming_usage_parser.py
2 files already formatted

$ mypy headroom/proxy/handlers/streaming.py
Success: no issues found in 1 source file
```

Regression set — the new tests against the unfixed handler (`git stash`
of `streaming.py` only):

```text
$ python -m pytest tests/test_streaming_usage_parser.py -q
FAILED ...[latest-snapshot-chat]      assert (10, 1, 2, 3) == (13, 5, 9, 1)
FAILED ...[latest-snapshot-responses]
FAILED ...[partial-chat]              assert (10, 1, 2, 3) == (10, 5, 2, 3)
FAILED ...[partial-responses]
FAILED ...[explicit-zero-chat]        assert (10, 1, 2, 3) == (0, 0, 0, 0)
FAILED ...[explicit-zero-responses]
6 failed, 2 passed in 1.07s
```

Six fail before the fix, all eight pass after — matching the issue
exactly.

Wider streaming and parser suites:

```text
$ python -m pytest tests/test_streaming_usage_parser.py tests/test_streaming_sse_malformed_events.py \
    tests/test_provider_billed_input.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_handler_helpers.py \
    tests/test_proxy_streaming_request_logger.py tests/test_route_advice.py tests/test_h2_stream_reset_retry.py \
    tests/test_proxy_streaming_resilience.py tests/test_proxy_streaming_ratelimit_headers.py -q
3 failed, 607 passed in 10.21s
```

The three failures are
`tests/test_proxy_cache_ttl_metrics.py::test_stats_endpoint_reports_*`,
which raise `SystemExit: 78` from `_check_rust_core()` in `server.py`.
They are environmental (unbuilt Rust core in this checkout) and fail
identically on unmodified `upstream/main` with this branch stashed —
unrelated to this change.

## Real Behavior Proof

- Environment: Windows, Python 3.13.11, this branch, `PYTHONPATH` set to
the checkout.
- Exact command / steps: replayed the issue's two usage snapshots as a
Chat stream through `_stream_openai_via_backend` with a stub backend,
then read the resulting `RequestOutcome`.
- Observed result: the corrected counts from the issue; previously 10 /
1 / 2.

```text
provider_input_tokens=13 (expect 13)
output_tokens        =5 (expect 5)
cache_read_tokens    =9 (expect 9)
```

- Not tested: a live `anyllm`/LiteLLM upstream over real HTTP (no
credentials in this environment). The replay drives the identical
handler and parser path the proxy uses; the frames are byte-for-byte the
issue's.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: token accounting for translated
OpenAI streams now records the final snapshot. This is the fix; counts
move up from truncated to complete.
- Kill switch / disable path: none beyond reverting this commit.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: `git revert`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Authorship is credited to @oxura, who diagnosed this in #4263 and wrote
the original patch on their fork; they were holding it back on the
ten-open-PR limit. This PR carries their change rebased onto current
`upstream/main` (their branch was cut from `fbf7bc32`, and
`streaming.py` has since drifted ~200 lines), with the commit message
and tests adapted to the current tree. Their `Authored-by` trailer is on
the commit.

One hunk from their patch was redirected: they documented the rule in
`wiki/proxy.md`, which `AGENTS.md:38` says is unpublished and must not
receive new content, so the paragraph now lives on the `POST
/v1/chat/completions` page in `docs/content/docs/proxy.mdx`. Thanks to
Devin Review for flagging it.

Co-authored-by: oxura <grozafroza44@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 19:51:50 -06:00
199912e020 test(cli): assert the --learn/--no-learn warning text (#4800)
## Description

`TestLearnNoLearnConflict::test_both_flags_warns_and_exits_zero` ends
with `assert "both --learn and --no-learn" in result.output or
(result.output is not None), ...`. The right side of the `or` is true
for any run (`result.output` is a string), so the assertion cannot fail
and the test passes without the warning. This PR drops the `or` so the
test checks the warning text it is named for.

Part of #4783 (found in a second pass; listed there as a test that
cannot fail). It is a one-line test change, no product code.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
- [x] Test-only change

## Changes Made

- `tests/test_cli_proxy_improvements.py`: the warning assertion is now
`assert "both --learn and --no-learn" in result.output, result.output`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`ruff check` and `ruff format --check` (0.17.3) are clean on the file.

### Test Output

```text
tests/test_cli_proxy_improvements.py -k TestLearnNoLearnConflict
  this branch:                                        4 passed

mutation: headroom/cli/proxy.py `if learn and no_learn:` -> `if False and learn and no_learn:`
  test_both_flags_warns_and_exits_zero, this branch:  FAILED
  test_both_flags_warns_and_exits_zero, main:         passed
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, click 8.5.1, pytest 9.2.0,
`main` at `3c66c91e`.
- Exact command / steps: ran the four tests of
`TestLearnNoLearnConflict`; then disabled the warning branch in
`headroom/cli/proxy.py` locally and ran
`test_both_flags_warns_and_exits_zero` with this change and with
`main`'s test (the product file was restored afterwards).
- Observed result: as in Test Output. `main`'s assertion passes with the
warning removed; this change fails.
- Not tested: other click versions (the assertion relies on
`result.output` containing stderr, which is how click 8.5.1's
`CliRunner` behaves), the whole suite, CI.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no (test only).
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes (the four
tests of the class; the whole suite was not run)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

AI assistance: this change and the runs above were done with Claude
(Sonnet 5.5), operated by me (HetCreep).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4800"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 19:51:19 -06:00
d379efe90c test: skip tests marked live by default (#4786)
## Description

`pyproject.toml` declares a `live` marker ("opt-in multi-turn tests that
hit real upstream APIs; require provider keys"), and two test files
carry it: `tests/test_thinking_signature_scope_live.py` ("Opt-in:
requires a real key and is gated behind `pytest.mark.live`. Run with
`pytest -m live ...`") and `tests/test_realignment_live_multi_turn.py`
(which describes a "default `pytest -m "not live"` run that never
touches the network"). `addopts` was `-v --tb=short` with no `-m`
filter, so a plain `pytest` run on a machine with `ANTHROPIC_API_KEY` in
the environment, or in the project's `.env` (loaded by
`tests/_dotenv.py`), collected these tests and called the real API.

This PR adds `-m 'not live'` to `addopts`, so the default run deselects
them. `pytest -m live <file>` still selects them (a later `-m` replaces
this one).

Part of #4783

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `pyproject.toml`: `addopts = "-v --tb=short -m 'not live'"`, with a
three-line comment saying why and how to opt in.
- `tests/test_realignment_live_multi_turn.py`: the module-wide `live`
marker is replaced by `@pytest.mark.live` on the eight tests that need a
provider key, so the offline
`test_classify_auth_mode_routes_payg_vs_oauth` keeps running by default.
- `tests/test_live_marker_default_deselect.py`: collection-only test
that the default run deselects live tests, keeps the offline one, and
that `-m live` selects them again (fails against the old `addopts`).

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

This changes pytest configuration only, so no new test is added; the
proof is the collection result below (`--collect-only`, with a dummy
`ANTHROPIC_API_KEY`, nothing contacted).

### Test Output

```text
Before (origin/main pyproject.toml):
pytest tests/test_thinking_signature_scope_live.py tests/test_realignment_live_multi_turn.py --collect-only -q
15 tests collected

After (this branch):
same command                           -> 1 test collected (the offline test_classify_auth_mode_routes_payg_vs_oauth), 14 deselected
same command with -m live              -> 14 tests collected
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, pytest 9.2.0,
`ANTHROPIC_API_KEY=sk-ant-dummy-collect-only` in the environment.
- Exact command / steps: `python -m pytest
tests/test_thinking_signature_scope_live.py
tests/test_realignment_live_multi_turn.py --collect-only -q`, with the
pyproject from `origin/main` and with this branch's, and once more with
`-m live`.
- Observed result: before, 15 tests collected (they would run and call
the API when a key is present); after, the 14 provider-calling tests are
deselected and the offline one is still collected; with `-m live`, 14
collected.
- Not tested: an actual live run (no provider key was used), and CI (the
CI workflows I read do not export a provider key to pytest, so this PR
is aimed at developer machines).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; test configuration only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for the library and proxy. The
default `pytest` run no longer collects tests marked `live`.
- Kill switch / disable path: run `pytest -m live ...` (or pass any `-m`
expression).
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes (the
whole suite was not run; see Testing)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- If a CI job runs `pytest -m live` deliberately it keeps working,
because the later `-m` wins. I did not find such a job.

AI assistance: this change and the runs above were done with Claude
(Sonnet 5.5), operated by me (HetCreep).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4786"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 19:48:28 -06:00
6665940656 fix(memory): return stored id from a background DirectMem0 save (#4759)
## Description

`DirectMem0Adapter.save_memory(background=True)` returned an id from a
second `_generate_id` call. That helper salts with the current time, so
the returned id never matched the Qdrant point the background task
wrote, and a later `memory_update` / `memory_delete` with it found
nothing.

The direct-write ids are now generated once, passed to the background
task, and returned. On the Mem0 extraction path (no pre-extracted data),
Mem0 assigns ids itself, so no id is known up front. The save is marked
`_id_pending`, and `memory_save` reports `memory_id: None` instead of an
invented id. The real id is available from `get_task_status(task_id)`.

Refs #4362 (item 6 only).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/memory/backends/direct_mem0.py`: generate fact ids before
scheduling the background task, then store and return those same ids.
`_write_facts_to_qdrant` accepts optional `memory_ids`. Mem0-path
background saves are marked `_id_pending`. Fixed the `_generate_id`
docstring, which said the ids were "deterministic".
- `headroom/memory/system.py`: `memory_save` returns `memory_id: None`
when the id is pending.
- Background saves snapshot `facts` before generating ids and schedule
that same snapshot, so a caller mutating its list after queueing can't
make the task fail. The returned `_all_memory_ids` is a copy for the
same reason.
- `tests/test_memory/test_direct_mem0_background_ids.py`: 4 regression
tests, using a fake Qdrant client and a fake Mem0 client.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_memory/test_direct_mem0_background_ids.py -v
test_background_save_returns_stored_fact_id PASSED
test_mutating_lists_after_queueing_does_not_break_the_save PASSED
test_background_mem0_extraction_does_not_invent_an_id PASSED
test_memory_save_tool_withholds_a_pending_id PASSED
4 passed in 0.36s
```

## Real Behavior Proof

- Environment: macOS; Qdrant v1.15.5 server in Docker
(`localhost:6333`); `mem0ai` 2.2.1 and `qdrant-client` 1.20.0 (real
`Memory.get/update/delete`); embeddings from Ollama 0.40.6
`nomic-embed-text` through its OpenAI-compatible API
(`OPENAI_BASE_URL=http://localhost:11434/v1`). `DirectMem0Adapter` ran
unmodified, with `enable_graph=False`.
- Exact command / steps: a script on a fresh 768-dim collection:
`save_memory(facts=[...], background=True)` → append to the caller's
`facts` list → `flush_pending()` → `get_memory` / `update_memory` /
`get_memory` / `delete_memory` / `get_memory`, all using the
**returned** id. Run once on this branch and once on `main` (`a385ea8e`)
as a control.
- Observed result: Returned IDs resolve to the stored points;
before/after results are below.

  | | this branch | `main` |
  |---|---|---|
  | task status | completed | completed |
  | returned id is a stored Qdrant point | **True** | False |
| `get_memory(returned id)` | `'Prefers Python over JavaScript'` |
`None` |
| `update_memory` → `get_memory` | `'Prefers Rust over Python'` |
`Memory with id 0ad08f… not found` |
| `delete_memory(returned id)` | `True`, 0 points left | `False`, 2
points left |
| fact appended after queueing | not stored (snapshot) | stored under an
id never returned to the caller |

- Not tested: the Mem0 LLM-extraction background path (`_id_pending`; it
needs an OpenAI LLM and none was available); Neo4j graph writes; hosted
Qdrant; OpenAI's own embedding API.

## Runtime Rollout Safety

- Rollout-managed feature(s): None
- Minimum rollout channel: NA
- Stable/default behavior changed: Only background saves on
`DirectMem0Adapter` (opt-in via `background=true` or
`async_writes=True`). The proxy doesn't use background saves.
Synchronous saves are unchanged.
- Kill switch / disable path: Don't use background saves.
- Unsafe override required: NA
- Qualification impact: None
- Rollback path: Revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4759"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-16 19:47:14 -06:00
JD DavisandCopilot 177fab3782 docs: add guidance for scripted PR/issue comment posting (#4760)
## Description

Documents safe patterns for agent/script-authored PR and issue comments,
so replies like `System.Collections.Hashtable` (seen on #4726) don't
ship unnoticed. Adds a "Posting PR/issue comments from scripts or
agents" section to `CONTRIBUTING.md` with shell-agnostic rules:
file-based bodies and a read-back requirement.

Refs #4757. Only the documentation checkbox of the issue is covered
here, so this does not close it; the issue's decision record is below.

## Type of Change

- [x] Documentation update

## Changes Made

- `CONTRIBUTING.md`: new shell-neutral section: single-string file-based
bodies, no collections in `-f`/`-F`, read-back verification,
delete-and-repost for bad comments.

**Decision on the optional follow-ups (acceptance criterion 2):**
deferred. A governance-workflow regex check and a `gh-reply` helper
script would add new CI/script surface to maintain for a one-off failure
mode. The documented read-back step is the main guardrail. Happy to
follow up if maintainers want either.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
Docs-only change; no code touched. Reviewed the rendered Markdown diff.
```

## Real Behavior Proof

- Environment: any OS/shell with `gh`
- Steps: reviewed `git diff origin/main -- CONTRIBUTING.md`; confirmed
the section renders before "Development setup" and contains no
shell-specific syntax.
- Not tested: not executed against a live PR (docs only).

## Runtime Rollout Safety

- Rollout-managed feature(s): N/A (docs only)
- Minimum rollout channel: N/A
- Stable/default behavior changed: No
- Kill switch / disable path: N/A
- Unsafe override required: No
- Qualification impact: None
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [x] I have made corresponding changes to the documentation
- [x] I did **not** edit `CHANGELOG.md`

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-11-16 19:43:55 -06:00
NAZIYAandJerrettDavis 6b41b6e451 feat(proxy): add cli and settings knob for ccr_inject_system_instructions (#4642)
<!-- maintainer-current-main -->
## Maintainer latest-main checkpoint
Published `d13ba17dff65a5b31253ef3cf0775ffc2504d8da` contains current
upstream main `e4a60ed72f6ef5b28569f8bfbe2163cfda32c948`. 150
CLI/settings/Gemini tests passed. Earlier live behavior proofs below
retain their commit/platform scope; contributor runtime logic is
unchanged.

The common shard-2 failure was independently confirmed on main's hosted
job 127069149772 and locally (four failures, six passes): the Gemini
memory fixture mocked an unused body reader and provided no request
stream. The maintainer test-only follow-up now constructs a real
FastAPI/ASGI Request carrying JSON bytes and exercises the actual body
reader. All six test-function ASTs/assertions are unchanged, and all 10
parameterized Gemini cases pass. Touched-file Ruff/format and whitespace
checks pass. This fixes the inherited fixture instead of asking
contributors to run CI or weakening the memory/content assertions. Fresh
exact-head hosted qualification is requested. Nothing merged.
<!-- /maintainer-current-main -->

## Description



Closes #4636.



`ccr_inject_system_instructions` (which instructs models how to retrieve
compressed context when CCR markers are present) previously existed as a
field on `ProxyConfig` but could only be toggled programmatically via
the Python API. Operators deploying `headroom proxy` as a container or
standalone service had no CLI flag, environment variable, or settings
store entry to enable it without wrapping the process in custom Python
code.



This PR exposes `ccr_inject_system_instructions` as:

1. CLI flag: `--ccr-inject-system-instructions`

2. Environment variable: `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS`

3. Curated settings store field: `ccr_inject_system_instructions` in
`settings_store.SETTINGS`

4. Server environment configuration: loaded in
`_proxy_config_from_env()`



The default remains `False` preserving exact existing behavior.



## Type of Change



- [x] New feature (non-breaking change that adds functionality)



## Changes Made



- `headroom/cli/proxy.py`: Add
`@click.option("--ccr-inject-system-instructions", is_flag=True,
envvar="HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS", ...)`, add parameter
to `proxy()`, and thread it into `ProxyConfig`.

- `headroom/settings_store.py`: Add
`SettingField("HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS",
"ccr_inject_system_instructions", ...)` to `SETTINGS` registry so it is
validated, saved to `settings.json`, and surfaced on `/settings`.

- `headroom/proxy/server.py`: Read
`HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS` in `_proxy_config_from_env()`
for multi-worker and container deployments.

- `docs/content/docs/proxy.mdx`: Document
`--ccr-inject-system-instructions` and
`HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS` in the CCR configuration
table.

- `tests/test_cli_proxy_env.py`: Add test cases verifying
`--ccr-inject-system-instructions`,
`HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS`, `settings.json`, and
`_proxy_config_from_env()` correctly populate
`ProxyConfig.ccr_inject_system_instructions`.



## Testing



- [x] Unit tests pass (`pytest tests/test_proxy/test_settings_store.py
tests/test_cli_proxy_env.py`)

- [x] Linting passes (`ruff check`)

- [x] Formatting passes (`ruff format --check`)

- [x] New tests added for new functionality

- [x] Manual testing performed



### Test Output



```text

$ pytest tests/test_proxy/test_settings_store.py -q

40 passed, 1 warning in 2.18s



$ pytest tests/test_cli_proxy_env.py -k "test_ccr_inject_system_instructions or test_settings_file_reaches_ccr_inject_system_instructions or test_proxy_config_from_env_reads_ccr_inject_system_instructions" -q

5 passed, 91 deselected in 3.42s



$ ruff check headroom/cli/proxy.py headroom/settings_store.py headroom/proxy/server.py tests/test_cli_proxy_env.py

All checks passed!



$ ruff format --check headroom/cli/proxy.py headroom/settings_store.py headroom/proxy/server.py tests/test_cli_proxy_env.py

4 files already formatted

```



## Real Behavior Proof



- Environment: Windows 11 x86_64, Python 3.14.5, Click CliRunner test
harness and settings store test harness.

- Exact command / steps: Executed `runner.invoke(main, ["proxy",
"--ccr-inject-system-instructions"])`, verified captured ProxyConfig;
tested env var `HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS=1`, verified
ProxyConfig; tested `settings.json` with
`{"ccr_inject_system_instructions": true}`, verified ProxyConfig; tested
`_proxy_config_from_env()`.

- Observed result: In all test cases,
ProxyConfig.ccr_inject_system_instructions resolved to True when enabled
and False by default, matching the behavior of sibling knobs like
HEADROOM_CCR_INLINE_RESOLVE and HEADROOM_CCR_BUFFERED_CHAT_STREAMING.

- Not tested: Live Kubernetes deployment or multi-node container
cluster.



## Runtime Rollout Safety



- Rollout-managed feature(s): none, opt-in configuration knob

- Minimum rollout channel: n/a

- Stable/default behavior changed: no, default remains False

- Kill switch / disable path: omit the flag or unset
HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS

- Unsafe override required: no

- Qualification impact: none while disabled

- Rollback path: git revert this commit



## Review Readiness



- [x] I have performed a self-review

- [x] This PR is ready for human review



## Checklist



- [x] My code follows the project's style guidelines

- [x] I have performed a self-review of my code

- [x] I have commented my code, particularly in hard-to-understand areas

- [x] I have made corresponding changes to the documentation

- [x] My changes generate no new warnings

- [x] I have added tests that prove my fix is effective or that my
feature works

- [x] New and existing unit tests pass locally with my changes

- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)



## Additional Notes



Follows the exact pattern established by `--ccr-inline-resolve` and
`--ccr-buffered-chat-streaming` (#4631).



<!-- devin-review-badge-begin -->



---



<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4642"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>

<!-- devin-review-badge-end -->



## Maintainer runtime and integration qualification

Published `9c0b0a10d63227554350a3ba2cb936e09cea1286` includes upstream
main `ee975d49c0b742e34b09df50a01c7b5680e85cdc` with a clean merge and
no contributor logic changes. Main is actively advancing, so final
qualification must check the then-current main again.



Windows/Python 3.12.13: `python -m pytest tests/test_cli_proxy_env.py
tests/test_proxy/test_settings_store.py -q` reports **140 passed** with
the existing development native extension. Touched-file Ruff, format and
whitespace checks pass; no fresh native build claimed.



Real CLI proof (no replaced run_server or provider transport): launch
the actual Click entry point `from headroom.cli import main; main()` in
six isolated subprocesses with `proxy --mode token --no-optimize
--no-cache --no-rate-limit --no-ccr-proactive-expansion
--no-memory-tools --no-memory-context --disable-kompress
--anthropic-api-url http://127.0.0.1:<fixture-port>`. Each process has
an isolated workspace/settings file, offline/beacon opt-out and a real
CompressionStore entry created before launch; GET `/v1/retrieve/{hash}`
confirms ownership. Send `/v1/messages` with that stored hash in a
historical tool-result marker, through the actual TCP proxy to a local
HTTP Anthropic-shaped provider fixture, and inspect the received
upstream JSON.



Observed: all six calls return HTTP 200. Default: no `Compressed Context
Available` instructions. CLI flag, environment
`HEADROOM_CCR_INJECT_SYSTEM_INSTRUCTIONS=1`, and a settings file with
`{"ccr_inject_system_instructions":true}` each produce the instructions
upstream. `--no-ccr` and `--lossless`, both requested alongside the
flag, produce neither instructions nor a retrieval tool despite the
owned historical marker. This verifies the disabled-CCR finding is
addressed on the real request path. The documented
warm-prefix/cache-mode skip remains an intentional limitation; this run
uses token mode.



Not tested: live cloud models, Kubernetes/multiple nodes, every provider
wire shape, warm-cache replay or a fresh native build. The local HTTP
fixture is synthetic; request/configuration/ownership paths and TCP
transport are real. Hosted CI must finish on the final head; nothing
merged.



### Maintainer current-main requalification

Exact head `b38e5a5937cf09c3b5127f3f066e0384db235955`, upstream main
`ea59ac3eef3a3dc02076e9fe4bb30d572ca9c1a9`.

Clean integration with upstream/main
ea59ac3eef. Resolved only the
independently introduced Gemini test-fixture conflict by retaining
main's stricter two-chunk ASGI receive fixture; existing assertions and
contributor production behavior remain intact.114 CLI-env/Gemini-memory
tests pass with actual isolated Click8.5.0; touched-source Ruff and
whitespace pass. Re-reviewed new port/Hermes/settings and
no-response-handling overlap: injection still defaults false and
no-CCR/lossless disable it. Prior six actual CLI/TCP
proxy/upstream-fixture request proofs remain scoped to unchanged
injection behavior; no fresh cloud/native build claim. No new source
finding; exact-head hosted qualification pending. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `5addea3958b4a1bd4eec05ae23a58c9732f516c6`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Final post-Dependabot requalification against upstream/main
c01cf07d94. All six Dependabot updates are
now merged. Relative to this branch's freshly tested prepared tree, the
final main sync changes only the three intended Cargo.lock records;
Python/source/test inputs are unchanged. Re-reviewed the integration and
confirmed complete native source/lock/toolchain objects match the tested
combined build and exact fresh native DLL used for this branch's local
tests.

Focused fresh-native qualification: 114 passed in 14.10s. Selected real
dependency overlays and native DLL paths are recorded in
final4642-fresh-native-tests.log; the DLL provenance/input hashes are in
final-native-artifact-proof.json. No redundant rerun is implied after
this Cargo-only merge. Whitespace passes. Prior source review and scoped
real-behavior evidence remain applicable to unchanged contributor
inputs.

No new source finding in the reviewed scope. Final formal approval
awaits exact-head hosted qualification. The author rebased the branch
after the preceding maintainer publication; independently compared old
and current trees differ only by one blank line in the Gemini fixture.
Current-head guarded synchronization preserves that change and the false
default/no-CCR/lossless behavior.

Maintainers handle hosted checks; no external-provider claim,
contributor semantic rewrite or non-Dependabot merge.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `7b0112872216941c25aa0f88e45701c39df3dd5c`, upstream main
`2b0315b2d4b0d54947050bbdad2de9e89df06378`.

Current-main requalification
against2b0315b2d4b0d54947050bbdad2de9e89df06378. All71 integration
patches exactly match c01cf07-to-current-main delta after diff-location
normalization; contributor flag/default/no-CCR/lossless policy
unchanged.277focusedCLI/Gemini/Kompress/must-keep/doctor tests
pass,1deprecationwarning. Six fresh actual CLI subprocess/TCP
proxy/local upstream cases pass with an owned marker:default
instructionsfalse/retrievaltrue;flag,environment,settings
instructionstrue/retrievaltrue;no-CCR andlossless
instructionsfalse/retrievalfalse. This addresses configuration/wire
behavior,not cloud processing;cache-mode limitation remains
documented.33changedPythonfiles pass Ruff lint/format;offline
lock/whitespacepass. Existingpr3982nativeDLLverified
fullinputhashes/artifactchecksum;no freshbuild claimed.
Currentauthor/Devin discussion checked,previouscode/docs/live-proof
findings remain addressed. No new sourcefinding in reviewed scope.
Exact-head hostedqualification pending beforeapproval;maintainers
handlechecks,nothingmerged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.


### Maintainer current-main requalification

Exact head `60a7ec28a53c5cb55e64e39a9182191b17fe9090`, upstream main
`0ddaa56c21cdc01c4774391fdcc69e987ecc4e5c`.

Clean integration with upstream/main
0ddaa56c21. All 17 integration patches
exactly match the prior-reviewed-main delta after diff-location
normalization; contributor flag/default/no-CCR/lossless policy
unchanged. Fresh CLI/doctor and incoming Vertex/backend tests: 298
passed, 11 skipped, 1 warning. Twelve changed Python files pass Ruff
lint/format; offline lock and whitespace pass. An initial test command
selected a nonexistent settings test path and collected nothing; the
corrected complete suite above passed.

Fresh actual CLI subprocess/TCP proxy/local-upstream cases pass with an
owned marker: default instructions false/retrieval true; flag,
environment and settings instructions true/retrieval true; no-CCR and
lossless instructions false/retrieval false. This establishes local
configuration/wire behavior, not cloud processing; documented cache-mode
limits remain. Author/Devin discussion checked; prior findings addressed
by source guards, documentation and scoped real TCP evidence. Static
production/test review found no unrelated executable, workflow or
dependency changes; flagged inherited workflow-validation job is
excluded by unchanged workflow paths. Full native
input-set/hash/artifact checksum verified before DLL reuse, no fresh
build claimed. Formal approval awaits exact published-head hosted
qualification. Maintainers handle checks; nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-16 19:43:14 -06:00
JD DavisandCopilot e9301f34c1 feat(tray): opt-in system tray icon with savings counter, dashboard and update alert (#4726)
## Description

Adds an **opt-in system tray icon** for Headroom while it runs (Windows
first; macOS/Linux planned). The icon is the Headroom logomark and its
right-click menu shows a compact savings counter (tokens and estimated $
saved), opens the dashboard, alerts when an update is available, and
lets you quit.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- New `headroom/tray/` package: `core.py` (GUI-free stats
polling/formatting), `icon.py` (logomark drawn with Pillow, no binary
assets in the wheel), `app.py` (pystray app), `launcher.py` (spawns the
tray as a child of the proxy).
- `headroom proxy --tray` starts the tray with the proxy; `headroom tray
--port N` attaches to a running proxy.
- Menu: live `▼ tokens · ~$ saved` counter (from `/stats` →
`persistent_savings.lifetime`, polled every 5s), Open Dashboard, "Update
available" (only when newer; reuses `headroom.update_check`, honours
`HEADROOM_UPDATE_CHECK=off`; one-time notification), Quit.
- Icon states: normal, orange dot when an update exists, grey when the
proxy is offline.
- New optional extra `headroom-ai[tray]` (`pystray`, `pillow`).
Unsupported platform / missing extra: `--tray` prints why and the proxy
starts normally.
- Docs: `docs/content/docs/system-tray.mdx` + nav entry; screenshots in
`docs/public/assets/tray/`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
> python -m ruff check headroom\tray headroom\cli tests\test_tray_core.py
All checks passed!
> python -m mypy headroom\tray headroom\cli\tray.py
Success: no issues found in 7 source files
> python -m pytest tests\test_tray_core.py tests\cli tests\test_offline_egress_chokepoint.py tests\test_proxy\test_settings_store.py -q
293 passed, 2 skipped
```

(Scoped runs; I did not run the full suite or `mypy headroom` over the
whole package.)

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.13, 3840x2160 display, `pystray`
+ `pillow` 12.3.0, branch based on `upstream/main` @ 1da8930ab.
- Exact command / steps: run the 4 steps below.
1. `HEADROOM_UPDATE_CHECK=off`, then `python -m headroom.cli proxy
--tray --port 8799`
2. Confirmed `GET /health` → 200 and a `python -m headroom.tray --host
127.0.0.1 --port 8799 --owns-proxy` child process.
3. Opened the taskbar overflow, right-clicked the Headroom icon,
captured the menu (below).
  4. Clicked **Quit Headroom**.
- Observed result: icon appeared with the logomark; menu showed `▼ 256M
tokens · ~$765.96 saved` (my real lifetime savings store), **Open
Dashboard**, **Quit Headroom**. After Quit, `/health` no longer
responded (proxy shut down gracefully).
- Not tested: the real update toast and "Update available" menu item
(covered only by a unit test of `available_update()` and a rendered
icon); clicking Open Dashboard / double-click default action; macOS and
Linux (intentionally unsupported for now); HiDPI/light-theme variations;
`headroom tray` standalone Exit in the UI.

## Runtime Rollout Safety

- Rollout-managed feature(s): None (not rollout-channel gated).
- Minimum rollout channel: N/A
- Stable/default behavior changed: No. Off by default; only active with
`--tray`.
- Kill switch / disable path: omit `--tray` (or pass `--no-tray`).
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: revert this PR; the new extra and command are additive.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes (only
`tests/test_tray_core.py` and `tests/cli` were run)
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

**Tray icon** (left: real icon in the Windows taskbar overflow, zoomed;
then rendered states: live, update available, proxy offline)

![Tray icon
states](https://raw.githubusercontent.com/JerrettDavis/headroom/3e824653772642c95fded190b454fff82f2ce86a/docs/public/assets/tray/tray-icon.png)

**Right-click menu** (real capture)

![Tray right-click
menu](https://raw.githubusercontent.com/JerrettDavis/headroom/3e824653772642c95fded190b454fff82f2ce86a/docs/public/assets/tray/tray-menu.png)

**How to use**

```bash
pip install "headroom-ai[tray]"
headroom proxy --tray        
headroom tray --port 8787    # attach to an already-running proxy
```

Windows hides new tray icons in the `^` overflow next to the clock; drag
the icon onto the taskbar to pin it, then right-click. Full guide:
`docs/content/docs/system-tray.mdx`.

## Feature spec

- **API surface**: `headroom proxy --tray/--no-tray`, `headroom tray
--port N`, extra `headroom-ai[tray]`. No change to existing behavior
when the flag is absent.
- **User stories**: Given the proxy runs with `--tray`, when I
right-click the icon, then I see tokens/$ saved, Open Dashboard and
Quit. Edge: given the proxy is down, the icon turns grey and the tooltip
says offline.
- **Failure modes / recovery**: unsupported platform or missing extra
prints a reason and the proxy starts normally; the tray child exits when
the proxy dies (stdin EOF); a stats fetch failure shows the offline
state and retries every 5s. Quit sends SIGINT for graceful shutdown
without a force-exit deadline.
- **Security**: tray only polls the local proxy `/stats`; no network
calls of its own (update info read from the existing cache; honours
`HEADROOM_UPDATE_CHECK=off`); child is spawned with a fixed argv.

## Dependency justification

- **Why**: `pystray` is the standard cross-platform tray library
(Win/macOS/Linux); `pillow` is required by it for icons and lets us draw
the logomark with no binary assets. No existing dep provides a tray.
- **Maintainers**: pystray (moses-palmer), BSD/LGPL, stable ~0.19.5
releases; Pillow (python-pillow org), very active, frequent security
releases.
- **Install surface**: optional extra only (not default). pystray pulls
`six`, plus pywin32-free ctypes on Windows; Pillow ships prebuilt
wheels. No install-time network access.
- **Why these versions**: `pystray>=0.19.5` is the current release
(required new functionality); `pillow>=12.3.0` matches the repo pin
level already used elsewhere to avoid known CVEs.

## Additional Notes

- The menu screenshot shows my real local savings numbers.
- The Quit item stops the proxy only when the tray was started via
`headroom proxy --tray`; the standalone `headroom tray` shows "Exit" and
only closes the tray.
- Follow-ups: macOS/Linux backends (pystray needs the main thread on
macOS, hence the child-process design).

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-11-16 19:40:55 -06:00
JD DavisandCopilot ba26fd7d92 docs(install): add Windows auto-start and warm-join happy path (#4725)
## Description

Documents the Windows happy path for persistent auto-start and
warm-join. Behavior already exists upstream (boot + 5-minute health
tasks, S4U-to-session fallback from #4627, per-port `wrap`
reuse/recovery); the docs had no end-to-end Windows walkthrough.

Closes #

## Type of Change

- [x] Documentation update

## Changes Made

- Add "Windows happy path: auto-start and warm-join" to
`docs/content/docs/persistent-installs.mdx`.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
Docs-only; no automated tests apply.
```

## Real Behavior Proof

- Environment: Windows 11, non-elevated user, headroom-ai 0.43.1
runtime; persistent-task profiles `local-agents` (9010), `local-copilot`
(9012).
- Exact command / steps: `headroom install status --profile
local-agents`; `headroom wrap claude --port 9010 -- --version` (joined
the running proxy, no second proxy); 3 concurrent `headroom wrap codex
--port 9555 -- --version` on a cold port (one "Starting Headroom proxy",
others joined); confirmed S4U `schtasks /Create` returns "Access is
denied" for this user, matching the documented fallback.
- Observed result: proxy healthy and reused; Codex traffic observed in
proxy log; one proxy per port.
- Not tested: S4U/logged-off recovery (denied on this account);
`headroom install apply` against 0.44 from a clean machine.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert docs commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4725"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-11-16 19:40:21 -06:00
66e0086952 feat(opencode): add repeatable --extra-model flag for custom wrap-time models (#2364)
## Description

`headroom wrap opencode` injects a hardcoded model list
(`HEADROOM_OPENCODE_MODELS`) into the `headroom` provider block of
`opencode.json`. There's no way to add custom models (for example a
free-tier DeepSeek) at wrap time. Hand-patching `opencode.json` is the
only workaround, and the next wrap overwrites it.

This PR adds a repeatable `--extra-model` flag and a
`HEADROOM_OPENCODE_EXTRA_MODELS` env var. Both take
`id[=name[=context_window[=output_limit]]]` specs. Extra models are
merged into the injected provider block alongside the built-ins, and
flag specs win over env specs on id collision. `name` defaults to the
id, `context_window` to 200000, and `output_limit` to `min(16384,
context_window)`.

Fields are separated by `=` rather than `:` because model ids routinely
contain `:`. Ollama ids are `name:tag` (`qwen2.5-coder:7b`), and
OpenRouter free-tier ids end in `:free`. With a `:` separator those were
silently cut down to a different id (Devin review, 2026-10-03).

The PR also fixes the wrap/unwrap lifecycle the flag depends on. A wrap
that starts without an `opencode.json` used to write the file with no
backup and no record of what Headroom put in it, so `headroom unwrap
opencode` left the injected models and the proxy base URL behind while
reporting that OpenCode was no longer routed through Headroom, and a
re-wrap under `--openai-api-url` kept an extra model Headroom had
injected but no longer advertised. Wrap now records what it owns in
`opencode.json.headroom-managed.json`, and unwrap uses that record: it
removes the model ids and the base URL Headroom wrote, keeps everything
else in the file, drops an extra model that is no longer requested, and
deletes the config file when Headroom created it and nothing else is
left in it.

Closes #2359

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [x] Documentation update
- [ ] Refactoring (no functional changes)

## Changes Made

- `headroom/providers/opencode/config.py`:
- New `parse_extra_model_spec()`, which splits on `=` with an optional
fourth `output_limit` field and raises `ValueError` on an empty id or a
non-integer / non-positive limit.
- New `extra_models_from_env()`, which reads comma-separated specs from
`HEADROOM_OPENCODE_EXTRA_MODELS`.
- An `extra_models` parameter threaded through
`headroom_provider_entry()`, `_render_provider_block()` and
`inject_opencode_provider_config()`.
- With `keep_user_entries` (#3125's `--openai-api-url` gate),
`--extra-model` entries win over a kept user entry with the same id.
- New managed-state sidecar (`opencode.json.headroom-managed.json`)
recording the model ids and base URL Headroom injected and whether
Headroom created the config file. `inject_opencode_provider_config()`
writes it on every injection; `record_opencode_config_origin()` marks a
config that Headroom creates because none existed.
- `remove_managed_opencode_provider()` drops the recorded model ids, the
recorded base URL, and the provider block once nothing else is left in
it. Entries the user added or edited are preserved.
- `_drop_stale_managed_models()` prunes an extra model Headroom injected
but no longer advertises, including an extra that overrode a built-in id
and is dropped again, so a re-wrap without the flag restores the
built-in defaults rather than keeping the old override.
- `snapshot_opencode_config_if_unwrapped()` no longer snapshots a config
Headroom wrote itself as pre-wrap content.
- `headroom/providers/opencode/runtime.py`: extras also reach the
launch-time `OPENCODE_CONFIG_CONTENT`.
- `headroom/cli/wrap.py`:
- A repeatable `--extra-model` option on `wrap opencode`. Env and flag
specs are parsed up front, and a malformed spec raises
`click.BadParameter`. The merged dict goes to both
`inject_opencode_provider_config` call sites (prepare-only and launch)
and to the launch env. Also adds a help text and docstring example.
- `wrap opencode` records the config origin before the first config
mutation.
- `unwrap opencode` undoes the recorded provider entries when there is
no pre-wrap backup, keeps the rest of the file, and removes a config
Headroom created once nothing Headroom-owned is left in it.
- `docs/content/docs/opencode.mdx`: new "Adding models" subsection
(format, defaults, Ollama/OpenRouter example, built-in id override, the
env var's no-comma limitation) and an env-var table row. The subsection
and the "How It Works Under The Hood" unwrap step now describe the
managed-state record and the automatic cleanup it enables.
- Tests:
- Spec parsing: full spec with output, defaults, the output cap at small
contexts, colon-bearing ids (`qwen2.5-coder:7b`,
`deepseek/deepseek-chat:free`, a name containing `:`), and invalid specs
including a bad output limit.
- Env parsing, provider-entry merge, and end-to-end inject. Also
@JerrettDavis's collision test across both `keep_user_entries` values.
- CLI: the flag reaches `opencode.json`, the env var persists, both
reach `OPENCODE_CONFIG_CONTENT` on launch, the flag wins over a kept
user entry under `--openai-api-url`, and a malformed spec exits with a
usage error.
- Managed state: the sidecar records the injected ids and base URL, a
config the user already had is not marked as Headroom-created, a re-wrap
does not snapshot Headroom's own config, a dropped extra model is pruned
(including an override of a built-in id, which returns to the built-in
defaults), a user model and a user-edited model survive, and the
provider removal keeps unrelated user entries.
- Lifecycle: a fresh wrap followed by an unwrap removes the config file
again (with and without the MCP entry), user content added after a fresh
wrap survives the unwrap, a re-wrap without the extra model drops the
stale one, and a directly injected provider entry is undone by the
unwrap callback as well.
- A user setting that is false, zero or an empty string is content, not
an empty section: such a setting survives the unwrap instead of costing
the user the whole file.

## Testing

- [x] Unit tests pass locally
- [x] Lint/format/type checks pass locally
- [x] New tests added for new functionality

The new parsing tests fail against the previous `:`-based parser:

```text
FAILED tests/test_providers_opencode_config.py::test_parse_extra_model_spec_keeps_colons_in_ids[qwen2.5-coder:7b-...]
FAILED tests/test_providers_opencode_config.py::test_parse_extra_model_spec_keeps_colons_in_ids[deepseek/deepseek-chat:free=DeepSeek Free-...]
FAILED tests/test_providers_opencode_config.py::test_parse_extra_model_spec_full
...
================ 21 failed, 2 passed, 103 deselected in 0.86s =================
```

With this branch (head 6be848942):

```text
$ pytest tests/test_providers_opencode_config.py tests/test_cli/test_wrap_opencode.py -q
179 passed, 2 skipped in 3.31s

$ pytest tests -q -k "opencode"
322 passed, 26 skipped

$ ruff check headroom/providers/opencode/config.py headroom/cli/wrap.py tests/test_providers_opencode_config.py tests/test_cli/test_wrap_opencode.py
All checks passed!
$ ruff format --check (same files)
4 files already formatted
$ mypy headroom --ignore-missing-imports --no-site-packages
headroom\fileperms.py:92: error: Module has no attribute "fchmod"; maybe "chmod"?  [attr-defined]
headroom\proxy\interceptors\astgrep.py:220: error: Module has no attribute "O_DIRECTORY"  [attr-defined]
...
Found 7 errors in 2 files (checked 561 source files)
```

The mypy errors are the Windows-only `os.fchmod` / `os.O_DIRECTORY`
false positives in files this PR does not touch. The same Windows
path-escaping failures that `test_build_launch_env_with_project` used to
hit (`tests/test_cli/test_unwrap_claude.py`,
`tests/test_wrap_code_memory.py`, `tests/test_install/test_planner.py`,
`tests/test_cli/test_wrap_stale_marker.py`) reproduce on unchanged
`main` and are unrelated to this change. `Dependency audit (pip-audit)`
passes on this head; only `Rust advisories (cargo audit)` fails, on
advisories published after upstream `main`'s last green run (2026-11-13
04:47) — `quinn-udp` `RUSTSEC-2026-0538` and `rustix`
`RUSTSEC-2026-0486`. `Cargo.lock` on this branch is byte-identical to
upstream `main` (`81818d49`), so the PR does not introduce them.

## Real Behavior Proof

- Environment: Windows 11, Python 3.13, this branch vs. the previous
`:`-based parser and vs. the previous unwrap lifecycle,
`HOME`/`USERPROFILE` pointed at a temp dir.
- Exact command / steps: `headroom wrap opencode --prepare-only
--no-serena --extra-model "qwen2.5-coder:7b" --extra-model
"deepseek/deepseek-chat:free=DeepSeek Free=65536"`, then read
`provider.headroom.models` from `~/.config/opencode/opencode.json`; then
`headroom unwrap opencode --no-stop-proxy` against a config that did not
exist before the wrap.
- Observed result: with the old parser the entries were `qwen2.5-coder`
(name `7b`) and `deepseek/deepseek-chat` (name `free=DeepSeek
Free=65536`, context 200000): wrong ids, and the paid DeepSeek model
instead of the free one. With this branch they're `qwen2.5-coder:7b`
(name defaults to the id, context 200000, output 16384) and
`deepseek/deepseek-chat:free` (name `DeepSeek Free`, context 65536,
output 16384), next to the built-in `gpt-4o` / `gpt-4.1`. After the
unwrap of a fresh wrap, `opencode.json` and its managed-state sidecar
are both gone and no `provider.headroom` block is left. After the unwrap
of a fresh wrap the user had added to, the file keeps the user's own
model id, key and `theme`, and only the injected ids and the
`http://127.0.0.1:8787/v1` base URL are gone.
- Not tested: a full (non-prepare-only) wrap launching the real OpenCode
binary against a live upstream; the OpenCode TS plugin side (extras are
user-supplied and intentionally not synced into
`plugins/opencode/src/provider.ts` `DEFAULT_MODELS`).

## Runtime Rollout Safety

- Rollout-managed feature(s): none; CLI-only opt-in flag and env var on
`headroom wrap opencode`, plus a `opencode.json.headroom-managed.json`
sidecar next to the OpenCode config
- Minimum rollout channel: stable
- Stable/default behavior changed: no. Without `--extra-model` or
`HEADROOM_OPENCODE_EXTRA_MODELS` the injected provider block is
unchanged. `unwrap opencode` now also removes the provider entries
Headroom recorded, which is what it already reported doing
- Kill switch / disable path: drop the flag and unset
`HEADROOM_OPENCODE_EXTRA_MODELS`, then re-wrap; delete the sidecar to
stop Headroom from tracking the config
- Unsafe override required: no
- Qualification impact: none; no proxy, compression or request-path code
changed
- Rollback path: revert the PR; `headroom unwrap opencode` restores the
pre-wrap `opencode.json` when a backup exists

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/2364"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-17 02:38:10 +05:30
863b3c7cc7 fix(opencode): percent-encode the project name in x-headroom-project (#4780)
## Description

The OpenCode plugin set `x-headroom-project` to the raw project name in
`plugins/opencode/src/transport.ts` (two sites) and
`plugins/opencode/src/plugin.ts`. A header value must be a ByteString,
so a project name with any character above U+00FF (Thai, Chinese,
Cyrillic, Arabic, `€`, emoji and so on) made `Headers.set` throw a
`TypeError` inside the patched `fetch` and `https.request`, and every
routed request failed.

`headroom/cli/wrap.py` already percent-encodes this header for the same
reason, and the proxy decodes it (`sanitize_project_name` calls
`unquote`). This PR does the same in the plugin through one helper,
`projectHeaderValue`, used at the three sites. Header values that
contain only characters from the URL-unreserved set are sent as before.

Closes #4779

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `plugins/opencode/src/transport.ts`: new exported
`projectHeaderValue(project)` (`encodeURIComponent`), used for
`x-headroom-project` in the fetch and Node request paths.
- `plugins/opencode/src/plugin.ts`: the model headers set by
`routeModelsThroughProxy` use `projectHeaderValue(project)`.
- `plugins/opencode/src/transport.test.ts`: two tests with a Thai,
Chinese, Cyrillic and Polish project name, one through `fetch` and one
through Node `https.request` against a loopback server.
- `headroom/providers/opencode/_dist/index.js` and
`hook-shim/handler.js`: regenerated with `npm run build:standalone` (the
CI step compares both with a fresh build; the staged blobs equal the
build output).

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

This is a TypeScript-only change, so the Python checks above do not
apply; the plugin equivalents were run (Node v24.22.2, vitest, `npm ci
--ignore-scripts` from the committed lockfile, Ubuntu on WSL2).

### Test Output

Fail before (`transport.ts` from `3c66c91e`, new tests kept):

```text
vitest run src/transport.test.ts -t "non-Latin-1"
 × percent-encodes a non-Latin-1 project name instead of throwing on routed fetch calls
 × percent-encodes a non-Latin-1 project name on routed Node https.request calls
TypeError: Cannot convert argument to a ByteString because the character at index 8 has a value of 3650 which is greater than 255.
TypeError: Cannot convert argument to a ByteString because the character at index 0 has a value of 3650 which is greater than 255.
Tests  2 failed | 47 skipped (49)
```

Pass after (this branch):

```text
vitest run src/transport.test.ts -t "non-Latin-1"
Tests  2 passed | 47 skipped (49)

vitest run (whole plugins/opencode suite)
Test Files  4 passed (4)
Tests  81 passed (81)

npm run typecheck -> exit 0
npm run build:standalone -> success; the committed bundle equals this build
```

## Real Behavior Proof

- Environment: Ubuntu on WSL2, Node v24.22.2, the committed wheel bundle
`headroom/providers/opencode/_dist/index.js` (not vitest), and a
loopback `node:http` server standing in for the proxy.
- Exact command / steps: load the bundle (`origin/main` for "before",
this branch for "after"), call `default.server({ project: { id: "p" },
directory: "/work" }, { proxyUrl: "http://127.0.0.1:<port>", project
})`, then `fetch("https://api.anthropic.com/v1/messages", { method:
"POST", body: "{}" })`, once with `project = "my-project"` and once with
a Thai, Chinese and Cyrillic string.
- Observed result: before, `my-project` was sent as `my-project` and the
Thai/Chinese/Cyrillic name made `fetch` throw `TypeError: Cannot convert
argument to a ByteString ... 3650 ... greater than 255` with 0 requests
recorded. After, `my-project` is still sent as `my-project`, and the
multilingual name reaches the recorder as
`%E0%B9%82...%2F%E4%B8%AD%E6%96%87%2F%D0%9F...` which
`decodeURIComponent` turns back into the original string.
- Not tested: a real Headroom proxy (the recorder only shows what is
sent; the decode step is from reading
`headroom/proxy/project_name_policy.py`), OpenCode 1.x or 2.x hosts, and
anything outside the proxy that reads the header.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; OpenCode plugin transport only.
- Minimum rollout channel: n/a
- Stable/default behavior changed: only for a project name with
characters outside the URL-unreserved set. Those are now percent-encoded
(a path such as `/home/u/app` is sent as `%2Fhome%2Fu%2Fapp`) and the
proxy decodes them; plain names like `my-project` are unchanged.
- Kill switch / disable path: n/a
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- The plugin test for `routeModelsThroughProxy` was not extended; the
`plugin.ts` site is the same one-line call to the helper, covered by the
bundle proof above.
- The shell env variable `HEADROOM_PROJECT` is unchanged: it is an
environment value, not a header.

AI assistance: this change, its tests and the runs above were done with
Claude (Sonnet 5.5), operated by me (HetCreep).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4780"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-17 02:37:46 +05:30
ParideboyandJerrettDavis 87c12931fb fix(opencode): stop routing a host's bare /messages or /responses to the proxy (#4781) (#4791)
## Description

A `POST` whose path ends in a wire endpoint but that sits at a host's
origin root (`/messages`, `/responses`, `/chat/completions`) is no
longer routed to the proxy unless that origin is a configured provider
base URL.

Closes #4781.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `plugins/opencode/src/transport.ts` — `hasWireRootBefore` no longer
treats "no segment before the endpoint" as a wire root, so a bare
`/messages` at an unknown host goes direct.
- `plugins/opencode/src/transport.test.ts` — the bare-endpoint
assertions from #4758 move into a #4781 case that pins both directions:
direct by default, routed once the base is registered; the #4758 case
keeps the prefixes that still route.
- `plugins/opencode/README.md` — states the actual routing rule, and the
two shapes that still route without a configured base.
- `headroom/providers/opencode/_dist/index.js`,
`headroom/providers/opencode/hook-shim/handler.js` — rebuilt, since the
`opencode-plugin` gate compares bytes against `dist-standalone`.
- `plugins/opencode/scripts/live-proof-4781.mjs` — the live proof below,
committed so it can be rerun.

## Why

#4758 stopped a suffix match from routing non-LLM hosts, but kept bare
endpoints routable "because compatible gateways use those paths". That
left exactly one non-LLM shape still leaking: `POST
https://rest.messagebird.com/messages` with `Authorization: AccessKey
...` went to the proxy, carrying a non-LLM credential (#4781).

Becoming the bare path is not evidence of a provider — it is equally the
shape of a messaging API. A provider mounted at an origin root is still
routed, because its base URL *is* a configured wire root
(`isConfiguredWireRoot` matches every path on that origin when the base
path is empty), which is how OpenCode reports
`provider.<id>.options.baseURL`.

```ts
return (
  last !== undefined &&
  (isVersionSegment(last) || IMMEDIATE_WIRE_ROOTS.has(last) || segments.some((s) => NAMED_WIRE_ROOTS.has(s)))
);
```

Everything else is untouched: version prefixes (`/v1`, `/v1beta`,
`/v4`), immediate `api`, named roots (`backend-api`, `codex`, `openai`,
`anthropic`, `zen`), Gemini colon actions, configured bases,
`excludeHosts`, and the method gate.

## Testing

- [x] New tests added for new functionality
- [x] Manual testing performed

`plugins/opencode` is a TypeScript package, so its gates are npm scripts
rather than `pytest` / `ruff` / `mypy`; `npm audit` and the rest run in
the repo's own CI.

### Test Output

```text
> vitest run
Test Files  4 passed (4)
     Tests  80 passed (80)

> node node_modules/@typescript/native/bin/tsc --noEmit
(clean, no output)

> node scripts/live-proof-4781.mjs
=== decisions (real sockets, real fetch, real built plugin) ===
PASS  MessageBird-shaped  POST /messages             -> left alone
PASS  MessageBird batches  POST /messages/batches    -> left alone
PASS  LLM control         POST /v1/messages          -> REROUTED to proxy
PASS  LLM control         POST /v1/chat/completions  -> REROUTED to proxy

=== what reached the proxy ===
  POST /v1/messages  x-headroom-base-url=http://localtest.me:8899  original-path=null
  POST /v1/chat/completions  x-headroom-base-url=http://localtest.me:8899  original-path=/v1/chat/completions

=== MessageBird-shaped requests that reached the proxy: 0 ===
=== what reached the upstream directly ===
  POST /messages  auth=AccessKey FAKE-TEST-VALUE  (no proxy headers: true)
  POST /messages/batches  auth=AccessKey FAKE-TEST-VALUE  (no proxy headers: true)

ALL CASES PASS
```

Before the fix, same script, source reverted to `HEAD~1` and the bundle
rebuilt:

```text
=== decisions (real sockets, real fetch, real built plugin) ===
FAIL  MessageBird-shaped  POST /messages             -> REROUTED to proxy
PASS  MessageBird batches  POST /messages/batches    -> left alone
PASS  LLM control         POST /v1/messages          -> REROUTED to proxy
PASS  LLM control         POST /v1/chat/completions  -> REROUTED to proxy

=== what reached the proxy ===
  POST /messages  x-headroom-base-url=http://localtest.me:8899  original-path=null
```

Reverting only `src/transport.ts` also fails the unit test (`1 failed |
79 passed`), satisfying "a test that fails before your fix and passes
after".

## Real Behavior Proof

- Environment: Windows 11 (win32), Node v26.10.0, this branch of
`plugins/opencode`, committed wheel bundle
`headroom/providers/opencode/_dist/index.js` loaded via
`plugin.server(...)` as `headroom wrap opencode` does; upstream and
proxy are local HTTP fixtures on real TCP sockets and `globalThis.fetch`
is Node's undici, not a stub.
- Exact command / steps: `cd plugins/opencode && npm ci && npm run
build:standalone && node scripts/live-proof-4781.mjs`, which POSTs a
MessageBird-shaped SMS payload and two LLM control requests to
`http://localtest.me:8899` (resolves to 127.0.0.1 but is not a loopback
spelling, so traffic really leaves the process) and records which server
received each one.
- Observed result: after the patch `POST /messages` and `POST
/messages/batches` reach the upstream directly with no
`x-headroom-base-url` header and the proxy receives zero
MessageBird-shaped requests, while `/v1/messages` and
`/v1/chat/completions` still reach the proxy with the upstream origin
header; with `src/transport.ts` reverted to `HEAD~1` the same run
reroutes `POST /messages` to the proxy, reproducing the reported leak
over a real socket (full output in `Testing` → `Test Output`).
- Not tested: a real MessageBird account and the real
`rest.messagebird.com` host, since the payload and `AccessKey` are fake
and routing is decided before DNS; the proxy's handling of an SMS body,
which never arrives after this change; and a live OpenCode host process,
since the fixture drives the plugin's own `server()` entry rather than
an OpenCode session.

## Runtime Rollout Safety

- Rollout-managed feature(s): none, no feature flag is involved.
- Minimum rollout channel: next released `headroom-opencode` / wheel.
- Stable/default behavior changed: yes, deliberately — a `POST` to
`/messages`, `/responses` or `/chat/completions` at a host's origin root
now goes direct unless that origin is a configured provider base URL;
users whose gateway is configured via `provider.<id>.options.baseURL`
are unaffected.
- Kill switch / disable path: `excludeHosts` /
`HEADROOM_OPENCODE_EXCLUDE_HOSTS` forces a host direct, and configuring
the base URL or listing it in `wireRoots` forces a bare-path provider
through the proxy.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert the commits; the transport and the committed
bundle revert together.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title

## Additional Notes

Residual, deliberately not solved and now stated in the README:
`/api/v2/messages` and `/v2/responses` still route. No path shape
separates those from a gateway rooted at `/api` or `/v2`; only the host
or the request body could, and reading the body inside a sync `fetch`
wrapper buys a new class of failure for a shape `excludeHosts` already
covers. I did not add a known-host allowlist either: it can only
recognise hosts someone already thought of, and it regresses compatible
gateways silently — the failure mode #4747 was filed about.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-17 02:36:58 +05:30
Tejas ChopraandClaude Opus 5.5 5023c84e52 fix(proxy): refuse a private x-headroom-base-url on the async chat and responses path (#4879)
## What changes

Main currently fails four tests in
`test_grok_shared_proxy_call_sites.py` and `test_upstream_guard.py`.

Two changes to `handlers/openai.py` merged independently:
- One made `_resolve_openai_upstream_base` reject a non-public
`x-headroom-base-url` with 400.
- The other moved the chat and responses handlers onto a new
`_resolve_openai_upstream_base_async`. It was written before the
rejection existed, so it still logged "ignoring" and fell back to the
configured upstream.

Since the handlers call the async resolver, a private override on
`/v1/chat/completions` and `/v1/responses` was served from the default
upstream instead of being refused.

Both resolvers now refuse through one shared
`_reject_unsafe_openai_override`, so they can't diverge again. The guard
check still runs off the event loop. The change is source-only in one
file, with no test changes: the existing tests on main already cover it.

## Verification

- `tests/test_grok_shared_proxy_call_sites.py`,
`tests/test_upstream_guard.py`,
`tests/test_proxy/test_openai_upstream_guard_off_loop.py`, and the two
OpenAI cache-key/stability suites: 109 passed. The four that fail on
main now pass.
- `tests/test_proxy -k "openai or upstream or base_url"`: 90 passed.
- `ruff check` and `ruff format --check` are clean. mypy reports no
errors in `openai.py`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4879"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:18:51 +05:30
d6df127d57 feat(compress): accept Gemini and Bedrock Converse bodies on the gateway turn (#4679)
## Description

`/v1/compress` in gateway mode (a request with a `gateway` block) now
accepts two more native request bodies:
- Gemini `generateContent`
- Bedrock Converse

It compresses their text slots and returns `body` in the same native
shape. This lets a gateway like LiteLLM compress native Gemini and
Bedrock passthrough traffic, which it couldn't before.

`config.mode: "ccr"` is rejected for these shapes with a 400. Every
other mode is marker-free for them.

(Was stacked on #4617, which has merged; the branch now carries current
`main`.)

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### How

This works the same way `gateway_responses` does for Codex: it builds a
view instead of translating the body.
- Every compressible text slot becomes one chat-shaped pseudo message.
- Compressed text is written back by its path in the body.
- Everything else stays byte-identical: `thoughtSignature`, thought
parts, `functionCall`, `inlineData`, `toolUse`, `cachePoint`,
`reasoningContent`, `tools`/`toolConfig`.
- Tool calls appear in the view, so tool-name rules still apply. `Read`
stays verbatim, for example.
- A JSON slot is only written back if it still parses to the same JSON
type.
- If the compressed list doesn't line up with the slots, the original
transcript is forwarded and a warning is logged.

### CCR on native bodies (review fix)

A native body can't carry `headroom_retrieve`. Its `tools`/`toolConfig`
are forwarded untouched, and `create_ccr_tool_definition` only builds
chat, Anthropic and Responses shapes. It also never gets a re-drive. So
a CCR marker in a native body points at content the model can never
fetch. Before this fix, `config.mode: "ccr"` on a Gemini or Converse
gateway turn did exactly that: log output came back as a marker with
`ccr_hashes: [<hash>]` and `obligations: []`.

- `handle_compress` now answers `ccr` on a native view with `400
invalid_request`. The message names the shape, explains why, and lists
the supported modes.
- The check runs where the mode is chosen, right after `attach_view` has
recorded the shape. That is before any compression, session work or CCR
store write.
- **The rejection does not depend on `gateway.can_redrive`.** Even a
gateway that can re-drive has no way to put the retrieval tool into a
native request, so there is no capability under which native `ccr`
works.
- **Without a `gateway` block nothing changes.** A Gemini body is
already a 400 (no `messages`). A Converse body goes through the plain
messages path, where its untyped blocks are opaque: 0 tokens saved, no
markers in any mode, `ccr` included (checked). So the rule is
consistent: reject `ccr` wherever Headroom reads a native shape.
- The bypass header still returns the body untouched with no
compression, in any mode.
- I checked every other mode on the same marker-producing input
(default, `lossless`, `lossy_inline`, `lossless_then_lossy`): no marker
and no hash in any of them. These modes build pipelines with
`ccr_inject_marker=False, ccr_enabled=False`. Stale-Read notices are
hashless in marker-free modes since #4678.
- **Second defect, same family:** with `HEADROOM_DEDUPE=1`, cross-turn
dedup wrote `[↑NL same as msg 4]` into a Converse tool result. That `4`
counts view messages (system blocks, tool calls, result slots), not the
caller's `messages`, so it named the wrong message. Native views now set
`cross_turn_dedup_recoverable=False`, as `lossless` mode already did.
- The `not self.native_shape` guards on CCR arming and re-drive in
`GatewayTurn.finish` stay as defence in depth.

### Caller contract

**Gemini**
- Send top-level `contents` as a list, with no `messages` or `input`.
- `systemInstruction`/`system_instruction`, `tools` and
`generationConfig` are optional and pass through.

**Converse**
- Send `messages` where every message has role `user` or `assistant`,
and every content block has no `type` key.
- At least one block must use a Converse key: `text`, `toolUse`,
`toolResult`, `cachePoint`, and so on.
- `system` is optional, as a list of blocks.

**Both shapes**
- `config.mode`: omit it (marker-free default), or use `lossless`,
`lossy_inline` or `lossless_then_lossy`. `ccr` is a 400.
- Only gateway turns get a view. Without a `gateway` block the endpoint
behaves exactly as before. A client-sent `_headroom_native_view` key is
dropped, and `BODY_CONTROL_KEYS` strips the real one before the body is
returned.
- `config.frozen_message_count` and `config.protect_recent` count native
messages: Gemini `contents` entries and Converse `messages`.
`session.frozen_message_count` comes back in native units. The
`/v1/usage` reply's `frozen_message_count` still counts view messages,
and the docs say so.
- No re-drive is ever armed, so `obligations` never contains `redrive`,
and request hooks run stream-safe-only. Tool compaction and output
steering are skipped.
- Relayed usage can be:
- Gemini `usageMetadata`, where `promptTokenCount` already includes
`cachedContentTokenCount`.
- Converse `usage`, where `inputTokens` excludes `cacheReadInputTokens`
and `cacheWriteInputTokens`.
  - The whole provider response that carries either.
- Every gateway answer, fail-open ones included, carries `wire_shape`:
`gemini`, `converse`, `openai_responses` or `messages`. A caller that
sent a native body and gets anything else, or no field at all, is
talking to an older Headroom and should forward its original body.
- Top-level `messages` in the answer is the Converse list for Converse
and the chat-shaped view for Gemini.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): no new errors; the 8
reported are in `ccr/mcp_server.py` and `cache/prefix_tracker.py` and
are identical on `main`
- [x] New tests added for new functionality
- [x] Manual testing performed (local replay, below)

New in `tests/gateway/test_native_shapes.py`:
- `test_ccr_mode_is_rejected_for_native_bodies[gemini|converse]`: 400,
`invalid_request`, message asserted.
- `test_other_modes_forward_native_bodies_without_ccr_markers`: 4 modes
x 2 shapes, on log output large enough to produce a marker. Checks no
hash, no marker, the shape echoed, non-message keys byte-identical and
the native list mapped back.
- `test_ccr_still_works_on_a_chat_gateway_turn`: the same log output in
a chat body with `ccr` still carries a marker. This proves the input
above is marker-producing, and that chat `ccr` didn't regress.
- `test_native_bodies_get_no_dedup_pointer_into_view_messages`.
- `test_native_turns_are_never_asked_to_redrive` now runs in the default
mode, because `ccr` is a 400.

### Test Output

```text
# before the fix (new tests only, rest of the file passing)
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[gemini]
FAILED tests/gateway/test_native_shapes.py::test_ccr_mode_is_rejected_for_native_bodies[converse]
FAILED tests/gateway/test_native_shapes.py::test_native_bodies_get_no_dedup_pointer_into_view_messages
3 failed, 25 passed

# after
tests/gateway/test_native_shapes.py: 28 passed

# tests/gateway tests/test_gateway_turn_unit.py tests/test_compress_api.py
# tests/test_proxy_compress_endpoint.py tests/test_compress_session_mode.py
# tests/test_compress_session_shared.py tests/test_compress_passthrough.py tests/test_compress_failure.py
1 failed, 507 passed, 4 skipped
# the one failure, test_lossy_inline_emits_no_ccr_markers[1-json_minifier], fails identically on main fda75de5

uvx ruff@0.17.3 check / format --check (changed files): All checks passed! / 3 files already formatted
```

## Real Behavior Proof

These are local synthetic replays. They are not provider qualification.

- Environment: macOS, Python venv, branch head `530c20aa`. Headroom runs
in-process via `create_app(ProxyConfig(optimize=True,
cache_enabled=False, rate_limit_enabled=False,
cost_tracking_enabled=False, log_requests=False, image_optimize=False))`
behind `fastapi.testclient.TestClient` from a loopback client. The
"provider" is an `httpx.MockTransport` that records each body it
receives and answers with a Gemini `generateContent` or Bedrock Converse
response, including usage. `LITELLM_LOCAL_MODEL_COST_MAP=True`. No real
provider calls.
- Exact command / steps: `LITELLM_LOCAL_MODEL_COST_MAP=True python -I -c
"import sys; sys.path.insert(0,'<worktree>');
exec(open('native_replay.py').read())"`. For each shape the script does
three things:
1. POSTs the native body plus `gateway: {can_redrive, session_affinity,
can_relay_response: true}` to `/v1/compress`.
2. Sends the returned `body` through the mock upstream at the real
Gemini or Bedrock URL.
3. POSTs the provider's whole response to `/v1/compress/response` with
the `turn_id`.

It then sends both native bodies again with `config.mode: "ccr"`. The
tool output is the 399-line grep listing used in the tests. The `ccr`
requests use 3000 log lines, which produce a marker in `ccr` mode.
- Observed result:

```text
[gemini] request half: wire_shape=gemini obligations=['relay_usage'] tokens 7246->3863 ccr_hashes=[] marker_in_body=False
[gemini] all 399 grep lines still readable in the forwarded body=True
[gemini] non-message keys unchanged=True first two contents unchanged=True
[gemini] provider got 16125 bytes (raw 29694), control keys forwarded=[], host=generativelanguage.googleapis.com
[gemini] response half: 200 {'action': 'done', ..., 'billed_usage': {'input_tokens': 900, 'output_tokens': 12, 'cache_read_input_tokens': 512}}
[converse] request half: wire_shape=converse obligations=['relay_usage'] tokens 8823->4843 ccr_hashes=[] marker_in_body=False
[converse] all 399 grep lines still readable in the forwarded body=True
[converse] non-message keys unchanged=True first two messages unchanged=True
[converse] provider got 16184 bytes (raw 29761), control keys forwarded=[], host=bedrock-runtime.us-west-2.amazonaws.com
[converse] response half: 200 {'action': 'done', ..., 'billed_usage': {'input_tokens': 300, 'output_tokens': 12, 'cache_read_input_tokens': 600, 'cache_creation_input_tokens': 0}}
[gemini ccr] 400 {'error': {'type': 'invalid_request', 'message': "config.mode='ccr' is not supported for gemini bodies: Headroom cannot add its retrieval tool to a native Gemini or Bedrock Converse request, so the model could not resolve a CCR marker. Use lossless, lossy_inline or lossless_then_lossy, or omit config.mode for the default marker-free mode."}}
[converse ccr] 400 {'error': {'type': 'invalid_request', 'message': "config.mode='ccr' is not supported for converse bodies: ..."}}
provider calls: 2
```

Before the fix, the same `ccr` requests returned 200 with `ccr_hashes`
of length 1, a `hash=` marker in the forwarded body, and `obligations:
[]`.
- Not tested:
- live Gemini (AI Studio or Vertex) and live Bedrock Converse: the
provider was mocked, so provider acceptance of the rewritten bodies is
unverified;
  - real Kong `ai-proxy` or a real LiteLLM gateway in front of Headroom;
- streaming callers (`streamGenerateContent`, `ConverseStream`): the
request half is identical, but no streamed response was relayed;
  - Gemini `thoughtsTokenCount` in output accounting;
  - session mode with a Redis store for native bodies;
- multimodal parts (`inlineData`, `image`, `document`) beyond their
byte-identical pass-through in unit tests.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. Native shapes only activate for
gateway-mode `/v1/compress` callers that send a `gateway` block and a
native body.
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for existing callers. Chat,
Anthropic and Responses bodies are unchanged. A Converse body without a
`gateway` block goes through the plain path as before. Before this PR, a
native body with a `gateway` block was misread as chat (Converse) or
rejected (Gemini).
- Kill switch / disable path: the gateway omits the `gateway` block, or
forwards its original body when `wire_shape` doesn't match.
- Unsafe override required: no
- Qualification impact: none for existing paths
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Known follow-ups, not in this PR:
- A `functionResponse.response` with any string field never compresses
its nested dict values. The JSON slot is only used when there are no
string fields. This is a missed saving, not a correctness bug.
- `handlers/gemini.py` (the proxy path) and `gateway_native.py` map
Gemini parts with different rules, and their part-key lists are kept in
step by hand. They should share one constant.
- `protect_recent` translation and stream-safe hook gating for native
turns have no dedicated test.
- Gemini `thoughtsTokenCount` is not added to output tokens when usage
is relayed, so output savings attribution undercounts for thinking
models.
- This is not specific to native shapes: the marker-free default folds
very repetitive logs to a single line (for example `[3000 lines omitted:
3000 INFO]`) on chat bodies too, with no retrieval path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-15 01:52:50 +05:30
9321a415a6 fix(memory): scope TrafficLearner's SQLite reads to its user_id (#4850)
## Description

`TrafficLearner` saves its rows under its configured `user_id`, but its
two direct SQLite reads ignored `user_id`:

- `_hydrate_persisted_state` (startup) loaded every user's
`traffic_learner` rows into the dedup maps, then applied `LIMIT
dedup_window`. Bob's learner could bind a pattern hash to Alice's row
and later bump *her* evidence instead of creating his own row. Other
users' rows also used up his window.
- `_load_persisted_patterns_from_sqlite` (used by
`_collect_all_patterns` when writing learned context files) returned
every user's patterns, so Alice's learned text ended up in Bob's files.

Both queries now filter on `user_id = ?`. In hydration the filter runs
before the `LIMIT`. Only matters when several user IDs share one memory
database; a single-user setup (`user_id="default"`) is unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/memory/traffic_learner.py`: `WHERE user_id = ?` in both
reads. `_load_persisted_patterns_from_sqlite(db_path, user_id)` takes
the user explicitly (private helper; its only caller is updated).
- `tests/test_memory/test_traffic_learner.py`: new
`TestUserScopedPersistence`, which checks that hydration leaves another
user's row alone and collection excludes other users. Hand-built test
schemas gain the `user_id` column that the real `memories` table has,
and direct helper calls pass `"default"`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# new tests against origin/main (c21b0813)
$ pytest tests/test_memory/test_traffic_learner.py -k TestUserScopedPersistence
E   assert ['Alice-only ...ge workflow."] == ["Bob's package workflow."]
FAILED ...::TestUserScopedPersistence::test_hydration_ignores_other_users_rows
FAILED ...::TestUserScopedPersistence::test_collected_patterns_exclude_other_users
====================== 2 failed, 236 deselected in 2.76s =======================

# this branch
$ pytest tests/test_memory/test_traffic_learner.py
============================= 238 passed in 18.28s =============================
$ pytest tests/test_openai_responses_traffic_learner.py tests/test_learn/test_worktree_merge.py tests/test_learn/test_section_lifecycle.py tests/test_memory/test_learn_flag.py
============================== 47 passed in 4.81s ==============================
$ ruff check / ruff format --check (0.17.3): All checks passed!
$ mypy headroom/memory/traffic_learner.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.12. One SQLite `memory.db` holding a
`traffic_learner` row for `alice` and one for `bob`.
- Exact command / steps: `TrafficLearner(backend=<backend pointing at
that db>, user_id="bob")._collect_all_patterns()`.
- Observed result:
```text
origin/main:  bob collects: ['Alice-only procedure.', "Bob's preference."]
this branch:  bob collects: ["Bob's preference."]
```
- Not tested: a full proxy run with `--learn --memory` and several
users.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: see Description
- Kill switch / disable path: none needed
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this PR

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Rows written before this change already carry the correct `user_id` (the
save path always set it), so no migration is needed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4850"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-15 01:52:47 +05:30
491afe98b1 fix(proxy): reject a private x-headroom-base-url on chat and responses with 400 (#4857)
## Description

When the upstream guard rejected an `x-headroom-base-url` override (an
address on a private, loopback, link-local or otherwise non-public
network, or a host outside `HEADROOM_ALLOWED_BASE_URLS` in allowlist
mode), `_resolve_openai_upstream_base` logged a warning and returned
`None`. `/v1/chat/completions` and `/v1/responses` then sent the request
— the prompt and the client's `Authorization`, meant for the gateway the
client named — to the configured `OPENAI_API_URL` instead. Every other
route that reads the header (`headroom/providers/proxy_routes.py`:
Anthropic messages, Gemini, models, embeddings, audio, passthrough)
refuses with `400 Rejected unsafe upstream base URL`.

The OpenAI chat and responses handlers now refuse the same way.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update

## Changes Made

- `headroom/proxy/handlers/openai.py`: `_resolve_openai_upstream_base`
raises `HTTPException(400, "Rejected unsafe upstream base URL")` instead
of returning `None` when the guard rejects the override (malformed /
non-http(s) values are unchanged: still ignored).
- `docs/content/docs/security-model.mdx`: one behaviour for all routes
(the page previously documented the two different outcomes).
-
`tests/test_upstream_guard.py::test_openai_routes_refuse_a_private_override_instead_of_using_the_default[chat|responses]`:
real app, configured OpenAI target is a local recording server; asserts
400 and that the configured target is never contacted.

**Sibling PRs that touch the same code** (merge order note):
`fix/upstream-guard-off-loop` adds `_resolve_openai_upstream_base_async`
next to this function; whichever merges second must make the async
resolver raise the same 400 (that PR's rejection branch is the one place
to change). `fix/upstream-idna-pin-once` touches the call sites only.
Suggested order: idna-pin-once, off-loop, then this PR rebased.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# New test on origin/main (c21b0813)
E   AssertionError: {"secret":"internal-only"}
E   assert 200 == 400                (chat: answered by the configured target)
E   assert 200 == 400                (responses: answered by the configured target)
======================= 2 failed, 61 deselected in 5.64s =======================

# This branch
======================= 2 passed, 61 deselected in 5.81s =======================

# Area suites: test_upstream_guard, test_grok_shared_proxy_call_sites, test_proxy/test_openai_upstream_header,
# test_proxy_openai_cache_key_integration, test_upstream_credential_scoping, test_header_isolation,
# test_proxy_hardening, test_openai_codex_routing, test_proxy_loopback_gating
================= 429 passed, 10 warnings in 87.64s (0:01:27) ==================

$ ruff check / ruff format --check (0.17.3) on changed files
All checks passed!
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, `python -m headroom.cli proxy
--no-optimize` from this branch and from origin/main,
`OPENAI_TARGET_API_URL` pointing at a local server that records every
request it receives.
- Exact command / steps:
`curl http://127.0.0.1:$PORT/v1/chat/completions -H 'authorization:
Bearer client-gateway-key' -H 'x-headroom-base-url:
http://192.168.1.50:8000/v1' -d '{"model":"gpt-4o","messages":[...]}'`,
and the same for `/v1/responses`.
- Observed result:
  ```text
  == origin/main
  POST /v1/chat/completions -> HTTP 200 {"...":"from-configured-target"}
  POST /v1/responses        -> HTTP 200 {"...":"from-configured-target"}
  configured target received:
    /v1/chat/completions auth= Bearer client-gateway-key
    /v1/responses        auth= Bearer client-gateway-key
  == this branch
POST /v1/chat/completions -> HTTP 400 {"detail":"Rejected unsafe
upstream base URL"}
POST /v1/responses -> HTTP 400 {"detail":"Rejected unsafe upstream base
URL"}
  configured target received: (nothing)
  ```
- Not tested: Codex WebSocket responses path (it does not read this
header).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: **yes.** A client that sends
`x-headroom-base-url` pointing at a private/LAN/loopback address (or, in
allowlist mode, an unlisted host) to `/v1/chat/completions` or
`/v1/responses` now gets a 400 instead of having its request answered by
the configured OpenAI target. Such a request never reached the host the
client named, so the only working setups affected are ones that relied
on the silent fallback. A DNS lookup that fails or times out for a
public gateway is also a rejection, so it now surfaces as a 400 instead
of quietly sending the request to the default target.
- Kill switch / disable path: operators who route to an internal gateway
add it to `HEADROOM_ALLOWED_BASE_URLS` (existing setting); clients can
drop the header to use the configured target.
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Fix adapted from a proposal by HetCreep.

Follow-up, not in this PR: the OpenCode plugin
(`plugins/opencode/src/transport.ts`, `isRoutableUpstream`) only keeps
loopback upstreams out of the proxy, so an OpenCode provider on a LAN
address is still routed through Headroom, which now answers 400 for it
(previously the request went to the default OpenAI target). Sending
private-network upstreams direct needs a source change plus regenerating
the committed bundles (`headroom/providers/opencode/_dist/index.js`,
`hook-shim/handler.js`), and hostnames that resolve to LAN addresses
cannot be recognised without DNS, so it is left for a separate PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4857"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-15 01:52:42 +05:30
2f58877732 fix(proxy): check x-headroom-base-url off the event loop in chat and responses (#4856)
## Description

`handle_openai_chat` and `handle_openai_responses` validated
`x-headroom-base-url` with the synchronous `is_safe_upstream_url`, which
waits for the DNS lookup (up to `HEADROOM_UPSTREAM_RESOLVE_TIMEOUT_S`, 3
s by default) on the event-loop thread. Chat did this three times per
request and responses twice, so one slow-resolving host held up every
other in-flight request for several seconds. The other routes in
`headroom/providers/proxy_routes.py` already use
`is_safe_upstream_url_async`.

Both handlers now resolve the header once, through a new
`_resolve_openai_upstream_base_async` that uses the async guard (lookup
on a worker thread, pin scope opened on the request's task), and reuse
that result for the destination and the extra-header decision.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement

## Changes Made

- `headroom/proxy/handlers/openai.py`: header parsing split into
`_openai_base_url_candidate`; sync `_resolve_openai_upstream_base` (kept
for `_resolve_openai_upstream`) and new
`_resolve_openai_upstream_base_async` share it;
`_openai_upstream_from(request, custom)` picks the upstream from an
already-resolved override; chat and responses call the async resolver
once.
- `tests/test_proxy/test_openai_upstream_guard_off_loop.py`: asserts the
guard runs once and not on the event-loop thread, for chat and
responses.
- `tests/test_proxy_openai_cache_key_integration.py`: its stub now
patches the async resolver the handler calls.

**Sibling PRs that touch the same code** (merge order note):
`fix/upstream-idna-pin-once` adds the identical `_openai_upstream_from`
and resolve-once hunks (this PR is a superset of that part);
`fix/openai-base-url-reject` changes the rejection branch of
`_resolve_openai_upstream_base`, which after this PR must be mirrored in
`_resolve_openai_upstream_base_async`. Suggested order: idna-pin-once,
then this PR, then reject (rebasing reject means raising the 400 from
the shared rejection path in both resolvers).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# New test on origin/main (c21b0813)
E   AssertionError: the guard ran on the event-loop thread
E   assert [True, True, True] == [False]      (chat)
E   assert [True, True] == [False]            (responses)
============================== 2 failed in 0.72s ===============================

# This branch: new test + cache-key integration
============================== 7 passed in 7.30s ===============================

# Area suites: test_upstream_guard, test_grok_shared_proxy_call_sites, test_proxy/test_openai_upstream_header,
# test_proxy_openai_cache_key_integration, test_upstream_credential_scoping, test_header_isolation,
# test_proxy_hardening, test_openai_codex_routing, test_proxy/test_openai_upstream_guard_off_loop
======================= 282 passed, 5 warnings in 56.32s =======================

$ ruff check / ruff format --check (0.17.3) on changed files
All checks passed!
3 files already formatted
$ mypy headroom/proxy/handlers/openai.py --ignore-missing-imports
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, full app from `create_app` driven
through `httpx.ASGITransport` on one event loop; `socket.getaddrinfo`
replaced so `slow.example` takes 2 s to answer (everything else real).
- Exact command / steps: start a `/v1/chat/completions` request with
`x-headroom-base-url: https://slow.example/v1`, then 0.2 s later send
`GET /livez` and time it.
- Observed result:
  ```text
  == origin/main
/livez sent 0.2s after a chat request naming a 2s-to-resolve host: HTTP
200 after 6.03s (0.20s if the loop is free)
  == this branch
/livez sent 0.2s after a chat request naming a 2s-to-resolve host: HTTP
200 after 0.20s (0.20s if the loop is free)
  ```
- Not tested: under real slow DNS.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: no change in which upstream is
chosen; the lookup moves to a worker thread and happens once per
request.
- Kill switch / disable path: none needed
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Fix adapted from a proposal by HetCreep.

Follow-up, not in this PR: `select_passthrough_base_url`
(`headroom/providers/proxy_targets.py`, Azure `api-key` branch) and
`headroom/providers/registry.py` still call the sync guard; the
passthrough route already checks the same header with the async guard
first, so the sync call there is a second lookup on the loop.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4856"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Reported by @JoaoMarcos44. Thank you for your contribution! 🩵

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-15 01:52:37 +05:30