mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(router): gate block-form tool output against cached lossy results (#4504)
## Description
The #1307 rule says a lossy transform that emitted no CCR retrieve
marker must not replace tool output. The string `role: "tool"` path
enforces it and keys its cache entries with `gated=True`
(`content_router.py:6740-6746`), so a gated entry and an ungated entry
for the same bytes never share a slot. The two block-form paths in
`_process_content_blocks` don't, which is part (b) of #4477:
- The `tool_result` block path passes `enforce_reversibility=True` to
`_compress_block_content` but builds its key without `gated=True`. That
is the key the ungated text-block path uses for the same bytes. When the
same text was compressed first as an ungated text block (for example
assistant text with `compress_assistant_text_blocks`), its unmarked
lossy result is in the cache, and the `tool_result` lookup returns it
before the gate runs. In the other direction, a gate skip for a
`tool_result` lands in the ungated namespace and suppresses compression
of that text elsewhere.
- A `role: "tool"` or `"function"` message whose content is a list of
text blocks goes through the text-block branch, which never enforces the
gate. The same text as a plain string is gated.
This PR keys `tool_result` blocks with `gated=True` and makes the
text-block branch gate `tool` and `function` roles, with the same key
partition as the string path.
Part (a) of the issue (whether per-request options such as
`kompress_model="disabled"` belong in the key) is a design question the
issue leaves to maintainers, so it is not changed here.
Related to #4477
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/content_router.py`:
- The `tool_result` branch calls `content_cache_key(..., gated=True)`.
- The text-block branch sets `enforce_reversibility = role in ("tool",
"function")`, keys with `gated=enforce_reversibility` and passes it to
`_compress_block_content`. Assistant, user and system text blocks keep
the ungated key and current behavior.
- The comment at the gate said `tool_result` blocks are "never cached",
which is no longer true. It now says gated callers key into the gated
namespace.
- `tests/test_content_router_tool_role_reversibility.py`: adds
block-form cases through `ContentRouter.apply()`, using the file's
existing stub of `compress()`:
- list-of-text `tool` and `function` messages keep their text when the
result is unmarked and lossy, and accept a result that carries a CCR
marker;
- a `tool_result` is not served an ungated cache entry for the same
text;
- a gate skip for a `tool_result` does not stop the same text
compressing as an assistant text block.
- `tests/test_transforms_content_router.py`:
`test_tool_role_text_blocks_compressed_by_default` used a mock whose
`strategy_used` is a `SimpleNamespace`. That is unhashable, so it raises
once the tool-role path reaches the gate. It is also an unmarked lossy
result, which this PR now keeps out of tool output. The test now returns
a real `CompressionStrategy.TEXT` result with a CCR marker and still
checks that tool-role text blocks are compressed.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): 2 errors, the same 2 as on
`main` (`headroom/memory/adapters/embedders.py:389`,
`headroom/integrations/langchain/chat_model.py:380`), none in
`content_router.py`.
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# new tests against main's content_router.py
$ pytest tests/test_content_router_tool_role_reversibility.py -q
4 failed, 12 passed
# this branch
$ pytest tests/test_content_router_tool_role_reversibility.py -q
16 passed
$ pytest tests/test_content_router_tool_role_reversibility.py tests/test_transforms_content_router.py \
tests/test_router_result_cache.py tests/test_no_ccr_lossy.py \
tests/test_content_router_concurrency.py tests/test_cross_turn_dedup.py -q
153 passed, 2 skipped
$ ruff check . && ruff format --check .
All checks passed!
1834 files already formatted
# full suite (Linux, Python 3.12, --timeout=120)
$ pytest tests -q --timeout=120
7 failed, 17051 passed, 596 skipped
```
None of the 7 failures comes from this change.
`test_mcp_reconcile::test_ordinary_install_does_not_adopt_serena`,
`test_wrap_sighup_proxy_leak::test_sighup_on_launch_tool_releases_the_proxy`
and both `test_cc_switch_reconciler` tests fail the same way with
`main`'s `content_router.py` in this environment.
`test_offline_egress_chokepoint`, `test_stateless_no_disk` and
`test_direct_chat_ccr_resolution` pass when run on their own on this
branch.
## Real Behavior Proof
- Environment: Linux x86_64 (Amazon Linux 2023), Python 3.12, this
branch on top of `main` at `17d38e56`, with the published
`headroom._core` binary and the project's `html` extra (`trafilatura`
2.3.1). Nothing is stubbed: the real `ContentRouter` picks the real HTML
strategy, and trafilatura's extraction is the unmarked lossy result
(7,559 to 5,707 chars, no CCR marker).
- Exact command / steps: call
`ContentRouter(ContentRouterConfig()).apply(...)` (`protect_recent=0`,
`protect_analysis_context=False`) on a 7.5 KB HTML build report: (1a) as
an assistant text block with `compress_assistant_text_blocks=True`, then
(1b) as a `tool_result` block on the same router; (1c) as a
`tool_result` on a fresh router (control); (2) as list-of-text `role:
"tool"` and `role: "function"` messages; (3) as a `role: "tool"` string
(control). Run once with `main`'s `content_router.py` and once with this
branch's.
- Observed result: on `main`, 1b and both case-2 messages forward the
extracted text with no CCR marker in place of the tool output, while the
1c and 3 controls keep it. On this branch every tool-output shape is
kept byte for byte, and the assistant text block (1a) still compresses
as before.
```text
# main (17d38e56)
1a assistant text block (ungated): REPLACED 7559 -> 5707 chars, CCR marker: False ['router:text_block:html']
1b tool_result, same router, same bytes: REPLACED 7559 -> 5707 chars, CCR marker: False ['router:tool_result:html']
1c tool_result, fresh router (control): VERBATIM (7559 chars) ['router:noop']
2 role=tool list-of-text: REPLACED 7559 -> 5707 chars, CCR marker: False ['router:text_block:html']
2 role=function list-of-text: REPLACED 7559 -> 5707 chars, CCR marker: False ['router:text_block:html']
3 role=tool string (control): VERBATIM (7559 chars) ['router:noop']
# this branch
1a assistant text block (ungated): REPLACED 7559 -> 5707 chars, CCR marker: False ['router:text_block:html']
1b tool_result, same router, same bytes: VERBATIM (7559 chars) ['router:noop']
1c tool_result, fresh router (control): VERBATIM (7559 chars) ['router:noop']
2 role=tool list-of-text: VERBATIM (7559 chars) ['router:noop']
2 role=function list-of-text: VERBATIM (7559 chars) ['router:noop']
3 role=tool string (control): VERBATIM (7559 chars) ['router:noop']
```
- Not tested: Kompress (the ONNX model is not available here), so the
HTML strategy stands in for the lossy path; a full proxy process taking
HTTP traffic (the run calls `ContentRouter.apply()`, which the proxy
pipeline calls); the disk tier (`ResultStore`), which uses the same key;
Windows and macOS.
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, for block-form tool output only:
an unmarked lossy result no longer replaces a `tool_result` block or a
list-of-text `tool`/`function` message, matching the string path. With
`ccr_inject_marker=False` the gate is off, as before. `tool_result`
cache keys change, so existing entries for that content miss once and
are recomputed.
- Kill switch / disable path: `ccr_inject_marker=False` (no-CCR mode)
turns the gate off for all paths, as today.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- No documentation change: the #1307 rule is internal and already
describes tool output in general.
- The string path gates `role == "tool"` only. Legacy `role: "function"`
string content is still ungated there; I left that alone to keep this PR
to what #4477 reports, and can follow up if you want both forms to
match.
- Part (a) of #4477 is untouched (see Description).
---
🤖 Opened with [Superhuman](https://github.com/gaurav0107/superhuman), an
open-source contribution agent.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
5daaf33637
commit
e4a60ed72f
@@ -8109,11 +8109,14 @@ class ContentRouter(Transform):
|
||||
route_counts["already_compressed"] += 1
|
||||
continue
|
||||
|
||||
# Two-tier compression cache → shared helper
|
||||
# Two-tier compression cache → shared helper. Tool output is
|
||||
# gated (#1307), so it keys into the gated namespace like the
|
||||
# string path: an ungated entry for the same bytes must not
|
||||
# answer this lookup before the gate runs (#4477).
|
||||
compressed_content, was_compressed = self._compress_block_content(
|
||||
content=tool_text,
|
||||
content_key=self._content_cache_key(
|
||||
tool_text, bias=bias, context=block_context
|
||||
tool_text, bias=bias, context=block_context, gated=True
|
||||
),
|
||||
context=block_context,
|
||||
bias=bias,
|
||||
@@ -8204,11 +8207,15 @@ class ContentRouter(Transform):
|
||||
route_counts["already_compressed"] += 1
|
||||
continue
|
||||
|
||||
# Two-tier compression cache → shared helper
|
||||
# Two-tier compression cache → shared helper. A role:"tool" /
|
||||
# "function" message in block-list form is tool output, the
|
||||
# same ground truth as its string form, so it gets the same
|
||||
# #1307 gate and the gated cache namespace (#4477).
|
||||
enforce_reversibility = role in ("tool", "function")
|
||||
compressed_content, _was_compressed = self._compress_block_content(
|
||||
content=text_content,
|
||||
content_key=self._content_cache_key(
|
||||
text_content, bias=1.0, context=context
|
||||
text_content, bias=1.0, context=context, gated=enforce_reversibility
|
||||
),
|
||||
context=context,
|
||||
bias=1.0,
|
||||
@@ -8219,6 +8226,7 @@ class ContentRouter(Transform):
|
||||
compressed_details=compressed_details,
|
||||
strategy_label="text_block",
|
||||
details_prefix="text",
|
||||
enforce_reversibility=enforce_reversibility,
|
||||
)
|
||||
if compressed_content is not None:
|
||||
new_blocks.append({**block, "text": compressed_content})
|
||||
@@ -8398,8 +8406,9 @@ class ContentRouter(Transform):
|
||||
# (kompress/text/code/html) that emitted no CCR retrieve marker is
|
||||
# unrecoverable, so the agent would act on a fabricated summary
|
||||
# (#1307). The string/`role=="tool"` path guards this; mirror it
|
||||
# here for tool_result blocks (never cached, so the Tier-2 path
|
||||
# above can't serve a poisoned entry).
|
||||
# here for tool output in block form. Callers that enforce the gate
|
||||
# key with ``gated=True``, so the Tier-2 path above never serves
|
||||
# them an ungated entry for the same bytes.
|
||||
#
|
||||
# EXCEPTION: no-CCR mode (config.ccr_inject_marker=False). Here the
|
||||
# operator has *deliberately* disabled retrieval markers — recovery
|
||||
|
||||
@@ -140,3 +140,102 @@ def test_tool_role_structured_strategies_accepted(monkeypatch, strategy) -> None
|
||||
"""Structured strategies are lossless/self-marking -> not gated, compressed kept."""
|
||||
result = _run(monkeypatch, _tool_msg(GREP_OUTPUT), strategy, LOSSY_SUMMARY)
|
||||
assert result.messages[0]["content"] == LOSSY_SUMMARY
|
||||
|
||||
|
||||
# --- Block-form tool output (#4477) -----------------------------------------
|
||||
#
|
||||
# Tool output also arrives as content blocks: an Anthropic ``tool_result`` block,
|
||||
# or a role="tool"/"function" message whose content is a list of text blocks.
|
||||
# Both shapes must get the same #1307 gate as the string path, and must read and
|
||||
# write the gated cache namespace so an ungated entry for byte-identical text
|
||||
# (an assistant message, a user text block) is never served as tool output.
|
||||
|
||||
|
||||
def _text_blocks_msg(role: str, content: str) -> dict:
|
||||
msg: dict = {"role": role, "content": [{"type": "text", "text": content}]}
|
||||
if role == "tool":
|
||||
msg["tool_call_id"] = "call_bash_1"
|
||||
else:
|
||||
msg["name"] = "qualification_tool"
|
||||
return msg
|
||||
|
||||
|
||||
def _tool_result_msg(content: str) -> dict:
|
||||
return {
|
||||
"role": "user",
|
||||
"content": [{"type": "tool_result", "tool_use_id": "toolu_bash_1", "content": content}],
|
||||
}
|
||||
|
||||
|
||||
def _apply(router: ContentRouter, message: dict, **kwargs):
|
||||
return router.apply(
|
||||
[message], _WordTokenizer(), protect_recent=0, protect_analysis_context=False, **kwargs
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("role", ["tool", "function"])
|
||||
def test_tool_role_text_blocks_lossy_unmarked_kept_verbatim(monkeypatch, role) -> None:
|
||||
"""List-of-text tool output gets the gate the string form already has."""
|
||||
result = _run(
|
||||
monkeypatch,
|
||||
_text_blocks_msg(role, GREP_OUTPUT),
|
||||
CompressionStrategy.KOMPRESS,
|
||||
LOSSY_SUMMARY,
|
||||
)
|
||||
assert result.messages[0]["content"] == [{"type": "text", "text": GREP_OUTPUT}]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("role", ["tool", "function"])
|
||||
def test_tool_role_text_blocks_lossy_with_ccr_marker_accepted(monkeypatch, role) -> None:
|
||||
"""A recoverable (marked) result still compresses list-of-text tool output."""
|
||||
result = _run(
|
||||
monkeypatch,
|
||||
_text_blocks_msg(role, GREP_OUTPUT),
|
||||
CompressionStrategy.KOMPRESS,
|
||||
CCR_MARKER_SUMMARY,
|
||||
)
|
||||
assert result.messages[0]["content"] == [{"type": "text", "text": CCR_MARKER_SUMMARY}]
|
||||
|
||||
|
||||
def test_tool_result_block_not_served_an_ungated_cache_entry(monkeypatch) -> None:
|
||||
"""An unmarked lossy result cached for ungated text must not reach a tool_result.
|
||||
|
||||
The assistant text-block path is not gated (assistant prose is not tool ground
|
||||
truth), so it caches the lossy summary. A later tool_result carrying the same
|
||||
bytes must miss that entry and be gated, not take the cache hit.
|
||||
"""
|
||||
router = ContentRouter()
|
||||
monkeypatch.setattr(
|
||||
router,
|
||||
"compress",
|
||||
lambda *a, **k: _force_result(CompressionStrategy.KOMPRESS, LOSSY_SUMMARY),
|
||||
)
|
||||
first = _apply(
|
||||
router,
|
||||
{"role": "assistant", "content": [{"type": "text", "text": GREP_OUTPUT}]},
|
||||
compress_assistant_text_blocks=True,
|
||||
)
|
||||
assert first.messages[0]["content"][0]["text"] == LOSSY_SUMMARY # ungated: compressed
|
||||
|
||||
second = _apply(router, _tool_result_msg(GREP_OUTPUT))
|
||||
assert second.messages[0]["content"][0]["content"] == GREP_OUTPUT
|
||||
|
||||
|
||||
def test_gated_tool_result_skip_does_not_suppress_ungated_text(monkeypatch) -> None:
|
||||
"""The reverse direction: a gate skip recorded for tool output stays in the
|
||||
gated namespace, so ungated text with the same bytes still compresses."""
|
||||
router = ContentRouter()
|
||||
monkeypatch.setattr(
|
||||
router,
|
||||
"compress",
|
||||
lambda *a, **k: _force_result(CompressionStrategy.KOMPRESS, LOSSY_SUMMARY),
|
||||
)
|
||||
first = _apply(router, _tool_result_msg(GREP_OUTPUT))
|
||||
assert first.messages[0]["content"][0]["content"] == GREP_OUTPUT # gated: kept
|
||||
|
||||
second = _apply(
|
||||
router,
|
||||
{"role": "assistant", "content": [{"type": "text", "text": GREP_OUTPUT}]},
|
||||
compress_assistant_text_blocks=True,
|
||||
)
|
||||
assert second.messages[0]["content"][0]["text"] == LOSSY_SUMMARY
|
||||
|
||||
@@ -1503,8 +1503,18 @@ def test_system_text_blocks_skipped_when_skip_system_true(
|
||||
def test_tool_role_text_blocks_compressed_by_default(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
router = _make_router_with_mock_compress(monkeypatch)
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
long_text = "T" * 1000
|
||||
recoverable = "T" * 100 + " [compressed] <<ccr:9f3a21>>"
|
||||
monkeypatch.setattr(
|
||||
router,
|
||||
"compress",
|
||||
lambda *a, **k: SimpleNamespace(
|
||||
compressed=recoverable,
|
||||
compression_ratio=0.5,
|
||||
strategy_used=CompressionStrategy.TEXT,
|
||||
),
|
||||
)
|
||||
msg = {"role": "tool", "content": [{"type": "text", "text": long_text}]}
|
||||
result = router._process_content_blocks(
|
||||
msg,
|
||||
@@ -1514,8 +1524,9 @@ def test_tool_role_text_blocks_compressed_by_default(
|
||||
set(),
|
||||
set(),
|
||||
)
|
||||
# tool role ≈ tool output — compress freely
|
||||
assert "[compressed]" in result["content"][0]["text"]
|
||||
# tool role ≈ tool output — not protected like user/system text; a
|
||||
# recoverable result is accepted (an unmarked lossy one is gated, #1307).
|
||||
assert result["content"][0]["text"] == recoverable
|
||||
|
||||
|
||||
def test_unknown_role_text_blocks_skipped_for_safety(
|
||||
|
||||
Reference in New Issue
Block a user