mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(router): validate lossless provider output on the excluded-tool path (#4486)
Fixes #4475
## Description
`ContentRouter._lossless_compact_excluded` (protected / excluded tool
output) returned the lossless provider's result unvalidated, although
`lossless_provider.py` says a malformed or empty result is ignored and
`kind` is sanitised. The general path (`_lossless_provider_result`)
already enforced this. On the excluded path a 3-tuple made `folded, kind
= compacted` raise `ValueError` out of `ContentRouter.apply`, an empty
fold replaced the tool result with `""`, and any `kind` string went into
`transforms_applied`.
This PR moves the contract check into one helper,
`_validate_provider_output`, and uses it on both paths.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/content_router.py`: new static
`_validate_provider_output(supplied)` holding the exact checks that were
inline in `_lossless_provider_result` (None, shape, str members,
non-empty, `kind` fullmatch with fallback label).
`_lossless_provider_result` now calls it (behaviour unchanged, memo
kept). `_lossless_compact_excluded` calls it on the provider's return
value.
- Unchanged semantics on the excluded path: a provider that raises still
falls back to the built-in folds; a provider result of `None` (now also
any ignored result) leaves the content unchanged and does not run the
built-in folds.
- `tests/test_lossless_excluded_compaction.py`: four tests through
`ContentRouter.apply` on an excluded tool (`Grep`): valid 2-tuple
control, 3-tuple, empty fold, hostile kind. The file's autouse fixture
resets the provider after each test.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Review round 1
Devin flagged that the Real Behavior Proof lacked runtime evidence
(`tests/test_lossless_excluded_compaction.py:302-303`). No code change:
the proof section now shows a real proxy run against a local stub
upstream, before and after, below. The commit is unchanged.
### Test Output
Fail before (source change reverted, new tests kept), `pytest -q
tests/test_lossless_excluded_compaction.py`:
```text
FAILED tests/test_lossless_excluded_compaction.py::test_excluded_three_tuple_is_ignored_not_fatal
FAILED tests/test_lossless_excluded_compaction.py::test_excluded_empty_fold_leaves_content_unchanged
FAILED tests/test_lossless_excluded_compaction.py::test_excluded_hostile_kind_is_replaced
======================== 3 failed, 33 passed in 20.53s =========================
```
Pass after:
```text
tests/test_lossless_excluded_compaction.py ............................. [ 80%]
....... [100%]
============================== 36 passed in 8.46s ==============================
```
Neighbours (every test file mentioning `lossless_provider`, `_lossless_`
or `set_lossless`, plus `tests/test_transforms/test_content_router.py`,
including the existing `test_provider_exception_falls_back_to_builtin`
and `test_registered_provider_is_authoritative`):
```text
============= 532 passed, 2 skipped, 1 warning in 79.36s (0:01:19) =============
```
Ruff:
```text
$ ruff check headroom/transforms/content_router.py tests/test_lossless_excluded_compaction.py
All checks passed!
$ ruff format --check headroom/transforms/content_router.py tests/test_lossless_excluded_compaction.py
2 files already formatted
```
## Real Behavior Proof
- Environment: real proxy `python -m headroom.proxy.server --host
127.0.0.1 --port 18961 --no-cache --no-rate-limit` (Python 3.12.15, WSL2
Ubuntu, Windows 11) at upstream main 3fef15d3 plus this commit,
`ANTHROPIC_TARGET_API_URL` pointing at a local stub upstream on
127.0.0.1:18962 that records what it receives; fake key `sk-ant-test`,
no provider API contacted. A proof-only extension registered through the
real `headroom.proxy_extension` entry point calls
`set_lossless_provider` at proxy start.
- Exact command / steps: start the stub, start the proxy with
`HEADROOM_PROXY_EXTENSIONS=lossless_ext
PROOF_PROVIDER_MODE=<tuple3|empty|hostile|good>`, then one non-streaming
`POST /v1/messages` whose last message is a `tool_result` of a `Grep`
tool_use (40 lines `src/app.py:N: handler_N()`); run once with this PR's
source change reverted (`git apply -R`) and once at PR head.
- Observed result: stub upstream tool_result, before vs after. 3-tuple
provider: before the proxy logs `Transform content_router failed;
aborting compression pipeline` / `ValueError` (the whole compression
pipeline is skipped for that request), after no error and the
tool_result arrives verbatim; empty-fold provider: before the stub
receives `' '` (the grep output blanked, 5 chars logged), after it
receives the original 1,264-char tool_result verbatim; hostile kind:
content applied in both (unchanged by design); valid 2-tuple control:
folded content (` | `-joined) reaches the stub in both,
`route_counts={'excluded_tool_lossless': 1}`.
- Not tested: the OpenAI Responses handler (`proxy/handlers/openai.py`,
`_lossless_compact_excluded` caller) end to end; the hostile-kind label
itself over the wire (it only lands in `transforms_applied`, covered by
the unit test); streaming; a real provider API; the optional verifier on
the excluded path (not consulted before or after this PR, out of scope);
`mypy`.
Raw lines (stub upstream receives; proxy log excerpts):
```text
BEFORE (change reverted)
3-tuple : proxy log: ERROR headroom.transforms.pipeline - Transform content_router failed; aborting compression pipeline
ValueError ... content_router.py:7684 in _process_content_blocks
proxy log: WARNING Optimization failed: ValueError ...; stub receives the original tool_result (client still HTTP 200)
empty : PERF tok_before=465 tok_after=26 tok_saved=439 stub: tool_result=' '
AFTER (PR head)
3-tuple : no ERROR/Traceback in proxy log; stub: tool_result == original (1,264 chars, verified equal)
empty : no ERROR/Traceback; PERF not blanked; stub: tool_result == original (1,264 chars, verified equal)
good : route_counts={'content_blocks': 1, 'excluded_tool_lossless': 1}; stub: 'src/app.py:1: handler_1() | src/app.py:2: handler_2() | ...'
```
Scripts and raw outputs (WSL):
`/var/tmp/cb-pub/proof/fix-lossless-excluded-provider-contract/`
(`run.sh`, `stub.py`, `client.py`, `ext/`, `before_*.out`,
`after_*.out`, `src.patch`).
## Runtime Rollout Safety
- Rollout-managed feature(s): none (lossless provider seam is an
operator-installed extension point, no rollout flag involved)
- Minimum rollout channel: n/a
- Stable/default behavior changed: no for the default (no provider
registered); with a provider registered, malformed or empty results on
excluded tool output are now ignored instead of raising or blanking the
result
- Kill switch / disable path: `set_lossless_provider(None)`
- Unsafe override required: no
- Qualification impact: none expected
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- The issue suggested calling `self._lossless_provider_result(content)`
directly. That helper catches provider exceptions and returns `None`,
which would make the excluded path stop falling back to the built-in
folds when a provider raises
(`test_provider_exception_falls_back_to_builtin` pins that). So the
shared piece is the validation, factored out as
`_validate_provider_output`; the excluded path keeps its own try/except
and has no memo (it had none before).
- Documentation: no docs change; the behaviour now matches what
`lossless_provider.py` already documents.
Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests above were run on my machine against a clone of the repository.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
JD Davis
parent
99cdaaf1c7
commit
9f068a6c13
@@ -3778,6 +3778,31 @@ class ContentRouter(Transform):
|
||||
|
||||
return cand, f"lossless_{kind}"
|
||||
|
||||
@staticmethod
|
||||
def _validate_provider_output(supplied: Any) -> tuple[str, str] | None:
|
||||
"""Contract check shared by the general and excluded-tool provider paths.
|
||||
|
||||
Returns ``(candidate, safe_kind)`` or ``None`` ("ignore"). Never raises.
|
||||
"""
|
||||
if supplied is None:
|
||||
return None
|
||||
if not isinstance(supplied, tuple | list) or len(supplied) != 2:
|
||||
logger.debug("lossless provider returned a malformed result; ignoring")
|
||||
return None
|
||||
if not isinstance(supplied[0], str) or not isinstance(supplied[1], str):
|
||||
logger.debug("lossless provider returned non-str members; ignoring")
|
||||
return None
|
||||
# An empty / whitespace-only "fold" wins every length comparison and
|
||||
# would silently delete the block's content. That is not lossless.
|
||||
if not supplied[0].strip():
|
||||
logger.debug("lossless provider returned an empty result; ignoring")
|
||||
return None
|
||||
# fullmatch, not match: Python's `$` also matches just BEFORE a
|
||||
# trailing newline, so `re.match` would let "log\n" through and
|
||||
# put a newline into a metric label.
|
||||
kind = supplied[1] if _PROVIDER_KIND_RE.fullmatch(supplied[1]) else _PROVIDER_KIND_FALLBACK
|
||||
return (supplied[0], kind)
|
||||
|
||||
def _lossless_provider_result(self, content: str) -> tuple[str, str] | None:
|
||||
"""Validated + memoized ``(candidate, safe_kind)`` from the provider.
|
||||
|
||||
@@ -3808,29 +3833,7 @@ class ContentRouter(Transform):
|
||||
|
||||
result: tuple[str, str] | None = None
|
||||
try:
|
||||
supplied = provider(content)
|
||||
if supplied is None:
|
||||
result = None
|
||||
# Validate defensively *inside* the try: unpacking a 3-tuple / a bare
|
||||
# string / a non-sequence raises, and that exception used to escape
|
||||
# all the way out of the request.
|
||||
elif not isinstance(supplied, tuple | list) or len(supplied) != 2:
|
||||
logger.debug("lossless provider returned a malformed result; ignoring")
|
||||
elif not isinstance(supplied[0], str) or not isinstance(supplied[1], str):
|
||||
logger.debug("lossless provider returned non-str members; ignoring")
|
||||
# An empty / whitespace-only "fold" wins every length comparison and
|
||||
# would silently delete the block's content. That is not lossless.
|
||||
elif not supplied[0].strip():
|
||||
logger.debug("lossless provider returned an empty result; ignoring")
|
||||
else:
|
||||
raw_kind = supplied[1]
|
||||
# fullmatch, not match: Python's `$` also matches just BEFORE a
|
||||
# trailing newline, so `re.match` would let "log\n" through and
|
||||
# put a newline into a metric label.
|
||||
kind = (
|
||||
raw_kind if _PROVIDER_KIND_RE.fullmatch(raw_kind) else _PROVIDER_KIND_FALLBACK
|
||||
)
|
||||
result = (supplied[0], kind)
|
||||
result = self._validate_provider_output(provider(content))
|
||||
except Exception: # noqa: BLE001 - a broken provider must not break routing
|
||||
logger.debug("lossless provider failed in _lossless_first", exc_info=True)
|
||||
result = None
|
||||
@@ -7337,7 +7340,7 @@ class ContentRouter(Transform):
|
||||
try:
|
||||
# A registered provider is authoritative for excluded-tool
|
||||
# compaction; fall back to the built-in folds only if it raises.
|
||||
return provider(content)
|
||||
return self._validate_provider_output(provider(content))
|
||||
except Exception: # noqa: BLE001
|
||||
logger.debug(
|
||||
"lossless provider failed; using built-in compaction",
|
||||
|
||||
@@ -281,3 +281,52 @@ def test_provider_exception_falls_back_to_builtin():
|
||||
# Falls back to the built-in fold rather than crashing or passing through raw.
|
||||
out, kind = _compact(GREP)
|
||||
assert kind == "search" and search_unheading(out) == GREP
|
||||
|
||||
|
||||
# --- excluded path enforces the same provider contract as the general path ----
|
||||
|
||||
|
||||
def _run_excluded_anthropic(tokenizer):
|
||||
router = ContentRouter(ContentRouterConfig())
|
||||
messages = [
|
||||
{"role": "user", "content": "find it"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": [{"type": "tool_use", "id": "t1", "name": "Grep", "input": {}}],
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": GREP}],
|
||||
},
|
||||
]
|
||||
result = router.apply(messages, tokenizer, compress_user_messages=True)
|
||||
return result.messages[2]["content"][0]["content"], result.transforms_applied
|
||||
|
||||
|
||||
def test_excluded_valid_two_tuple_still_applied(tokenizer):
|
||||
set_lossless_provider(lambda content: ("compact", "myfold"))
|
||||
out, applied = _run_excluded_anthropic(tokenizer)
|
||||
assert out == "compact"
|
||||
assert "router:excluded:lossless_myfold" in applied
|
||||
|
||||
|
||||
def test_excluded_three_tuple_is_ignored_not_fatal(tokenizer):
|
||||
set_lossless_provider(lambda content: ("compact", "k", "extra"))
|
||||
out, applied = _run_excluded_anthropic(tokenizer)
|
||||
assert out == GREP
|
||||
assert not any(t.startswith("router:excluded:lossless_") for t in applied)
|
||||
|
||||
|
||||
def test_excluded_empty_fold_leaves_content_unchanged(tokenizer):
|
||||
set_lossless_provider(lambda content: (" ", "k"))
|
||||
out, applied = _run_excluded_anthropic(tokenizer)
|
||||
assert out == GREP
|
||||
assert not any(t.startswith("router:excluded:lossless_") for t in applied)
|
||||
|
||||
|
||||
def test_excluded_hostile_kind_is_replaced(tokenizer):
|
||||
set_lossless_provider(lambda content: ("compact", "BAD LABEL{x=1}" + chr(10)))
|
||||
out, applied = _run_excluded_anthropic(tokenizer)
|
||||
assert out == "compact"
|
||||
labels = [t for t in applied if t.startswith("router:excluded:lossless_")]
|
||||
assert labels == ["router:excluded:lossless_provider"]
|
||||
|
||||
Reference in New Issue
Block a user