mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
Current status (2026-10-24 20:50 UTC): exact head `2333a5ccf7700f86f8767d1c5d02fbfb903c966c` is conflict-free and [approved by JerrettDavis](https://github.com/headroomlabs-ai/headroom/pull/4334#pullrequestreview-5639834650). All seven held workflows are now successful: [CI](https://github.com/headroomlabs-ai/headroom/actions/runs/40128152028) has 17 successful jobs and five skips; [Security](https://github.com/headroomlabs-ai/headroom/actions/runs/40128151955) has 12 successful jobs. Both [Wrap E2E](https://github.com/headroomlabs-ai/headroom/actions/runs/40128151965/job/120974772335) and CI's Docker-native job actually execute the new npm hook: two tests pass, zero fail or skip. Logs check out synthetic merge `91669e2` (this head into `3fef15d3`). The maintainer independently reports 146 tests passed with ten optional skips, including the nine new regressions, plus Ruff lint/formatting. These hosted and maintainer results supplement, rather than replace or relabel, the author's limited local checks. Security passes upstream policy, including three excepted bundled npm advisories until 2026-11-15; this is not a zero-findings claim. Live-provider billing/cache behavior remains unqualified. The prior approval-pending statement and local Docker limitation in the [author response](https://github.com/headroomlabs-ai/headroom/pull/4334#issuecomment-6388924792) describe the earlier observation, now superseded by these hosted results. ### Previous-head qualification (historical) Current status (2026-10-24 19:54 UTC): exact head `a618561542a8bb48fdb29b07c66c3c6647b7a690` is conflict-free and [approved by JerrettDavis](https://github.com/headroomlabs-ai/headroom/pull/4334#pullrequestreview-5639593849); the maintainer explicitly cleared the earlier dependency qualification hold. All seven previously held workflows completed successfully, including [CI](https://github.com/headroomlabs-ai/headroom/actions/runs/40116172110) (17 successful jobs, five skipped), [Security](https://github.com/headroomlabs-ai/headroom/actions/runs/40116172235) (12 successful jobs), Rust and Wrap E2E. Logs check out synthetic merge `2ff995d67ef4bb52590c1c0ca95514c85e9693ab` (this head into `3fef15d3`). Maintainer qualification reports 137 tests passed with ten optional skips plus Ruff/whitespace; these are their results, not another author-run suite. Security passes upstream policy, not a zero-findings claim: E2E npm retains three excepted bundled advisories under upstream's existing policy until 2026-11-15. No additional exception was introduced here. Live-provider behavior remains unqualified. Earlier dated results below are historical. ## Description Fixes #4332. Replaying a cached compression for an OpenAI `role: tool` message currently converts list-form content to a string, discarding its client-supplied `cache_control` and other block metadata. A single text block now keeps its shape and metadata; only its text is replaced in the existing deep copy. For multiple blocks or non-text/malformed block shapes, replay leaves the message unchanged, before a cache lookup is counted. The cached joined string does not contain enough information to safely reconstruct block boundaries or preserve media. String-form OpenAI and Anthropic behavior are unchanged. This does not duplicate #4333, which explicitly excludes the token-mode lost-breakpoint issue. ## Type of Change - [x] Bug fix ## Changes Made - Preserve the deep-copied single-text OpenAI content block and replace text only. - Leave ambiguous lists unchanged before cache hit accounting. - Share the eligibility guard across cache replay, the rewrite helper and the real interceptor path before callbacks, disclosure state and savings accounting. - Add direct-helper/cache/interceptor regressions for unsupported shapes, metadata isolation and progressive-disclosure isolation. - Earlier compatible dependency updates are now incorporated in upstream main; the final PR delta keeps the cache fix and regressions, with dependency constraints and locks unchanged from main; the tools manifest now adds only a test script. ## Testing - [x] Focused regression tests pass. - [x] Lint / formatting and changed-module type checks passed at d74aa5c; runtime/test code is unchanged bya00f387. - [x] Independent review performed. - [x] Current2333a5cchosted CI/native and Docker regression-hook checks passed; live-provider qualification is not claimed. - [x] Current2333a5ccSecurity gates pass under upstream policy, with existing exceptions disclosed above. ### Test Output ```text # Final lockfile bytes published as a00f387; Linux / Python 3.12.14 PYTHONPATH=. python -m pytest -q tests/test_compression_cache.py tests/test_tool_result_interceptors.py --disable-warnings 115 passed in 31.60s uv lock --check: passed cargo metadata --locked: passed cargo audit (0.22.2): zero reported vulnerabilities pip-audit of frozen production [all] export: 24 vulnerabilities in Pillow 12.3.0 remain ``` The Rust audit retains the repository's existing RUSTSEC-2024-0436 ignore, unchanged; no new ignores were added. certifi's updated CA bundle successfully loads 120 CA certificates. The exact remote Cargo.lock / uv.lock Git blob hashes match the tested files. Atd74aa5c, the five adjacent cache/registry/interceptor/session/token-mode suites passed 169 tests with zero skips. Whole-repository Ruff and formatting (1805 files), mypy for both modified source modules and whitespace checks passed. The maintainer independently reported 144 Windows/Python 3.12 tests passing with ten optional skips; that is maintainer evidence, not my local run. Reproduce focused coverage atd74aa5c(compare added executable lines against base4c7acba): ```sh python -m pytest tests/test_compression_cache.py tests/test_tool_result_interceptors.py --cov=headroom.cache.compression_cache --cov=headroom.proxy.interceptors.base --cov-branch --cov-report=json:coverage.json --cov-report=term-missing -q ``` Independent local coverage atd74aa5c: 115 cache/interceptor tests passed; all 19 changed executable lines were covered and changed-line branches had no misses. Whole-module coverage was 92% / 86%, not a whole-project coverage claim. Codecov's 00:01:55 UTC comment preceded the relevant shard uploads at 00:02:59 and 00:05:00; final service aggregation remains unverified. Historical hosted CI run39657491771 and rust run39657491768 passed for d74aa5c; Security run39657491814 failed. These are not finala00f387checks. Maintainers verify hosted checks; no contributor workflow rerun was requested or initiated. ### Historical dependency blocker (cleared ona618561)a00f387updates rustls-webpki to 0.103.16, webpki-roots to 1.0.10 and certifi to 2026.10.19, only in the lockfiles. Pillow 12.3.0 remains vulnerable. Pillow 13 requires Python >=3.11; Headroom supports >=3.10. fastembed 0.8.0 also requires Pillow <13. An actual uv resolver attempt using conditional Pillow >=13 failed; the attempted manifest edit was reverted. No overrides, audit suppressions or Python-support removal were introduced. A compatible fastembed/Pillow release or a separately agreed dependency/support-policy change is still required. The initial environment's full-suite collection failed on missing optional/native dependencies; full native/runtime/live-provider qualification has not been established locally. Focused checks do not imply complete qualification. ## Real Behavior Proof - Environment: Linux, Python 3.12.14, pytest 9.1.1, Ruff 0.17.1. Synthetic data only, HEADROOM_OFFLINE=1, HEADROOM_BEACON=off, DO_NOT_TRACK=1; no LLM/provider calls or mocks. - Exact command / steps: Run the following Python snippet in the project environment. I called the real public `CompressionCache.update_from_result` API to seed a cached string compression, then the real shared `prepare_turn` session engine with a new list-form tool result. Minimal reproduction: ```python from headroom.cache.compression_cache import CompressionCache from headroom.proxy.session_engine import prepare_turn, FREEZE_POLICY_REPLAYABLE cache = CompressionCache() raw = "synthetic records " * 50 older = {"role": "tool", "tool_call_id": "c1", "content": raw} cache.update_from_result([older], [{**older, "content": "<compressed>"}]) newest = {"role": "tool", "tool_call_id": "c2", "content": [ {"type": "text", "text": raw, "cache_control": {"type": "ephemeral", "ttl": "1h"}} ]} prepared = prepare_turn(cache, [ {"role": "user", "content": "synthetic request"}, newest ], policy=FREEZE_POLICY_REPLAYABLE) print(prepared.pipeline_input[-1]) ``` - Observed result: Before: content is the string `<compressed>`; the breakpoint is absent. After: ```text {'role': 'tool', 'tool_call_id': 'c2', 'content': [{'type': 'text', 'text': '<compressed>', 'cache_control': {'type': 'ephemeral', 'ttl': '1h'}}]} original_unchanged: True frozen_message_count: 1 ``` - Not tested: live OpenCode/gateway/Bedrock requests, provider cache billing or hit rates, author-run Windows/macOS, final-head native/runtime behavior, full suite. Maintainer Windows results are separately attributed above. This proof establishes session-preparation output, not a live provider outcome. ## Runtime Rollout Safety - Rollout-managed feature(s): none; no new feature/configuration. Compatible certificate dependency lock versions are refreshed; no manifest constraints changed. - Minimum rollout channel: not applicable; this is a cache correctness fix, not a rollout-managed feature. - Kill switch / disable path: no new switch; revert this patch to restore previous behavior. - Qualification impact: no qualification settings changed; hosted Rust/native checks passed on the current merge tree; live-provider qualification remains unverified. - Stable/default behavior changed: cached replay and direct interception preserve single-text-block shape/metadata; unsupported layouts pass through before transformations or savings accounting. - Unsafe override required: no. - Rollback path: revert the change. ## Review Readiness - [x] I have performed a self-review Independent review has also been performed. - [x] This PR is ready for human review Current-head hosted checks and the regression hook passed; maintainer approval and real behavior proof are recorded above. - [x] No CHANGELOG edits.
1407 lines
50 KiB
Python
1407 lines
50 KiB
Python
"""Tests for the tool_result interceptor framework + ast-grep Read outliner."""
|
||
|
||
from __future__ import annotations
|
||
|
||
import asyncio
|
||
import os
|
||
import sys
|
||
import textwrap
|
||
from types import SimpleNamespace
|
||
|
||
import pytest
|
||
|
||
from headroom.proxy.interceptors import (
|
||
INTERCEPTORS,
|
||
ToolResultInterceptor,
|
||
ToolResultInterceptorTransform,
|
||
apply_to_messages,
|
||
interceptor_failure_counts,
|
||
register,
|
||
)
|
||
from headroom.proxy.interceptors.astgrep import (
|
||
AstGrepReadOutline,
|
||
ReadVerificationResult,
|
||
_open_regular_file_under_root,
|
||
_verify_read_against_disk,
|
||
)
|
||
from headroom.proxy.interceptors.base import reset_interceptor_failure_counts
|
||
from headroom.proxy.project_context import get_registered_cwd, set_registered_cwd
|
||
from headroom.tokenizer import Tokenizer
|
||
|
||
|
||
class _FakeTokenCounter:
|
||
"""Deterministic 4-chars-per-token counter for unit tests."""
|
||
|
||
def count_text(self, text: str) -> int:
|
||
return max(1, len(text) // 4)
|
||
|
||
def count_messages(self, messages) -> int:
|
||
total = 0
|
||
for m in messages:
|
||
c = m.get("content")
|
||
if isinstance(c, str):
|
||
total += self.count_text(c)
|
||
elif isinstance(c, list):
|
||
for b in c:
|
||
if isinstance(b, dict):
|
||
inner = b.get("content") or b.get("text") or ""
|
||
if isinstance(inner, str):
|
||
total += self.count_text(inner)
|
||
return total
|
||
|
||
|
||
@pytest.fixture
|
||
def tokenizer() -> Tokenizer:
|
||
# Real Tokenizer wrapping the fake counter; mirrors production construction.
|
||
return Tokenizer(_FakeTokenCounter()) # type: ignore[arg-type]
|
||
|
||
|
||
# -------- Framework basics ----------------------------------------------- #
|
||
|
||
|
||
def test_astgrep_interceptor_registered_by_default():
|
||
assert any(i.name == "ast-grep" for i in INTERCEPTORS)
|
||
|
||
|
||
def test_register_is_idempotent_on_name():
|
||
before = len(INTERCEPTORS)
|
||
register(AstGrepReadOutline()) # same name
|
||
assert len(INTERCEPTORS) == before
|
||
|
||
|
||
def test_custom_interceptor_plugs_in(tokenizer):
|
||
class UpperCase:
|
||
name = "uppercase-test"
|
||
|
||
def matches(self, tool_name, tool_input, tool_output):
|
||
return tool_name == "Echo"
|
||
|
||
def transform(self, tool_name, tool_input, tool_output):
|
||
# Must REDUCE tokens — use a single short marker.
|
||
return "X"
|
||
|
||
dummy: ToolResultInterceptor = UpperCase() # type: ignore[assignment]
|
||
register(dummy)
|
||
try:
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [{"type": "tool_use", "id": "1", "name": "Echo", "input": {}}],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [
|
||
{
|
||
"type": "tool_result",
|
||
"tool_use_id": "1",
|
||
"content": "hello " * 100,
|
||
}
|
||
],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
assert any(s.tool == "uppercase-test" for s in result.spans)
|
||
swapped = result.messages[1]["content"][0]["content"]
|
||
assert swapped == "X"
|
||
finally:
|
||
INTERCEPTORS[:] = [i for i in INTERCEPTORS if i.name != "uppercase-test"]
|
||
|
||
|
||
def test_pass_through_when_no_interceptor_matches(tokenizer):
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [{"type": "tool_use", "id": "1", "name": "Unknown", "input": {}}],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "1", "content": "x" * 5000}],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
assert result.spans == []
|
||
assert result.messages[1] is messages[1] # untouched identity
|
||
|
||
|
||
# -------- ast-grep interceptor ------------------------------------------- #
|
||
|
||
|
||
_PY_FIXTURE = textwrap.dedent(
|
||
'''
|
||
"""Payments module fixture."""
|
||
from decimal import Decimal
|
||
|
||
def compute_subtotal(items):
|
||
total = Decimal("0")
|
||
for item in items:
|
||
total += item.price * item.qty
|
||
return total
|
||
|
||
|
||
def apply_promo(subtotal, code):
|
||
if not code:
|
||
return subtotal
|
||
if code == "SAVE10":
|
||
return subtotal * Decimal("0.9")
|
||
return subtotal
|
||
|
||
|
||
def compute_tax(subtotal, rate):
|
||
return (subtotal * rate).quantize(Decimal("0.01"))
|
||
|
||
|
||
def process_payment(items, promo, tax_rate):
|
||
"""Main entry point."""
|
||
subtotal = compute_subtotal(items)
|
||
after = apply_promo(subtotal, promo)
|
||
tax = compute_tax(after, tax_rate)
|
||
return after + tax
|
||
|
||
|
||
def refund(order_id, amount):
|
||
"""Issue a refund."""
|
||
return {"order": order_id, "refund": str(amount)}
|
||
|
||
|
||
def list_orders_for_user(user_id, limit=20):
|
||
"""Placeholder DB lookup for a user's orders."""
|
||
return [{"user": user_id, "order": i} for i in range(limit)]
|
||
|
||
|
||
def cancel_order(order_id, reason=None):
|
||
"""Cancel an order, logging the reason if provided."""
|
||
return {"order": order_id, "cancelled": True, "reason": reason or "unspecified"}
|
||
|
||
|
||
def summarize_cart(items):
|
||
"""Return a one-line summary of cart contents."""
|
||
skus = [i.sku for i in items]
|
||
total_qty = sum(i.qty for i in items)
|
||
return f"{len(items)} line items ({total_qty} units): {', '.join(skus)}"
|
||
|
||
|
||
def format_receipt(order_id, items, total):
|
||
"""Render a textual receipt."""
|
||
lines = [f"Order {order_id}"]
|
||
for i in items:
|
||
lines.append(f" {i.sku} x {i.qty} @ {i.unit_price} = {i.qty * i.unit_price}")
|
||
lines.append(f"Total: {total}")
|
||
return "\\n".join(lines)
|
||
'''
|
||
).strip()
|
||
|
||
|
||
def test_astgrep_outlines_large_python_read(tokenizer):
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "abc",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "abc", "content": _PY_FIXTURE}],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
assert len(result.spans) == 1
|
||
span = result.spans[0]
|
||
assert span.tool == "ast-grep"
|
||
assert span.tokens_after < span.tokens_before
|
||
new_content = result.messages[1]["content"][0]["content"]
|
||
assert "outlined by ast-grep" in new_content
|
||
assert "body elided" in new_content
|
||
assert "def process_payment" in new_content
|
||
assert "def apply_promo" in new_content
|
||
# Bodies should NOT leak through unchanged.
|
||
assert "total += item.price * item.qty" not in new_content
|
||
# Complete-file control: no truncation banner in the input -> no truncation marker.
|
||
assert "truncated upstream" not in new_content
|
||
|
||
|
||
def test_astgrep_flags_truncated_read(tokenizer):
|
||
truncated_source = (
|
||
_PY_FIXTURE + "\n\n[Truncated: PARTIAL view — /repo/payments.py: "
|
||
"showing lines 1-42 of 90 total (26031 tokens, cap 25000). "
|
||
"Call Read with offset=43 to see more.]\n"
|
||
)
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "abc",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "abc", "content": truncated_source}],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
assert len(result.spans) == 1
|
||
new_content = result.messages[1]["content"][0]["content"]
|
||
assert "truncated upstream" in new_content
|
||
assert "showing through line 42 of 90 total" in new_content
|
||
# Still lists the definitions actually present in the visible portion.
|
||
assert "def process_payment" in new_content
|
||
assert "def apply_promo" in new_content
|
||
|
||
|
||
def _read_result_messages(content: str, file_path: str = "/repo/payments.py"):
|
||
return [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "abc",
|
||
"name": "Read",
|
||
"input": {"file_path": file_path},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "abc", "content": content}],
|
||
},
|
||
]
|
||
|
||
|
||
def test_astgrep_flags_truncated_read_wording_variants(tokenizer):
|
||
"""The signature tolerates wording/casing/dash variation, but only inside
|
||
the recognized envelope — not as a synonym match over arbitrary prose."""
|
||
truncated_source = (
|
||
_PY_FIXTURE + "\n\n[TRUNCATED: PARTIAL VIEW — /repo/payments.py: "
|
||
"SHOWING LINES 1–42 of 90 TOTAL. Call Read with offset=43 to see more.]\n"
|
||
)
|
||
messages = _read_result_messages(truncated_source)
|
||
result = apply_to_messages(messages, tokenizer)
|
||
assert len(result.spans) == 1
|
||
new_content = result.messages[1]["content"][0]["content"]
|
||
assert "truncated upstream" in new_content
|
||
assert "showing through line 42 of 90 total" in new_content
|
||
|
||
|
||
def test_astgrep_ignores_truncation_phrase_in_comment(tokenizer):
|
||
"""A count-shaped phrase in a plain comment, with no bracketed envelope,
|
||
must not be read as an upstream truncation claim."""
|
||
source_with_comment = (
|
||
_PY_FIXTURE + "\n\n# API pagination showing lines 10-20 of 30 total records\n"
|
||
)
|
||
messages = _read_result_messages(source_with_comment)
|
||
result = apply_to_messages(messages, tokenizer)
|
||
new_content = result.messages[1]["content"][0]["content"]
|
||
assert "truncated upstream" not in new_content
|
||
|
||
|
||
def test_astgrep_ignores_bracketed_phrase_without_signature(tokenizer):
|
||
"""Brackets plus a count-shaped phrase aren't enough on their own — the
|
||
exact recognized signature phrase must also be present."""
|
||
source_with_bracket = (
|
||
_PY_FIXTURE + "\n\n[Truncation happened; showing lines 1-42 of 90 total]\n"
|
||
)
|
||
messages = _read_result_messages(source_with_bracket)
|
||
result = apply_to_messages(messages, tokenizer)
|
||
new_content = result.messages[1]["content"][0]["content"]
|
||
assert "truncated upstream" not in new_content
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
"banner_numbers",
|
||
[
|
||
pytest.param("50-90 of 90", id="end_equals_total"),
|
||
pytest.param("42-10 of 90", id="end_less_than_start"),
|
||
pytest.param("0-42 of 90", id="start_is_zero"),
|
||
pytest.param("1-5000 of 9000", id="end_exceeds_visible_payload"),
|
||
],
|
||
)
|
||
def test_astgrep_ignores_malformed_truncation_counts(tokenizer, banner_numbers):
|
||
truncated_source = (
|
||
_PY_FIXTURE + "\n\n[Truncated: PARTIAL view — /repo/payments.py: "
|
||
f"showing lines {banner_numbers} total (26031 tokens, cap 25000). "
|
||
"Call Read with offset=43 to see more.]\n"
|
||
)
|
||
messages = _read_result_messages(truncated_source)
|
||
result = apply_to_messages(messages, tokenizer)
|
||
new_content = result.messages[1]["content"][0]["content"]
|
||
assert "outlined by ast-grep" in new_content
|
||
assert "truncated upstream" not in new_content
|
||
|
||
|
||
def test_astgrep_accepts_truncation_at_start_equals_end(tokenizer):
|
||
"""`end >= start` is inclusive — a single-line visible window is valid."""
|
||
truncated_source = (
|
||
_PY_FIXTURE + "\n\n[Truncated: PARTIAL view — /repo/payments.py: "
|
||
"showing lines 42-42 of 90 total (26031 tokens, cap 25000). "
|
||
"Call Read with offset=43 to see more.]\n"
|
||
)
|
||
messages = _read_result_messages(truncated_source)
|
||
result = apply_to_messages(messages, tokenizer)
|
||
new_content = result.messages[1]["content"][0]["content"]
|
||
assert "truncated upstream" in new_content
|
||
|
||
|
||
# -------- Disk verification: client-independent truncation fallback ----- #
|
||
|
||
|
||
def test_set_get_registered_cwd_round_trips():
|
||
set_registered_cwd("/repo/project")
|
||
try:
|
||
assert get_registered_cwd() == "/repo/project"
|
||
finally:
|
||
set_registered_cwd(None)
|
||
|
||
|
||
class TestVerifyReadAgainstDisk:
|
||
def test_missing_file_path_is_unknown(self):
|
||
verdict, info = _verify_read_against_disk(None, "abc", "/repo")
|
||
assert verdict is ReadVerificationResult.UNKNOWN
|
||
assert info is None
|
||
|
||
def test_no_registered_cwd_is_unknown(self):
|
||
verdict, info = _verify_read_against_disk("payments.py", "abc", None)
|
||
assert verdict is ReadVerificationResult.UNKNOWN
|
||
assert info is None
|
||
|
||
def test_untrusted_request_never_touches_disk(self, tmp_path, monkeypatch):
|
||
"""A valid file/content combo must short-circuit to UNKNOWN -- and
|
||
never call os.open -- when no root is registered."""
|
||
f = tmp_path / "payments.py"
|
||
f.write_text(_PY_FIXTURE, encoding="utf-8")
|
||
opened: list[object] = []
|
||
real_open = os.open
|
||
|
||
def _tracking_open(*args, **kwargs):
|
||
opened.append(args)
|
||
return real_open(*args, **kwargs)
|
||
|
||
monkeypatch.setattr(os, "open", _tracking_open)
|
||
verdict, info = _verify_read_against_disk(str(f), _PY_FIXTURE, None)
|
||
assert verdict is ReadVerificationResult.UNKNOWN
|
||
assert info is None
|
||
assert opened == []
|
||
|
||
def test_missing_file_under_workspace_root_is_unknown(self, tmp_path):
|
||
verdict, info = _verify_read_against_disk(
|
||
str(tmp_path / "missing.py"), "abc", str(tmp_path)
|
||
)
|
||
assert verdict is ReadVerificationResult.UNKNOWN
|
||
assert info is None
|
||
|
||
@pytest.mark.skipif(
|
||
sys.platform == "win32",
|
||
reason="dir_fd disk verification unavailable on Windows; falls back to UNKNOWN"
|
||
" (see test_open_regular_file_under_root_falls_back_unknown_when_dir_fd_unsupported)",
|
||
)
|
||
def test_exact_match_is_complete(self, tmp_path):
|
||
f = tmp_path / "payments.py"
|
||
f.write_text(_PY_FIXTURE, encoding="utf-8")
|
||
verdict, info = _verify_read_against_disk(str(f), _PY_FIXTURE, str(tmp_path))
|
||
assert verdict is ReadVerificationResult.COMPLETE
|
||
assert info is None
|
||
|
||
@pytest.mark.skipif(
|
||
sys.platform == "win32",
|
||
reason="dir_fd disk verification unavailable on Windows; falls back to UNKNOWN"
|
||
" (see test_open_regular_file_under_root_falls_back_unknown_when_dir_fd_unsupported)",
|
||
)
|
||
def test_strict_prefix_is_truncated(self, tmp_path):
|
||
f = tmp_path / "payments.py"
|
||
f.write_text(_PY_FIXTURE, encoding="utf-8")
|
||
partial = _PY_FIXTURE[:200]
|
||
verdict, info = _verify_read_against_disk(str(f), partial, str(tmp_path))
|
||
assert verdict is ReadVerificationResult.TRUNCATED
|
||
assert info == (len(partial.splitlines()), len(_PY_FIXTURE.splitlines()))
|
||
|
||
@pytest.mark.skipif(
|
||
sys.platform == "win32",
|
||
reason="dir_fd disk verification unavailable on Windows; falls back to UNKNOWN"
|
||
" (see test_open_regular_file_under_root_falls_back_unknown_when_dir_fd_unsupported)",
|
||
)
|
||
def test_relative_path_resolves_against_registered_cwd(self, tmp_path):
|
||
f = tmp_path / "payments.py"
|
||
f.write_text(_PY_FIXTURE, encoding="utf-8")
|
||
partial = _PY_FIXTURE[:200]
|
||
verdict, info = _verify_read_against_disk("payments.py", partial, str(tmp_path))
|
||
assert verdict is ReadVerificationResult.TRUNCATED
|
||
assert info is not None
|
||
|
||
def test_content_mismatch_is_unknown_not_truncated(self, tmp_path):
|
||
# File diverged since the client read it -- not a clean prefix.
|
||
f = tmp_path / "payments.py"
|
||
f.write_text(_PY_FIXTURE.replace("compute_subtotal", "compute_total"), encoding="utf-8")
|
||
verdict, info = _verify_read_against_disk(str(f), _PY_FIXTURE, str(tmp_path))
|
||
assert verdict is ReadVerificationResult.UNKNOWN
|
||
assert info is None
|
||
|
||
def test_absolute_path_outside_workspace_root_is_unknown(self, tmp_path):
|
||
root = tmp_path / "project"
|
||
root.mkdir()
|
||
sibling = tmp_path / "other"
|
||
sibling.mkdir()
|
||
f = sibling / "secret.py"
|
||
f.write_text(_PY_FIXTURE, encoding="utf-8")
|
||
verdict, info = _verify_read_against_disk(str(f), _PY_FIXTURE, str(root))
|
||
assert verdict is ReadVerificationResult.UNKNOWN
|
||
assert info is None
|
||
|
||
def test_relative_traversal_escapes_workspace_is_unknown(self, tmp_path):
|
||
root = tmp_path / "project"
|
||
root.mkdir()
|
||
f = tmp_path / "secret.py"
|
||
f.write_text(_PY_FIXTURE, encoding="utf-8")
|
||
verdict, info = _verify_read_against_disk("../secret.py", _PY_FIXTURE, str(root))
|
||
assert verdict is ReadVerificationResult.UNKNOWN
|
||
assert info is None
|
||
|
||
def test_sibling_directory_sharing_string_prefix_is_not_inside_workspace(self, tmp_path):
|
||
"""Containment is a path-segment check (relpath/segment split), not
|
||
a string-prefix check -- a target that merely starts with the
|
||
root's string must not be treated as inside it."""
|
||
root = tmp_path / "project"
|
||
root.mkdir()
|
||
other = tmp_path / "project-other"
|
||
other.mkdir()
|
||
f = other / "secret.py"
|
||
f.write_text(_PY_FIXTURE, encoding="utf-8")
|
||
verdict, info = _verify_read_against_disk(str(f), _PY_FIXTURE, str(root))
|
||
assert verdict is ReadVerificationResult.UNKNOWN
|
||
assert info is None
|
||
|
||
def test_directory_passed_as_file_path_is_unknown(self, tmp_path):
|
||
root = tmp_path / "project"
|
||
subdir = root / "subdir"
|
||
subdir.mkdir(parents=True)
|
||
verdict, info = _verify_read_against_disk("subdir", "abc", str(root))
|
||
assert verdict is ReadVerificationResult.UNKNOWN
|
||
assert info is None
|
||
|
||
def test_oversized_file_exceeding_byte_cap_is_unknown(self, tmp_path, monkeypatch):
|
||
monkeypatch.setenv("HEADROOM_VERIFY_TRUNCATION_MAX_BYTES", "10")
|
||
f = tmp_path / "payments.py"
|
||
f.write_text(_PY_FIXTURE, encoding="utf-8")
|
||
assert len(_PY_FIXTURE.encode("utf-8")) > 10
|
||
# Content matches exactly -- would be COMPLETE without the cap.
|
||
verdict, info = _verify_read_against_disk(str(f), _PY_FIXTURE, str(tmp_path))
|
||
assert verdict is ReadVerificationResult.UNKNOWN
|
||
assert info is None
|
||
|
||
|
||
# -------- _open_regular_file_under_root: race-safety + off-loop (review round 2) --------- #
|
||
|
||
|
||
@pytest.mark.skipif(sys.platform == "win32", reason="symlinks need elevated privilege on Windows")
|
||
def test_open_regular_file_under_root_rejects_symlinked_final_file(tmp_path):
|
||
"""A symlinked leaf file must be rejected by O_NOFOLLOW, not followed.
|
||
|
||
A static symlink is sufficient race evidence here: O_NOFOLLOW rejects it
|
||
the same way whether it's always been there or appeared 2ms ago -- a
|
||
concurrent-race harness would prove nothing more and would be flaky.
|
||
"""
|
||
root = tmp_path / "project"
|
||
root.mkdir()
|
||
outside = tmp_path / "outside.py"
|
||
outside.write_text(_PY_FIXTURE, encoding="utf-8")
|
||
link = root / "link.py"
|
||
link.symlink_to(outside)
|
||
assert _open_regular_file_under_root("link.py", root, 10_000) is None
|
||
|
||
|
||
@pytest.mark.skipif(sys.platform == "win32", reason="symlinks need elevated privilege on Windows")
|
||
def test_open_regular_file_under_root_rejects_symlinked_intermediate_directory(tmp_path):
|
||
"""A symlinked directory component must be rejected at that hop, not
|
||
walked into."""
|
||
root = tmp_path / "project"
|
||
root.mkdir()
|
||
real_dir = tmp_path / "realdir"
|
||
real_dir.mkdir()
|
||
(real_dir / "secret.py").write_text(_PY_FIXTURE, encoding="utf-8")
|
||
link_dir = root / "linkdir"
|
||
link_dir.symlink_to(real_dir)
|
||
assert _open_regular_file_under_root("linkdir/secret.py", root, 10_000) is None
|
||
|
||
|
||
@pytest.mark.skipif(sys.platform == "win32", reason="symlinks need elevated privilege on Windows")
|
||
def test_open_regular_file_under_root_rejects_symlinked_root_anchor(tmp_path):
|
||
"""The root anchor itself must be opened O_NOFOLLOW too, not just the
|
||
segments below it -- otherwise a symlink swapped in at the resolved root
|
||
path would be followed, the exact TOCTOU shape review round 2 flagged,
|
||
just moved from the leaf to the anchor."""
|
||
real_root = tmp_path / "real-project"
|
||
real_root.mkdir()
|
||
(real_root / "x.py").write_text("content", encoding="utf-8")
|
||
root_link = tmp_path / "project-link"
|
||
root_link.symlink_to(real_root)
|
||
assert _open_regular_file_under_root("x.py", root_link, 10_000) is None
|
||
|
||
|
||
def test_open_regular_file_under_root_refuses_on_event_loop_thread(tmp_path):
|
||
"""Calling from inside a running event loop must refuse immediately
|
||
rather than perform the read."""
|
||
f = tmp_path / "x.py"
|
||
f.write_text("content", encoding="utf-8")
|
||
|
||
async def _call_from_loop():
|
||
return _open_regular_file_under_root(str(f), tmp_path, 10_000)
|
||
|
||
assert asyncio.run(_call_from_loop()) is None
|
||
|
||
|
||
@pytest.mark.skipif(
|
||
sys.platform == "win32",
|
||
reason="dir_fd disk verification unavailable on Windows; falls back to UNKNOWN"
|
||
" (see test_open_regular_file_under_root_falls_back_unknown_when_dir_fd_unsupported)",
|
||
)
|
||
def test_open_regular_file_under_root_proceeds_from_plain_sync_context(tmp_path):
|
||
"""Sanity check: with no running loop, the read must still succeed."""
|
||
f = tmp_path / "x.py"
|
||
f.write_text("content", encoding="utf-8")
|
||
assert _open_regular_file_under_root(str(f), tmp_path, 10_000) == "content"
|
||
|
||
|
||
@pytest.mark.skipif(
|
||
sys.platform == "win32",
|
||
reason="dir_fd disk verification unavailable on Windows; falls back to UNKNOWN"
|
||
" (see test_open_regular_file_under_root_falls_back_unknown_when_dir_fd_unsupported)",
|
||
)
|
||
def test_open_regular_file_under_root_rejects_growth_between_stat_and_read(tmp_path, monkeypatch):
|
||
"""A file that grows past max_bytes between the fstat check and the
|
||
read must fail closed to None, not return the oversized content -- the
|
||
fstat result can go stale by the time the read actually runs. The file
|
||
genuinely contains >max_bytes on disk; only the fstat result lies, so a
|
||
passing test proves the read itself is bounded, not that fstat was
|
||
merely re-checked."""
|
||
f = tmp_path / "x.py"
|
||
f.write_bytes(b"x" * 1_000_001)
|
||
|
||
real_fstat = os.fstat
|
||
|
||
def lying_fstat(fd):
|
||
real_st = real_fstat(fd)
|
||
return SimpleNamespace(st_mode=real_st.st_mode, st_size=1)
|
||
|
||
monkeypatch.setattr(os, "fstat", lying_fstat)
|
||
assert _open_regular_file_under_root("x.py", tmp_path, max_bytes=8) is None
|
||
|
||
|
||
def test_open_regular_file_under_root_falls_back_unknown_when_dir_fd_unsupported(
|
||
tmp_path, monkeypatch
|
||
):
|
||
"""Fails closed (None) rather than falling back to a less-safe pattern
|
||
when dir_fd-relative opens aren't supported (notably Windows)."""
|
||
import headroom.proxy.interceptors.astgrep as astgrep_module
|
||
|
||
f = tmp_path / "x.py"
|
||
f.write_text("content", encoding="utf-8")
|
||
monkeypatch.setattr(astgrep_module, "_dir_fd_walk_supported", lambda: False)
|
||
assert _open_regular_file_under_root(str(f), tmp_path, 10_000) is None
|
||
|
||
|
||
def test_open_regular_file_under_root_directory_passed_as_target_is_unknown(tmp_path):
|
||
"""A directory at the leaf position must be rejected -- the target must
|
||
be a file."""
|
||
(tmp_path / "subdir").mkdir()
|
||
assert _open_regular_file_under_root("subdir", tmp_path, 10_000) is None
|
||
|
||
|
||
@pytest.mark.skipif(sys.platform == "win32", reason="os.mkfifo unavailable on Windows")
|
||
def test_open_regular_file_under_root_rejects_fifo_without_hanging(tmp_path):
|
||
"""A FIFO must be rejected by the S_ISREG check, and -- because the open
|
||
uses O_NONBLOCK -- must return promptly rather than blocking on a
|
||
writer that will never arrive. If this test hangs, O_NONBLOCK regressed."""
|
||
fifo_path = tmp_path / "pipe"
|
||
os.mkfifo(fifo_path)
|
||
assert _open_regular_file_under_root("pipe", tmp_path, 10_000) is None
|
||
|
||
|
||
# -------- End-to-end: apply_to_messages, real registered_cwd --------- #
|
||
|
||
|
||
def _disk_verification_messages(partial: str) -> list[dict]:
|
||
return [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "abc",
|
||
"name": "Read",
|
||
"input": {"file_path": "payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "abc", "content": partial}],
|
||
},
|
||
]
|
||
|
||
|
||
@pytest.mark.skipif(
|
||
sys.platform == "win32",
|
||
reason="dir_fd disk verification unavailable on Windows; falls back to UNKNOWN"
|
||
" (see test_astgrep_disk_verification_stays_unknown_when_dir_fd_unsupported)",
|
||
)
|
||
def test_astgrep_disk_verification_flags_truncation_when_no_banner(
|
||
tokenizer, tmp_path, monkeypatch
|
||
):
|
||
"""Opted in, root registered, real on-disk prefix, no banner -- disk
|
||
verification alone qualifies the header."""
|
||
monkeypatch.setenv("HEADROOM_VERIFY_TRUNCATION_ON_DISK", "1")
|
||
monkeypatch.setenv("HEADROOM_INTERCEPT_READ_MIN_CHARS", "50")
|
||
f = tmp_path / "payments.py"
|
||
f.write_text(_PY_FIXTURE, encoding="utf-8")
|
||
marker = "\n\ndef format_receipt"
|
||
partial = _PY_FIXTURE[: _PY_FIXTURE.index(marker)] # no banner text anywhere
|
||
assert "truncated" not in partial.lower()
|
||
|
||
set_registered_cwd(str(tmp_path))
|
||
try:
|
||
result = apply_to_messages(_disk_verification_messages(partial), tokenizer)
|
||
finally:
|
||
set_registered_cwd(None)
|
||
|
||
assert len(result.spans) == 1
|
||
new_content = result.messages[1]["content"][0]["content"]
|
||
assert "truncated upstream" in new_content
|
||
visible_lines = len(partial.splitlines())
|
||
total_lines = len(_PY_FIXTURE.splitlines())
|
||
assert f"showing through line {visible_lines} of {total_lines} total" in new_content
|
||
assert "def compute_subtotal" in new_content
|
||
assert "def apply_promo" in new_content
|
||
assert "def format_receipt" not in new_content
|
||
|
||
|
||
def test_astgrep_disk_verification_stays_unknown_when_dir_fd_unsupported(
|
||
tokenizer, tmp_path, monkeypatch
|
||
):
|
||
"""Fail-closed contract, simulated cross-platform: dir_fd unsupported ->
|
||
UNKNOWN -> no false "truncated upstream" claim. Stands in for the real
|
||
Windows behavior the skipif'd tests above can't exercise here."""
|
||
import headroom.proxy.interceptors.astgrep as astgrep_module
|
||
|
||
monkeypatch.setattr(astgrep_module, "_dir_fd_walk_supported", lambda: False)
|
||
monkeypatch.setenv("HEADROOM_VERIFY_TRUNCATION_ON_DISK", "1")
|
||
monkeypatch.setenv("HEADROOM_INTERCEPT_READ_MIN_CHARS", "50")
|
||
f = tmp_path / "payments.py"
|
||
f.write_text(_PY_FIXTURE, encoding="utf-8")
|
||
marker = "\n\ndef format_receipt"
|
||
partial = _PY_FIXTURE[: _PY_FIXTURE.index(marker)] # no banner text anywhere
|
||
assert "truncated" not in partial.lower()
|
||
|
||
set_registered_cwd(str(tmp_path))
|
||
try:
|
||
result = apply_to_messages(_disk_verification_messages(partial), tokenizer)
|
||
finally:
|
||
set_registered_cwd(None)
|
||
|
||
# The outline rewrite is independent of the disk-verify verdict.
|
||
assert len(result.spans) == 1
|
||
new_content = result.messages[1]["content"][0]["content"]
|
||
assert "truncated upstream" not in new_content
|
||
|
||
|
||
def test_astgrep_disk_verification_skips_when_no_root_registered(tokenizer, tmp_path, monkeypatch):
|
||
"""No registered root at all (the default) -- opted in, real on-disk
|
||
prefix, but never flags. Spoofed-header-doesn't-work is covered
|
||
end-to-end in test_proxy_workspace_registration_middleware.py."""
|
||
monkeypatch.setenv("HEADROOM_VERIFY_TRUNCATION_ON_DISK", "1")
|
||
monkeypatch.setenv("HEADROOM_INTERCEPT_READ_MIN_CHARS", "50")
|
||
f = tmp_path / "payments.py"
|
||
f.write_text(_PY_FIXTURE, encoding="utf-8")
|
||
marker = "\n\ndef format_receipt"
|
||
partial = _PY_FIXTURE[: _PY_FIXTURE.index(marker)]
|
||
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "abc",
|
||
"name": "Read",
|
||
"input": {"file_path": "payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "abc", "content": partial}],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
|
||
new_content = result.messages[1]["content"][0]["content"]
|
||
assert "truncated upstream" not in new_content
|
||
|
||
|
||
def test_astgrep_banner_detection_takes_priority_over_disk_verification(tokenizer, monkeypatch):
|
||
"""When a banner is already present, disk verification must not run at
|
||
all (no cwd bound here -- if it ran, resolution would fail anyway), and
|
||
the banner's own numbers must be what the header reports."""
|
||
monkeypatch.setenv("HEADROOM_VERIFY_TRUNCATION_ON_DISK", "1")
|
||
truncated_source = (
|
||
_PY_FIXTURE + "\n\n[Truncated: PARTIAL view — /repo/payments.py: "
|
||
"showing lines 1-42 of 90 total (26031 tokens, cap 25000). "
|
||
"Call Read with offset=43 to see more.]\n"
|
||
)
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "abc",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "abc", "content": truncated_source}],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
new_content = result.messages[1]["content"][0]["content"]
|
||
assert "showing through line 42 of 90 total" in new_content
|
||
|
||
|
||
def test_astgrep_skips_small_files(tokenizer):
|
||
small = "def foo(): return 1\n"
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "x",
|
||
"name": "Read",
|
||
"input": {"file_path": "/a.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "x", "content": small}],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
assert result.spans == []
|
||
|
||
|
||
def test_astgrep_skips_non_code_extensions(tokenizer):
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "r",
|
||
"name": "Read",
|
||
"input": {"file_path": "/notes.txt"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "r", "content": "x" * 3000}],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
assert result.spans == []
|
||
|
||
|
||
# -------- OpenAI-format tool_result -------------------------------------- #
|
||
|
||
|
||
def test_astgrep_skips_when_line_range_requested(tokenizer):
|
||
"""If the tool_input specifies a line range, the model wants those lines — pass through."""
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "r",
|
||
"name": "Read",
|
||
"input": {
|
||
"file_path": "/repo/payments.py",
|
||
"offset": 30,
|
||
"limit": 20,
|
||
},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "r", "content": _PY_FIXTURE}],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
assert result.spans == []
|
||
|
||
|
||
def test_progressive_disclosure_second_read_passes_through(tokenizer):
|
||
"""First Read of a file gets outlined; second Read of the same path is untouched."""
|
||
messages = [
|
||
# Turn 1: Read foo.py → outlined
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "t1",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": _PY_FIXTURE}],
|
||
},
|
||
# Turn 2: Read foo.py again (model came back for more) → pass through
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "t2",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "t2", "content": _PY_FIXTURE}],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
# Only the first Read is rewritten; the second keeps its full body.
|
||
assert len(result.spans) == 1
|
||
first_tr = result.messages[1]["content"][0]["content"]
|
||
second_tr = result.messages[3]["content"][0]["content"]
|
||
assert "outlined by ast-grep" in first_tr
|
||
assert "outlined by ast-grep" not in second_tr
|
||
assert "def process_payment" in second_tr
|
||
# Second Read preserves the bodies.
|
||
assert "subtotal = compute_subtotal(items)" in second_tr
|
||
|
||
|
||
def test_progressive_disclosure_different_file_still_outlined(tokenizer):
|
||
"""Reading a DIFFERENT file after the first outline should still outline."""
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "t1",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": _PY_FIXTURE}],
|
||
},
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "t2",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/other.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "t2", "content": _PY_FIXTURE}],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
# Both files get outlined — different keys.
|
||
assert len(result.spans) == 2
|
||
|
||
|
||
def test_openai_format_tool_result_is_rewritten(tokenizer):
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": None,
|
||
"tool_calls": [
|
||
{
|
||
"id": "call_1",
|
||
"type": "function",
|
||
"function": {
|
||
"name": "Read",
|
||
"arguments": '{"file_path": "/x/payments.py"}',
|
||
},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "tool",
|
||
"tool_call_id": "call_1",
|
||
"content": _PY_FIXTURE,
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
assert len(result.spans) == 1
|
||
new_content = result.messages[1]["content"]
|
||
assert "outlined by ast-grep" in new_content
|
||
|
||
|
||
# -------- Failure isolation & safety guarantees -------------------------- #
|
||
|
||
|
||
def test_failing_interceptor_does_not_crash_request(tokenizer):
|
||
"""If transform() raises, the request still succeeds unchanged."""
|
||
reset_interceptor_failure_counts()
|
||
|
||
class BoomInterceptor:
|
||
name = "boom"
|
||
|
||
def matches(self, tool_name, tool_input, tool_output):
|
||
return tool_name == "Read"
|
||
|
||
def transform(self, tool_name, tool_input, tool_output):
|
||
raise RuntimeError("simulated interceptor bug")
|
||
|
||
register(BoomInterceptor())
|
||
try:
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "b",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "b", "content": _PY_FIXTURE}],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
# No span recorded for boom; request survives.
|
||
assert not any(s.tool == "boom" for s in result.spans)
|
||
# The failure counter incremented.
|
||
assert interceptor_failure_counts().get("boom") == 1
|
||
finally:
|
||
INTERCEPTORS[:] = [i for i in INTERCEPTORS if i.name != "boom"]
|
||
|
||
|
||
def test_failing_key_skips_interceptor_entirely(tokenizer):
|
||
"""Broken progressive_disclosure_key() must skip, not fire without a key."""
|
||
reset_interceptor_failure_counts()
|
||
fire_count = {"n": 0}
|
||
|
||
class BadKey:
|
||
name = "bad-key"
|
||
|
||
def matches(self, tool_name, tool_input, tool_output):
|
||
return tool_name == "Read"
|
||
|
||
def transform(self, tool_name, tool_input, tool_output):
|
||
fire_count["n"] += 1
|
||
return "X" # reduces tokens
|
||
|
||
def progressive_disclosure_key(self, tool_name, tool_input):
|
||
raise RuntimeError("cannot compute key")
|
||
|
||
register(BadKey())
|
||
try:
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "k",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "k", "content": _PY_FIXTURE}],
|
||
},
|
||
]
|
||
apply_to_messages(messages, tokenizer)
|
||
assert fire_count["n"] == 0 # transform never ran
|
||
assert interceptor_failure_counts().get("bad-key") == 1
|
||
finally:
|
||
INTERCEPTORS[:] = [i for i in INTERCEPTORS if i.name != "bad-key"]
|
||
|
||
|
||
def test_refuses_to_enlarge(tokenizer):
|
||
"""If rewrite has MORE tokens than original, pass through unchanged.
|
||
|
||
Uses a non-code tool path so only the Inflater runs (ast-grep passes
|
||
through on non-Read tools).
|
||
"""
|
||
original_content = "some data " * 200
|
||
|
||
class Inflater:
|
||
name = "inflater"
|
||
|
||
def matches(self, tool_name, tool_input, tool_output):
|
||
return tool_name == "FetchPage"
|
||
|
||
def transform(self, tool_name, tool_input, tool_output):
|
||
return tool_output + (" padding" * 200)
|
||
|
||
register(Inflater())
|
||
try:
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "i",
|
||
"name": "FetchPage",
|
||
"input": {"url": "https://example.com"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [
|
||
{"type": "tool_result", "tool_use_id": "i", "content": original_content}
|
||
],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
assert not any(s.tool == "inflater" for s in result.spans)
|
||
# Original content preserved.
|
||
assert result.messages[1]["content"][0]["content"] == original_content
|
||
finally:
|
||
INTERCEPTORS[:] = [i for i in INTERCEPTORS if i.name != "inflater"]
|
||
|
||
|
||
def test_orphaned_tool_result_does_not_crash(tokenizer):
|
||
"""A tool_result with no matching tool_use still runs safely (no tool_name)."""
|
||
messages = [
|
||
# No tool_use block — the model's prior turn is missing.
|
||
{
|
||
"role": "user",
|
||
"content": [
|
||
{"type": "tool_result", "tool_use_id": "orphan-id", "content": _PY_FIXTURE}
|
||
],
|
||
},
|
||
]
|
||
result = apply_to_messages(messages, tokenizer)
|
||
# ast-grep.matches() returns False when tool_name is None, so no span.
|
||
assert result.spans == []
|
||
# The orphan message is preserved.
|
||
assert result.messages[0]["content"][0]["content"] == _PY_FIXTURE
|
||
|
||
|
||
# -------- Transform adapter tests ---------------------------------------- #
|
||
|
||
|
||
def test_transform_adapter_applies_interceptors(tokenizer):
|
||
"""ToolResultInterceptorTransform.apply() runs interceptors + records tokens."""
|
||
transform = ToolResultInterceptorTransform()
|
||
messages = [
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "a",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "a", "content": _PY_FIXTURE}],
|
||
},
|
||
]
|
||
result = transform.apply(messages, tokenizer)
|
||
assert result.tokens_after < result.tokens_before
|
||
assert "interceptor:ast-grep" in result.transforms_applied
|
||
|
||
|
||
def test_transform_adapter_respects_frozen_message_count(tokenizer):
|
||
"""Messages in the frozen prefix must be untouched to preserve prefix caches."""
|
||
transform = ToolResultInterceptorTransform()
|
||
messages = [
|
||
# Frozen prefix (first tool_result) — MUST pass through unchanged.
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "t1",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/a.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": _PY_FIXTURE}],
|
||
},
|
||
# Mutable tail (second Read of a different file) — free to outline.
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "t2",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/b.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [{"type": "tool_result", "tool_use_id": "t2", "content": _PY_FIXTURE}],
|
||
},
|
||
]
|
||
result = transform.apply(messages, tokenizer, frozen_message_count=2)
|
||
# Frozen prefix identity preserved (exact same list refs).
|
||
assert result.messages[0] is messages[0]
|
||
assert result.messages[1] is messages[1]
|
||
# Tail got outlined.
|
||
assert "outlined by ast-grep" in result.messages[3]["content"][0]["content"]
|
||
|
||
|
||
def test_progressive_disclosure_respects_frozen_prefix_history(tokenizer):
|
||
"""If a file was Read in the frozen prefix, re-reading it in the mutable
|
||
tail passes through — even though apply_to_messages only sees the tail
|
||
for rewriting, it pre-scans the frozen prefix to seed `fired` keys.
|
||
"""
|
||
transform = ToolResultInterceptorTransform()
|
||
messages = [
|
||
# Frozen prefix: first Read of payments.py. This is cached, so we
|
||
# don't outline it; but it counts as "already disclosed."
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "frozen-read",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [
|
||
{
|
||
"type": "tool_result",
|
||
"tool_use_id": "frozen-read",
|
||
"content": _PY_FIXTURE,
|
||
}
|
||
],
|
||
},
|
||
# Mutable tail: model reads payments.py again — should pass through
|
||
# because the frozen prefix already served it.
|
||
{
|
||
"role": "assistant",
|
||
"content": [
|
||
{
|
||
"type": "tool_use",
|
||
"id": "tail-read",
|
||
"name": "Read",
|
||
"input": {"file_path": "/repo/payments.py"},
|
||
}
|
||
],
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": [
|
||
{
|
||
"type": "tool_result",
|
||
"tool_use_id": "tail-read",
|
||
"content": _PY_FIXTURE,
|
||
}
|
||
],
|
||
},
|
||
]
|
||
result = transform.apply(messages, tokenizer, frozen_message_count=2)
|
||
# Tail re-read preserved (not outlined) because the frozen prefix
|
||
# already exposed the file.
|
||
tail_content = result.messages[3]["content"][0]["content"]
|
||
assert "outlined by ast-grep" not in tail_content
|
||
assert "def process_payment" in tail_content
|
||
assert "subtotal = compute_subtotal(items)" in tail_content
|
||
|
||
|
||
def test_transform_adapter_tokens_before_is_baseline_not_reconstruction(tokenizer):
|
||
"""tokens_before must reflect the real original messages, not back-calc."""
|
||
transform = ToolResultInterceptorTransform()
|
||
messages = [
|
||
{"role": "user", "content": [{"type": "text", "text": "plain non-tool message"}]},
|
||
]
|
||
result = transform.apply(messages, tokenizer)
|
||
# No spans, no change.
|
||
assert result.tokens_before == result.tokens_after
|
||
assert result.transforms_applied == []
|
||
|
||
|
||
def test_proxy_pipeline_includes_interceptor_when_env_enabled(monkeypatch):
|
||
"""An eligible legacy request installs the interceptor in both pipelines."""
|
||
monkeypatch.setenv("HEADROOM_INTERCEPT_ENABLED", "1")
|
||
monkeypatch.setenv("HEADROOM_ROLLOUT_CHANNEL", "canary")
|
||
from headroom.proxy.interceptors import ToolResultInterceptorTransform
|
||
from headroom.proxy.models import ProxyConfig
|
||
from headroom.proxy.server import HeadroomProxy
|
||
|
||
proxy = HeadroomProxy(ProxyConfig())
|
||
for pipeline in (proxy.anthropic_pipeline, proxy.openai_pipeline):
|
||
transforms = pipeline.transforms
|
||
assert len(transforms) > 0
|
||
assert isinstance(transforms[0], ToolResultInterceptorTransform)
|
||
|
||
|
||
def test_proxy_pipeline_blocks_interceptor_below_rollout_channel(monkeypatch):
|
||
"""A legacy request cannot bypass the stable rollout-channel boundary."""
|
||
monkeypatch.setenv("HEADROOM_INTERCEPT_ENABLED", "1")
|
||
monkeypatch.setenv("HEADROOM_ROLLOUT_CHANNEL", "stable")
|
||
from headroom.proxy.interceptors import ToolResultInterceptorTransform
|
||
from headroom.proxy.models import ProxyConfig
|
||
from headroom.proxy.server import HeadroomProxy
|
||
|
||
proxy = HeadroomProxy(ProxyConfig())
|
||
for pipeline in (proxy.anthropic_pipeline, proxy.openai_pipeline):
|
||
assert not any(isinstance(t, ToolResultInterceptorTransform) for t in pipeline.transforms)
|
||
|
||
|
||
def test_proxy_pipeline_excludes_interceptor_when_env_not_set(monkeypatch):
|
||
"""When HEADROOM_INTERCEPT_ENABLED is unset, no interceptor in either pipeline."""
|
||
monkeypatch.delenv("HEADROOM_INTERCEPT_ENABLED", raising=False)
|
||
from headroom.proxy.interceptors import ToolResultInterceptorTransform
|
||
from headroom.proxy.models import ProxyConfig
|
||
from headroom.proxy.server import HeadroomProxy
|
||
|
||
proxy = HeadroomProxy(ProxyConfig())
|
||
for pipeline in (proxy.anthropic_pipeline, proxy.openai_pipeline):
|
||
transforms = pipeline.transforms
|
||
assert not any(isinstance(t, ToolResultInterceptorTransform) for t in transforms)
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
"content",
|
||
[
|
||
[
|
||
{"type": "text", "text": "first long block " * 20},
|
||
{"type": "text", "text": "second long block " * 20},
|
||
],
|
||
[
|
||
{"type": "image_url", "image_url": {"url": "data:image/png;base64,AA=="}},
|
||
{"type": "text", "text": "long text " * 40},
|
||
],
|
||
[],
|
||
["not a block"],
|
||
[{"type": "text"}],
|
||
[{"type": "text", "text": 42}],
|
||
],
|
||
)
|
||
def test_unsupported_openai_blocks_skip_interceptors_and_savings(content, tokenizer, monkeypatch):
|
||
from copy import deepcopy
|
||
|
||
class Shortener:
|
||
name = "shape-regression"
|
||
calls = 0
|
||
|
||
def progressive_disclosure_key(self, *_):
|
||
self.calls += 1
|
||
return "same-key"
|
||
|
||
def matches(self, *_):
|
||
self.calls += 1
|
||
return True
|
||
|
||
def transform(self, *_):
|
||
self.calls += 1
|
||
return "short"
|
||
|
||
interceptor = Shortener()
|
||
monkeypatch.setattr("headroom.proxy.interceptors.base.INTERCEPTORS", [interceptor])
|
||
unsupported = {"role": "tool", "tool_call_id": "1", "content": content}
|
||
snapshot = deepcopy(unsupported)
|
||
result = apply_to_messages([unsupported], tokenizer)
|
||
assert result.messages == [snapshot]
|
||
assert result.messages[0] is unsupported
|
||
assert result.spans == []
|
||
assert interceptor.calls == 0
|
||
assert unsupported == snapshot
|
||
pipeline_result = ToolResultInterceptorTransform().apply([unsupported], tokenizer)
|
||
assert pipeline_result.messages == [snapshot]
|
||
assert pipeline_result.transforms_applied == []
|
||
assert pipeline_result.tokens_before == pipeline_result.tokens_after
|
||
assert interceptor.calls == 0
|
||
|
||
# A skipped layout must not consume progressive-disclosure state.
|
||
valid = {"role": "tool", "tool_call_id": "2", "content": "long output " * 40}
|
||
result = apply_to_messages([unsupported, valid], tokenizer)
|
||
assert result.messages[0] == snapshot
|
||
assert result.messages[1]["content"] == "short"
|
||
assert len(result.spans) == 1
|
||
assert interceptor.calls == 3
|
||
|
||
|
||
def test_single_openai_text_block_interceptor_preserves_metadata(tokenizer, monkeypatch):
|
||
from copy import deepcopy
|
||
|
||
class Shortener:
|
||
name = "metadata-regression"
|
||
|
||
def matches(self, *_):
|
||
return True
|
||
|
||
def transform(self, *_):
|
||
return "short"
|
||
|
||
monkeypatch.setattr("headroom.proxy.interceptors.base.INTERCEPTORS", [Shortener()])
|
||
message = {
|
||
"role": "tool",
|
||
"tool_call_id": "1",
|
||
"content": [
|
||
{
|
||
"type": "text",
|
||
"text": "long output " * 40,
|
||
"cache_control": {"type": "ephemeral", "ttl": "1h"},
|
||
"metadata": {"nested": [1, 2]},
|
||
}
|
||
],
|
||
}
|
||
original = deepcopy(message)
|
||
result = apply_to_messages([message], tokenizer)
|
||
expected = deepcopy(message)
|
||
expected["content"][0]["text"] = "short"
|
||
assert result.messages == [expected]
|
||
assert len(result.spans) == 1
|
||
assert result.spans[0].tokens_saved > 0
|
||
result.messages[0]["content"][0]["metadata"]["nested"].append(3)
|
||
assert message == original
|