Files
headroom/tests/test_content_router_compact_json.py
Matt Van HornandMatt Van Horn 545f544147 fix: preserve JSON object fields in SmartCrusher (#3648)
## Description

Change `crush_object` in `crushers.rs` to preserve every property
instead of treating independent object fields as interchangeable sampled
records; remove the key-selection algorithm and update its inline tests,
including the test that currently requires default-mode key loss. Keep
its existing production call in `SmartCrusher::process_value_with_hook`
in `crusher.rs`, which first recursively compresses nested values, so
legitimate large-array compaction and CCR-backed row offload remain
available without deleting their enclosing keys. In
`smart_crush_content_with_hook`, return the original input bytes with an
unmodified result when an object and its descendants were left
unchanged, covering whitespace and escaping without falsely advertising
compression; retain serialization and strategy attribution when
descendants actually change.

SmartCrusher removes the non-null `tags` and `subtasks` properties from
an MCP task response while retaining nulls and small fields, without
emitting a CCR marker or recovery hash. The issue contains complete
before/after JSON and a `/v1/compress` reproduction, followed by a
reporter retest on 0.37.0 confirming the same behavior. The full example
is 813 bytes before compression, with 299 request tokens reported; the
approximate 350-byte description and claimed 500-token default should
not be used as implementation constants. The clone's Python and Rust
defaults are 200 tokens, and `crush_object` uses a per-property estimate
including overhead, explaining how this object can cross its gate.

Closes #3634

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Change `crush_object` in `crushers.rs` to preserve every property
instead of treating independent object fields as interchangeable sampled
records; remove the key-selection algorithm and update its inline tests,
including the test that currently requires default-mode key loss. Keep
its existing production call in `SmartCrusher::process_value_with_hook`
in `crusher.rs`, which first recursively compresses nested values, so
legitimate large-array compaction and CCR-backed row offload remain
available without deleting their enclosing keys. In
`smart_crush_content_with_hook`, return the original input bytes with an
unmodified result when an object and its descendants were left
unchanged, covering whitespace and escaping without falsely advertising
compression; retain serialization and strategy attribution when
descendants actually change.

## Testing

- [ ] Unit tests pass (`pytest`)
  Not run in this workspace, so this is left for CI to confirm.
- [ ] Linting passes (`ruff check .`)
  Not run in this workspace, so this is left for CI to confirm.
- [ ] Type checking passes (`mypy headroom`)
  Not run in this workspace, so this is left for CI to confirm.
- [x] New tests added for new functionality
- [ ] Manual testing performed
Not verified: this needs a person on the named hardware or environment.
- Embed the complete issue JSON as a local fixture literal, preserving
its original order and encoded tags string. With default configuration,
all properties and nested task records survive; compact input remains
byte-identical, and no fake CCR hash or key-drop strategy is reported. -
Exercise wide objects above the existing floor with arbitrary names,
long strings, nested dictionaries, nulls, booleans, and empty arrays.

### Test Output

```text
All 28 required checks are green on e214a5e (18 workflows):
test (1) (2) (3) (4), test-extras, test-agno, test (ubuntu),
parity, audit, CodeQL, simulator e2e (ubuntu/macos/windows),
python-314-wheels (ubuntu/macos), wheels (x86_64-apple-darwin,
aarch64-apple-darwin, x86_64-unknown-linux-gnu).

Run: https://github.com/headroomlabs-ai/headroom/pull/3648/checks
test (1): https://github.com/headroomlabs-ai/headroom/actions/runs/35326929298/job/105542328074
```


## Real Behavior Proof

- Environment: GitHub Actions hosted runners (ubuntu-latest,
macos-latest, windows-latest) on e214a5e. Not run on local hardware.
- Exact command / steps: the repository's own CI matrix on this head;
the new inline tests in `crushers.rs` embed the complete issue JSON from
#3634 as a fixture literal and assert every property survives with
default configuration.
- Observed result: all properties and nested task records survive;
compact input remains byte-identical; no CCR hash or key-drop strategy
is reported when nothing changed. 28/28 checks green.
- Not tested: no manual `/v1/compress` reproduction against a running
server. The issue's own reproduction and the reporter's 0.37.0 retest
stand as the field evidence; I did not re-run them.


## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is not behind a rollout flag.
- Minimum rollout channel: n/a, ships with the code.
- Stable/default behavior changed: yes. `crush_object` no longer drops
object properties in default mode, and `smart_crush_content_with_hook`
returns the original bytes unmodified when nothing changed. That is the
bug fix; the prior behavior was silent data loss.
- Kill switch / disable path: none. Revert the commit to restore prior
behavior.
- Unsafe override required: no.
- Qualification impact: one existing inline test that asserted
default-mode key loss was updated, since it encoded the bug as expected
behavior.
- Rollback path: `git revert` of this PR's merge commit. No migration,
no persisted state, no config change.


## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
  Not run in this workspace, so this is left for CI to confirm.

## Screenshots (if applicable)



## Additional Notes

Nothing beyond what is described above.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-09-21 10:05:51 -07:00

323 lines
11 KiB
Python

"""Compact machine-generated JSON must not evade compression (token-estimate bug).
Whitespace-split token counting made a compact JSON payload (no spaces —
the default output of json.dumps with separators, JSON.stringify, boto3)
count as ~1 token, so compression ratios computed as ~1.0 and the
min_ratio gate rejected SmartCrusher's real output.
"""
from __future__ import annotations
import json
import random
import pytest
from headroom.transforms.content_router import (
ContentRouter,
ContentRouterConfig,
_estimate_tokens,
)
def test_estimate_tokens_monotone_on_compact_json() -> None:
small = json.dumps([{"a": 1}] * 5, separators=(",", ":"))
large = json.dumps([{"a": 1}] * 500, separators=(",", ":"))
assert _estimate_tokens(large) > _estimate_tokens(small) > 1
def test_compact_json_tool_result_compresses() -> None:
tokenizer = pytest.importorskip("headroom.tokenizers.estimator")
from headroom.tokenizer import Tokenizer
tok = Tokenizer(tokenizer.EstimatingTokenCounter())
random.seed(42)
rows = [
{
"serviceArn": f"arn:aws:ecs:us-east-1:123456789012:service/x/svc-{s:03d}",
"serviceName": f"svc-{s:03d}",
"status": "ACTIVE",
"desiredCount": random.randint(1, 6),
"runningCount": random.randint(0, 6),
}
for s in range(150)
]
payload = json.dumps(rows, separators=(",", ":"))
assert " " not in payload[:200] # genuinely compact
messages = [
{"role": "user", "content": "Investigate."},
{
"role": "assistant",
"content": [
{"type": "text", "text": "Checking."},
{"type": "tool_use", "id": "toolu_1", "name": "list_services", "input": {}},
],
},
{
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": "toolu_1", "content": payload}],
},
{"role": "user", "content": "Summarize in one sentence."},
]
router = ContentRouter(ContentRouterConfig(skip_user_messages=False))
before = tok.count_messages(messages)
result = router.apply(
[json.loads(json.dumps(m)) for m in messages],
tok,
context="Summarize",
frozen_message_count=0,
)
after = tok.count_messages(result.messages)
assert after < before * 0.9, (before, after, result.transforms_applied[:5])
# ---------------------------------------------------------------------------
# Issue #3634: SmartCrusher must not drop object fields (tags/subtasks)
# ---------------------------------------------------------------------------
# Exact MCP `saga-mcp_task_get` payload from the issue (813 bytes, original
# key order, tags encoded as a JSON string). Do not pretty-print or reorder.
MCP_TASK_GET_JSON = (
'{"id":79,"epic_id":9,"title":"test","description":null,'
'"status":"done","priority":"medium","sort_order":0,'
'"assigned_to":null,"estimated_hours":null,"actual_hours":null,'
'"due_date":null,"source_ref":null,"metadata":"{}",'
'"created_at":"2026-09-15 20:32:06","updated_at":"2026-09-16 11:59:27",'
'"description_locked":0,"is_deleted":0,"deleted_at":null,'
'"deleted_by":null,"delete_reason":null,"epic_name":"latency-routing",'
'"tags":"[\\"cherry-pick\\",\\"dedicated branch\\"]",'
'"subtasks":[{"id":163,"task_id":79,"title":"test2","status":"todo",'
'"sort_order":1,"created_at":"2026-09-15 20:32:13",'
'"updated_at":"2026-09-16 14:02:26"},'
'{"id":165,"task_id":79,"title":"test4","status":"todo","sort_order":2,'
'"created_at":"2026-09-15 20:38:47","updated_at":"2026-09-17 20:29:04"}],'
'"notes":[],"comments":[],"depends_on":[],"dependents":[]}'
)
_TOOL_CALL_ID = "call_task_get_79"
_TOOL_NAME = "saga-mcp_task_get"
def _offline_tokenizer():
tokenizer = pytest.importorskip("headroom.tokenizers.estimator")
from headroom.tokenizer import Tokenizer
return Tokenizer(tokenizer.EstimatingTokenCounter())
def _object_preserving_router() -> ContentRouter:
# Real ContentRouter + Rust SmartCrusher. Disable unrelated ML / log
# fallbacks so a no-savings crush cannot be rewritten by Kompress.
return ContentRouter(
ContentRouterConfig(
skip_user_messages=False,
enable_kompress=False,
enable_log_compressor=False,
enable_search_compressor=False,
relevance_split=False,
protect_analysis_context=False,
exclude_tools=set(),
)
)
def _openai_task_messages(payload: str) -> list[dict]:
return [
{"role": "user", "content": "read task 79"},
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": _TOOL_CALL_ID,
"type": "function",
"function": {"name": _TOOL_NAME, "arguments": '{"id":79}'},
}
],
},
{
"role": "tool",
"tool_call_id": _TOOL_CALL_ID,
"content": payload,
},
]
def _anthropic_task_messages(payload: str) -> list[dict]:
return [
{"role": "user", "content": "read task 79"},
{
"role": "assistant",
"content": [
{
"type": "tool_use",
"id": _TOOL_CALL_ID,
"name": _TOOL_NAME,
"input": {"id": 79},
}
],
},
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": _TOOL_CALL_ID,
"content": payload,
}
],
},
]
def _apply(messages: list[dict]):
router = _object_preserving_router()
tok = _offline_tokenizer()
return router.apply(
[json.loads(json.dumps(m)) for m in messages],
tok,
context="read task 79",
frozen_message_count=0,
protect_recent=0,
protect_analysis_context=False,
)
def _openai_tool_content(messages: list[dict]) -> str:
for msg in messages:
if msg.get("role") == "tool" and msg.get("tool_call_id") == _TOOL_CALL_ID:
content = msg.get("content")
assert isinstance(content, str), content
return content
raise AssertionError(f"OpenAI tool message {_TOOL_CALL_ID} missing")
def _anthropic_tool_content(messages: list[dict]) -> str:
for msg in messages:
content = msg.get("content")
if not isinstance(content, list):
continue
for block in content:
if block.get("type") == "tool_result" and block.get("tool_use_id") == _TOOL_CALL_ID:
inner = block.get("content")
assert isinstance(inner, str), inner
return inner
raise AssertionError(f"Anthropic tool_result {_TOOL_CALL_ID} missing")
def _assert_task_payload_preserved(original: str, compressed: str) -> dict:
assert "<<ccr:" not in compressed, compressed[:200]
parsed = json.loads(compressed)
expected = json.loads(original)
assert parsed == expected, (sorted(expected), sorted(parsed))
assert "tags" in parsed
assert "subtasks" in parsed
assert parsed["tags"] == '["cherry-pick","dedicated branch"]'
assert len(parsed["subtasks"]) == 2
assert parsed["subtasks"][0]["title"] == "test2"
assert parsed["subtasks"][1]["title"] == "test4"
return parsed
def _wide_renamed_task_json() -> str:
"""Renamed, oversized variant so a below-threshold skip cannot hide key drop.
Field names are arbitrary (not tags/subtasks) and expensive values sit at
the beginning, middle, and end so a boundary/stride selector cannot
accidentally keep them all.
"""
original = json.loads(MCP_TASK_GET_JSON)
expensive = {
"nested": {"label": "child", "note": "n" * 180},
"items": [],
"ok": True,
"missing": None,
}
wide: dict = {
"leading_blob": expensive,
"label_bundle": original.pop("tags"),
"child_records": original.pop("subtasks"),
}
wide.update(original)
for i in range(20):
wide[f"extra_field_{i:02d}"] = (
f"this is a relatively long value string for entry number {i} with content"
)
wide["trailing_blob"] = {
"nested": {"label": "tail", "note": "t" * 180},
"items": [],
"ok": False,
"missing": None,
}
payload = json.dumps(wide, separators=(",", ":"))
assert len(payload) > 1500
assert len(wide) > 8
return payload
def test_mcp_task_fixture_is_the_issue_payload() -> None:
assert len(MCP_TASK_GET_JSON) == 813
parsed = json.loads(MCP_TASK_GET_JSON)
assert list(parsed)[21] == "tags"
assert list(parsed)[22] == "subtasks"
assert parsed["tags"] == '["cherry-pick","dedicated branch"]'
def test_mcp_task_object_preserved_openai_tool_role() -> None:
messages = _openai_task_messages(MCP_TASK_GET_JSON)
result = _apply(messages)
content = _openai_tool_content(result.messages)
assert content == MCP_TASK_GET_JSON
_assert_task_payload_preserved(MCP_TASK_GET_JSON, content)
# Tool-call pairing stays intact.
tool_msg = next(m for m in result.messages if m.get("role") == "tool")
assistant = next(m for m in result.messages if m.get("role") == "assistant")
assert tool_msg["tool_call_id"] == _TOOL_CALL_ID
assert assistant["tool_calls"][0]["id"] == _TOOL_CALL_ID
assert assistant["tool_calls"][0]["function"]["name"] == _TOOL_NAME
assert not any("object:adaptive" in t for t in result.transforms_applied)
def test_mcp_task_object_preserved_anthropic_tool_result() -> None:
messages = _anthropic_task_messages(MCP_TASK_GET_JSON)
result = _apply(messages)
content = _anthropic_tool_content(result.messages)
assert content == MCP_TASK_GET_JSON
_assert_task_payload_preserved(MCP_TASK_GET_JSON, content)
assistant = next(m for m in result.messages if m.get("role") == "assistant")
assert assistant["content"][0]["id"] == _TOOL_CALL_ID
user_blocks = next(
m["content"]
for m in result.messages
if m.get("role") == "user" and isinstance(m.get("content"), list)
)
assert user_blocks[0]["tool_use_id"] == _TOOL_CALL_ID
assert not any("object:adaptive" in t for t in result.transforms_applied)
def test_wide_renamed_object_fields_preserved_openai_tool_role() -> None:
payload = _wide_renamed_task_json()
result = _apply(_openai_task_messages(payload))
content = _openai_tool_content(result.messages)
assert "<<ccr:" not in content
parsed = json.loads(content)
expected = json.loads(payload)
assert parsed == expected
assert "label_bundle" in parsed
assert "child_records" in parsed
assert "leading_blob" in parsed
assert "trailing_blob" in parsed
assert parsed["label_bundle"] == '["cherry-pick","dedicated branch"]'
assert len(parsed["child_records"]) == 2
assert parsed["leading_blob"]["nested"]["note"] == "n" * 180
assert parsed["trailing_blob"]["nested"]["note"] == "t" * 180
def test_malformed_json_tool_result_passes_through() -> None:
payload = "{" + "not-json, still a tool result. " * 40
assert len(payload) > 500
result = _apply(_openai_task_messages(payload))
content = _openai_tool_content(result.messages)
assert content == payload
assert "<<ccr:" not in content