mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description `OpenAICacheOptimizer._normalize_whitespace` measured a line's indentation, collapsed the line with `" ".join(stripped.split())` (which already drops the indentation), and then sliced `content_part[leading:]`. So every indented line in a system prompt lost its first `leading` characters: `" def f():"` became `" f():"`, `" return x"` became `" "`, and tab-indented lines lost characters and had their tabs turned into spaces. Unindented lines were fine, which is why the existing test didn't catch it. The fix keeps the line's original leading whitespace and appends the collapsed content: `line[:leading] + content_part`. Inner runs of spaces are still collapsed, trailing whitespace is still trimmed, and tabs in the indentation are now kept as tabs instead of being rewritten as spaces. Fixes #4336 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/openai.py`: in `_normalize_whitespace`, emit `line[:leading] + content_part` instead of `" " * leading + content_part[leading:]`. - `tests/test_cache/test_openai.py`: `test_whitespace_normalization_keeps_indented_lines_intact` checks that a space-indented code block round-trips unchanged, that a tab-indented block keeps its tabs while inner spaces are collapsed, and that the same system prompt comes out of `optimize()` unchanged. It fails on `main` and passes with the fix. - Follow-up from review: trim only surrounding newlines at the end of `_normalize_whitespace`, so a prompt that starts with an indented line keeps its indentation (`test_whitespace_normalization_keeps_first_line_indentation`). ## Testing - [x] Unit tests pass (`pytest`) - `tests/test_cache/`: 298 passed, 2 skipped - [x] Linting passes (`ruff check .`) - `ruff check` and `ruff format --check` on the two changed files - [x] Type checking passes (`mypy headroom`) - `mypy headroom/cache/openai.py`, no issues - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New test on `main` (fix stashed): ```text $ pytest tests/test_cache/test_openai.py -q -k indented E AssertionError: assert 'Example:\n f():' == 'Example:\n ... return x' E Example: E - def f(): E ? ---- - E + f(): E - return x ======================= 1 failed, 13 deselected in 0.26s ======================= ``` With the fix: ```text $ pytest tests/test_cache/test_openai.py -q -k indented ======================= 1 passed, 13 deselected in 0.24s ======================= $ pytest tests/test_cache/ -q ======================= 298 passed, 2 skipped in 15.08s ======================== ``` ## Real Behavior Proof - Environment: macOS 26.2, Python 3.14.7, repo deps via `uv sync --extra dev`, optimizer resolved the same way `HeadroomClient` does it (`CacheOptimizerRegistry.get("openai")`), default `CacheConfig`. - Exact command / steps: a small script that gets the `"openai"` optimizer from the registry and runs `optimize()` on a system prompt with a space-indented code block, a nested bullet and a tab-indented block: ```python from headroom.cache import OptimizationContext from headroom.cache.registry import CacheOptimizerRegistry opt = CacheOptimizerRegistry.get("openai") prompt = "You are a code reviewer. Follow this style:\n def handler(event):\n return process(event)\n - nested bullet\n\tif ok:\n\t\treturn x" res = opt.optimize([{"role": "system", "content": prompt}, {"role": "user", "content": "hi"}], OptimizationContext(provider="openai", model="gpt-4o")) print(repr(res.messages[0]["content"])) ``` - Observed result: indentation and text are preserved after the fix. Before and after output: Before (`main`): ```text 'You are a code reviewer. Follow this style:\n handler(event):\n rocess(event)\n nested bullet\n f ok:\n turn x' ``` After: ```text 'You are a code reviewer. Follow this style:\n def handler(event):\n return process(event)\n - nested bullet\n\tif ok:\n\t\treturn x' ``` Indentation and text are preserved; the double space in `nested bullet` and `return x` is still collapsed. - Not tested: a live request through `HeadroomClient` to the OpenAI API (no key used); the optimizer output above is exactly what that path sends upstream. The Anthropic and Google optimizers were not touched. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: yes, only in that indented system-prompt lines are no longer truncated. Tab indentation is now kept as tabs instead of being converted to spaces. - Kill switch / disable path: `CacheConfig(normalize_whitespace=False)` (unchanged) - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`, it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Comments and docs: the existing comments already describe the intended behavior ("Collapse multiple spaces in content (not indentation)"), so no new comment or doc change was needed. The cache prefix hash for prompts with indented lines will change once, since the normalized text is now correct. --------- Co-authored-by: Abelo9996 <Abelo9996@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
229 lines
8.1 KiB
Python
229 lines
8.1 KiB
Python
"""Tests for OpenAICacheOptimizer."""
|
|
|
|
import pytest
|
|
|
|
from headroom.cache import CacheConfig, OpenAICacheOptimizer, OptimizationContext
|
|
from headroom.cache.base import CacheStrategy
|
|
|
|
|
|
class TestOpenAICacheOptimizer:
|
|
"""Test OpenAICacheOptimizer functionality."""
|
|
|
|
@pytest.fixture
|
|
def optimizer(self):
|
|
"""Create optimizer instance."""
|
|
return OpenAICacheOptimizer()
|
|
|
|
@pytest.fixture
|
|
def context(self):
|
|
"""Create optimization context."""
|
|
return OptimizationContext(
|
|
provider="openai",
|
|
model="gpt-4",
|
|
)
|
|
|
|
def test_optimizer_properties(self, optimizer):
|
|
"""Test optimizer properties."""
|
|
assert optimizer.name == "openai-prefix-stabilizer"
|
|
assert optimizer.provider == "openai"
|
|
assert optimizer.strategy == CacheStrategy.PREFIX_STABILIZATION
|
|
|
|
def test_optimize_simple_messages(self, optimizer, context):
|
|
"""Test optimizing simple messages."""
|
|
messages = [
|
|
{"role": "system", "content": "You are a helpful assistant."},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
|
|
result = optimizer.optimize(messages, context)
|
|
|
|
assert result.messages is not None
|
|
assert len(result.messages) == 2
|
|
assert result.metrics.stable_prefix_hash != ""
|
|
|
|
def test_date_extraction(self, optimizer, context):
|
|
"""Test that dates are extracted from system prompt."""
|
|
messages = [
|
|
{
|
|
"role": "system",
|
|
"content": "Today is January 7, 2026. You are a helpful assistant.",
|
|
},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
|
|
result = optimizer.optimize(messages, context)
|
|
|
|
# Check that date was extracted and moved
|
|
system_content = result.messages[0]["content"]
|
|
# The date should be moved to a dynamic section at the end
|
|
assert "You are a helpful assistant" in system_content
|
|
|
|
def test_whitespace_normalization(self, optimizer, context):
|
|
"""Test whitespace normalization."""
|
|
messages = [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant.",
|
|
},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
|
|
result = optimizer.optimize(messages, context)
|
|
|
|
# Whitespace should be normalized
|
|
system_content = result.messages[0]["content"]
|
|
assert " " not in system_content # Multiple spaces collapsed
|
|
|
|
def test_whitespace_normalization_keeps_indented_lines_intact(self, optimizer, context):
|
|
"""Indented lines keep their indentation and their first characters (#4336)."""
|
|
code = "Example:\n def f():\n return x"
|
|
assert optimizer._normalize_whitespace(code) == code
|
|
|
|
tabbed = "Example:\n\tif ok:\n\t\treturn x"
|
|
assert optimizer._normalize_whitespace(tabbed) == "Example:\n\tif ok:\n\t\treturn x"
|
|
|
|
messages = [
|
|
{"role": "system", "content": code},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
result = optimizer.optimize(messages, context)
|
|
assert result.messages[0]["content"] == code
|
|
|
|
def test_whitespace_normalization_keeps_first_line_indentation(self, optimizer, context):
|
|
"""A prompt that starts with an indented line keeps that indentation."""
|
|
code = " if ready:\n run()"
|
|
assert optimizer._normalize_whitespace(code) == code
|
|
|
|
# Surrounding blank lines and trailing whitespace are still trimmed.
|
|
padded = "\n \n if ready:\n run() \n\n"
|
|
assert optimizer._normalize_whitespace(padded) == code
|
|
|
|
messages = [
|
|
{"role": "system", "content": code},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
result = optimizer.optimize(messages, context)
|
|
assert result.messages[0]["content"] == code
|
|
|
|
def test_optimize_disabled(self, context):
|
|
"""Test optimization when disabled."""
|
|
config = CacheConfig(enabled=False)
|
|
optimizer = OpenAICacheOptimizer(config)
|
|
|
|
messages = [
|
|
{"role": "system", "content": "Test"},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
|
|
result = optimizer.optimize(messages, context)
|
|
assert result.transforms_applied == []
|
|
|
|
def test_prefix_stability_tracking(self, optimizer, context):
|
|
"""Test that prefix stability is tracked."""
|
|
messages = [
|
|
{"role": "system", "content": "You are a helpful assistant."},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
|
|
# First call
|
|
optimizer.optimize(messages, context)
|
|
|
|
# Second call with same messages
|
|
result2 = optimizer.optimize(messages, context)
|
|
|
|
# Second call should detect stable prefix
|
|
assert result2.metrics.estimated_cache_hit is True
|
|
assert result2.metrics.prefix_changed_from_previous is False
|
|
|
|
def test_prefix_change_detection(self, optimizer, context):
|
|
"""Test detection of prefix changes."""
|
|
messages1 = [
|
|
{"role": "system", "content": "You are a helpful assistant."},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
messages2 = [
|
|
{"role": "system", "content": "You are a different assistant."},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
|
|
optimizer.optimize(messages1, context)
|
|
result2 = optimizer.optimize(messages2, context)
|
|
|
|
# Second call should detect prefix change
|
|
assert result2.metrics.prefix_changed_from_previous is True
|
|
|
|
def test_token_threshold_warning(self, optimizer, context):
|
|
"""Test warning when below token threshold."""
|
|
messages = [
|
|
{"role": "system", "content": "Short."},
|
|
{"role": "user", "content": "Hi"},
|
|
]
|
|
|
|
result = optimizer.optimize(messages, context)
|
|
|
|
# Should have warning about being below threshold
|
|
assert any("1024" in w for w in result.warnings)
|
|
|
|
def test_estimate_savings_below_threshold(self, optimizer, context):
|
|
"""Test savings estimation below threshold."""
|
|
messages = [
|
|
{"role": "system", "content": "Short system prompt."},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
|
|
savings = optimizer.estimate_savings(messages, context)
|
|
assert savings == 0.0 # Below threshold
|
|
|
|
def test_estimate_savings_above_threshold(self, optimizer, context):
|
|
"""Test savings estimation above threshold."""
|
|
messages = [
|
|
{"role": "system", "content": "You are helpful. " * 500},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
|
|
# First call to establish baseline
|
|
optimizer.optimize(messages, context)
|
|
|
|
# Second call should show savings
|
|
savings = optimizer.estimate_savings(messages, context)
|
|
assert savings > 0.0
|
|
|
|
def test_uuid_pattern_detection(self, optimizer, context):
|
|
"""Test detection of UUIDs in content."""
|
|
messages = [
|
|
{
|
|
"role": "system",
|
|
"content": "Request ID: 12345678-1234-1234-1234-123456789012. Be helpful.",
|
|
},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
|
|
result = optimizer.optimize(messages, context)
|
|
# UUID should be detected as dynamic content
|
|
assert result.metrics.stable_prefix_hash != ""
|
|
|
|
def test_content_block_format(self, optimizer, context):
|
|
"""Test handling of content block format."""
|
|
messages = [
|
|
{
|
|
"role": "system",
|
|
"content": [{"type": "text", "text": "You are helpful."}],
|
|
},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
|
|
result = optimizer.optimize(messages, context)
|
|
assert result.messages is not None
|
|
|
|
def test_metrics_recording(self, optimizer, context):
|
|
"""Test that metrics are recorded."""
|
|
messages = [
|
|
{"role": "system", "content": "You are helpful."},
|
|
{"role": "user", "content": "Hello!"},
|
|
]
|
|
|
|
optimizer.optimize(messages, context)
|
|
metrics = optimizer.get_metrics()
|
|
|
|
assert metrics.stable_prefix_hash != ""
|