mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
Python's grammar exposes the `;` between two statements on one physical
line as a root-level anonymous token. The leftover top-level sweep
captured it as a fragment, so assembly put a bare `;` on its own line:
```python
v=json.load(open(B+'v.json'))
;
m=json.load(open(B+'m.json'))
```
The result no longer parses, `compress()` discards it and returns the
original, and the proxy logs `Code compression produced invalid syntax
for python (...) returning original` for every such tool result. Two
live fenced blocks (220 and 117 tokens) hit this in the same proxy
session.
The line-based statement collection has a second symptom: when two
statements share one physical line (`x=1; y=2`), both AST nodes map to
the same row. Function bodies appended that row twice; class bodies
appended the full line twice, producing syntactically valid but wrong
output (`x=1; y=2` executed twice, compression ratio inflated to 1.429).
A third symptom surfaced once the output became valid: Python runs
top-level statements in order, but imports are always emitted first.
`import sys; sys.path.insert(0, 'vendor'); import plugin` now compressed
to valid code with `import plugin` hoisted above the path update, so the
plugin failed to load with `ModuleNotFoundError`.
Demoting such imports fixed ordering inside the top-level bucket, but
definitions are assembled before that bucket. `import sys; marker = 1;
import decimal` followed by `class Money(decimal.Decimal)` still moved
the class ahead of `import decimal` and raised `NameError` at definition
time.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Skip unnamed `;`/`,` children in the top-level leftover sweep so a
standalone separator can never become a code fragment.
- Merge overlapping row ranges in `_compress_function_ast` before the
budget loop. Taking the union (instead of dropping the later range)
preserves continuation lines of a multi-line statement that starts after
`;`.
- Track the last emitted row in `_compress_class_ast` and emit shared
rows once, trimming only rows already emitted so continuation lines
survive.
- For Python only, keep a root-level import behind executable top-level
code: the top-level sweep demotes such imports to the top-level code
stream (source order preserved) instead of hoisting them above the code
they depend on. Other languages are untouched because their imports/uses
are order-independent or must lead the file.
- For Python only, decline reassembly when executable top-level code
precedes any declaration. Definitions are emitted before the top-level
code bucket, so a class base, decorator, or default expression after
that code would be evaluated before the code or import it needs. Safe
reassembly cannot be guaranteed, so the original snippet is returned
unchanged. Snippets with no declarations keep compressing with demoted
imports in source order.
- Added `TestSemicolonSeparatedStatements` with seven regression tests:
top-level fallback, class-body duplication, class-body continuation,
function-body shared-line/continuation behavior, import-order
preservation that executes the compressed output, and executable
class-base and decorator/default ordering regressions.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run; this change only
touches untyped helpers
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q
109 passed in 5.22s
$ .venv/bin/ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!
$ .venv/bin/ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted
$ # same new tests against unpatched origin/main 5c8772b7 in a detached worktree
3 failed, 92 deselected in 0.69s
```
## Real Behavior Proof
- Environment: macOS 27.2 (Darwin, arm64), Python 3.12.11 from the repo
`.venv`, head `d581720d` based on main `31dcaf41`, both synced by the
reviewer without changing compressor source.
- Exact command / steps: ran `pytest
tests/test_transforms/test_code_compressor.py -q -k Semicolon` on a
detached worktree of unpatched `origin/main` and on this branch. Then
compressed the two real fenced Python tool results captured from
`~/.headroom/logs/proxy-8788.log` (220- and 117-token blocks) with
`CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=1))`,
and parsed each output with `ast.parse`. For the import-order case,
wrote a module to a temp `vendor/` directory, compressed `import sys;
sys.path.insert(0, '<vendor>'); import <module>`, and executed the
output. For the cross-bucket case, compressed the reviewer's `import
sys; marker = 1; import decimal\nclass Money(decimal.Decimal):\n pass\n`
and a decorator/default variant using `functools.lru_cache` with
`value=marker`, then executed both. Re-ran both counterexamples on the
same 0.42.0 compressor build carrying the import-demotion patch without
the fallback to confirm they reproduced.
- Observed result: on unpatched `5c8772b7` all three original semicolon
tests fail — top-level compression returns the unchanged original after
`Code compression produced invalid syntax for python (35 tokens)`, the
class sample emits `x=1; y=2` twice, and the function sample emits the
shared line twice. On `d581720d` all seven pass; full file 109 passed.
For the 220-token live sample: before the fix it fell back to the
original after the warning; after, ratio `0.995`, no bare `;` line,
`ast.parse` succeeds. The 117-token sample is valid with no bare `;`
(ratio `1.0`, so the router still skips it for low savings). Class
sample: `x=1; y=2` count `2 -> 1`, ratio `1.429 -> 1.0`. Function
sample: shared line count `2 -> 1` and continuation line count `2 -> 1`.
Import-order sample: before the import fix the output was `import sys /
import semicolon_order_plugin / (blank) / sys.path.insert(...)` and
`exec` raised `ModuleNotFoundError: No module named
'semicolon_order_plugin'`; after, the order is `import sys`,
`sys.path.insert(...)`, `import semicolon_order_plugin` and `exec`
succeeds. Cross-bucket samples: before the fallback the class variant
emitted `import sys / class Money(decimal.Decimal) / marker = 1 / import
decimal` and `exec` raised `NameError: name 'decimal' is not defined`;
the decorator variant raised `NameError: name 'functools' is not
defined` at decoration time. After the fallback both return the original
snippet byte-for-byte and `exec` succeeds (`Money` is a
`decimal.Decimal` subclass, `compute()` returns `1`). A semicolon
snippet without declarations (`import sys; sys.path.insert(...); import
json`) still compresses, emitting `import sys`, then the path update,
then `import json`.
- Not tested: the Rust core port
(`crates/headroom-core/src/transforms/code_compressor.rs`) has the same
structure and was not changed; end-to-end proxy traffic was not rerun on
this branch; non-Python fixtures for shared lines were not added (the
range merge is mechanical, but only Python exposes `;` as a root token,
and only Python demotes imports or applies the ordering fallback).
Function output after the fix (shared line and continuation each appear
once):
```python
def f():
x=1; y=(2 +
3)
a=3
b=4
c=5
# [3 lines omitted]
pass
```
## Runtime Rollout Safety
- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: content with semicolon-separated
statements is compressed instead of falling back to the original; shared
lines are emitted once instead of duplicated; Python imports that follow
executable top-level code stay after it; Python snippets where
executable code precedes a definition are returned unchanged instead of
being reordered
- Kill switch / disable path: existing syntax validation and
`HEADROOM_CODE_SYNTAX_BREAKER=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
The top-level separator was only observed for Python: tree-sitter-python
exposes the assignments/calls on either side of `;` as direct module
children, with the `;` itself as an anonymous root sibling. The same
line-based collection is shared by the other grammars, so the range
merge also protects them, but no non-Python fixture was added.
Import demotion is Python-only and only applies after the sweep has
already emitted executable top-level code. Imports before any executable
code keep their place in the hoisted import block, so module docstrings
and license headers are unaffected. `_verify_syntax` also runs
`compile()`, which rejects misplaced `__future__` imports, so a demotion
that violated that rule would fall back to the original instead of
emitting broken ordering.
The ordering fallback is Python-only and is deliberately conservative:
any declaration after executable top-level code disables AST reassembly
for the snippet, even when the definition happens not to reference the
preceding code. The cost is lost compression for that shape; the
alternative is a semantically reordered result that syntax validation
cannot detect.
The Rust core port mirrors these functions (`extract_structure`, body
statement collection, class-body reconstruction). Parity there would
need its own fixtures and is out of scope for this PR.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2574 lines
92 KiB
Python
2574 lines
92 KiB
Python
"""Tests for Code-Aware Compressor using tree-sitter AST parsing.
|
|
|
|
Comprehensive tests covering:
|
|
- CodeCompressorConfig: Configuration validation and defaults
|
|
- CodeAwareCompressor: Core AST-based compression functionality
|
|
- Language detection: Auto-detection from extensions and content
|
|
- Transform interface: apply(), should_apply() methods
|
|
- Syntax preservation: Guarantees valid output syntax
|
|
- Edge cases: Empty content, unavailable dependency, fallbacks
|
|
"""
|
|
|
|
import decimal
|
|
import sys
|
|
import textwrap
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
|
|
import headroom.transforms.code_compressor as cc
|
|
from headroom.cache.compression_store import (
|
|
CompressionStore,
|
|
clear_request_compression_store,
|
|
set_request_compression_store,
|
|
)
|
|
from headroom.transforms.code_compressor import (
|
|
CodeAwareCompressor,
|
|
CodeCompressionResult,
|
|
CodeCompressorConfig,
|
|
CodeLanguage,
|
|
DocstringMode,
|
|
coerce_language,
|
|
detect_language,
|
|
is_tree_sitter_available,
|
|
is_tree_sitter_loaded,
|
|
unload_tree_sitter,
|
|
)
|
|
|
|
# Try to import for availability check
|
|
try:
|
|
import tree_sitter_language_pack # noqa: F401
|
|
|
|
TREE_SITTER_INSTALLED = True
|
|
except ImportError:
|
|
TREE_SITTER_INSTALLED = False
|
|
|
|
|
|
# =============================================================================
|
|
# Test Fixtures
|
|
# =============================================================================
|
|
|
|
|
|
@pytest.fixture
|
|
def default_config():
|
|
"""Default CodeCompressorConfig for testing."""
|
|
return CodeCompressorConfig(
|
|
min_tokens_for_compression=10, # Low threshold for tests
|
|
enable_ccr=False, # Disable CCR for unit tests
|
|
)
|
|
|
|
|
|
@pytest.fixture
|
|
def compressor(default_config):
|
|
"""CodeAwareCompressor instance with default config."""
|
|
return CodeAwareCompressor(default_config)
|
|
|
|
|
|
@pytest.fixture
|
|
def tokenizer():
|
|
"""Get a tokenizer for Transform interface tests."""
|
|
from headroom.providers import OpenAIProvider
|
|
from headroom.tokenizer import Tokenizer
|
|
|
|
provider = OpenAIProvider()
|
|
token_counter = provider.get_token_counter("gpt-4o")
|
|
return Tokenizer(token_counter, "gpt-4o")
|
|
|
|
|
|
# =============================================================================
|
|
# Test Data Generators
|
|
# =============================================================================
|
|
|
|
|
|
def generate_python_code(n_functions: int = 5, n_classes: int = 1) -> str:
|
|
"""Generate Python code for testing."""
|
|
lines = [
|
|
'"""Module with classes and functions."""',
|
|
"",
|
|
"import os",
|
|
"import sys",
|
|
"from typing import Any, Optional, List",
|
|
"from dataclasses import dataclass",
|
|
"",
|
|
]
|
|
|
|
for c in range(n_classes):
|
|
lines.extend(
|
|
[
|
|
"@dataclass",
|
|
f"class TestClass{c}:",
|
|
' """A test class with docstring."""',
|
|
" name: str",
|
|
" value: int = 0",
|
|
"",
|
|
" def method(self, arg: Any) -> str:",
|
|
' """Process the argument."""',
|
|
" result = str(arg)",
|
|
" for i in range(10):",
|
|
' result += f"iteration {i}"',
|
|
" return result",
|
|
"",
|
|
]
|
|
)
|
|
|
|
for i in range(n_functions):
|
|
lines.extend(
|
|
[
|
|
f"def function_{i}(arg: Any, optional: Optional[str] = None) -> str:",
|
|
f' """Process argument {i}.',
|
|
"",
|
|
" This is a longer docstring with multiple lines.",
|
|
" It explains what the function does in detail.",
|
|
"",
|
|
" Args:",
|
|
" arg: The argument to process.",
|
|
" optional: An optional parameter.",
|
|
"",
|
|
" Returns:",
|
|
" A string result.",
|
|
' """',
|
|
" result = str(arg)",
|
|
" if optional:",
|
|
" result += optional",
|
|
" for i in range(10):",
|
|
' result += f"iteration {i}"',
|
|
" try:",
|
|
" int(result)",
|
|
" except ValueError:",
|
|
' result = "0"',
|
|
" return result",
|
|
"",
|
|
]
|
|
)
|
|
|
|
return "\n".join(lines)
|
|
|
|
|
|
def generate_javascript_code(n_functions: int = 5) -> str:
|
|
"""Generate JavaScript code for testing."""
|
|
lines = [
|
|
"// Module with various functions",
|
|
'import { something } from "module";',
|
|
'const config = require("./config");',
|
|
"",
|
|
]
|
|
|
|
for i in range(n_functions):
|
|
lines.extend(
|
|
[
|
|
"/**",
|
|
f" * Process function {i}",
|
|
" * @param {any} arg - The argument",
|
|
" * @returns {string} The result",
|
|
" */",
|
|
f"function processFunction{i}(arg) {{",
|
|
" let result = String(arg);",
|
|
" for (let j = 0; j < 10; j++) {",
|
|
" result += `iteration ${j}`;",
|
|
" }",
|
|
" try {",
|
|
" JSON.parse(result);",
|
|
" } catch (e) {",
|
|
" console.error(e);",
|
|
" }",
|
|
" return result;",
|
|
"}",
|
|
"",
|
|
]
|
|
)
|
|
|
|
lines.append("export { processFunction0 };")
|
|
return "\n".join(lines)
|
|
|
|
|
|
def generate_go_code(n_functions: int = 3) -> str:
|
|
"""Generate Go code for testing."""
|
|
lines = [
|
|
"package main",
|
|
"",
|
|
'import "fmt"',
|
|
"",
|
|
"// Config holds configuration",
|
|
"type Config struct {",
|
|
" Name string",
|
|
" Value int",
|
|
"}",
|
|
"",
|
|
]
|
|
|
|
for i in range(n_functions):
|
|
lines.extend(
|
|
[
|
|
f"// Process{i} processes the input",
|
|
f"func Process{i}(input string) (string, error) {{",
|
|
" result := input",
|
|
" for i := 0; i < 10; i++ {",
|
|
' result = fmt.Sprintf("%s-%d", result, i)',
|
|
" }",
|
|
" if len(result) == 0 {",
|
|
' return "", fmt.Errorf("empty result")',
|
|
" }",
|
|
" return result, nil",
|
|
"}",
|
|
"",
|
|
]
|
|
)
|
|
|
|
return "\n".join(lines)
|
|
|
|
|
|
@pytest.mark.skipif(not TREE_SITTER_INSTALLED, reason="tree-sitter not installed")
|
|
@pytest.mark.parametrize(
|
|
("language", "code", "comment_prefix"),
|
|
[
|
|
pytest.param("python", generate_python_code(), "#", id="python"),
|
|
pytest.param("javascript", generate_javascript_code(), "//", id="javascript"),
|
|
pytest.param("typescript", generate_javascript_code(), "//", id="typescript"),
|
|
pytest.param("go", generate_go_code(), "//", id="go"),
|
|
],
|
|
)
|
|
def test_ccr_marker_preserves_code_syntax(language, code, comment_prefix):
|
|
"""CCR metadata must be a comment in the final output, not just before storage."""
|
|
store = CompressionStore()
|
|
set_request_compression_store(store)
|
|
try:
|
|
compressor = CodeAwareCompressor(
|
|
CodeCompressorConfig(
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=True,
|
|
fallback_to_kompress=False,
|
|
)
|
|
)
|
|
result = compressor.compress(code, language=language)
|
|
|
|
assert result.cache_key is not None
|
|
assert result.syntax_valid
|
|
# Parse the returned source, including the marker appended after the
|
|
# compressor's internal syntax check (#4478, item 4).
|
|
assert compressor._verify_syntax(result.compressed, CodeLanguage(language))
|
|
marker = result.compressed.splitlines()[-1]
|
|
assert marker.startswith(f"{comment_prefix} [")
|
|
assert f"Retrieve more: hash={result.cache_key}." in marker
|
|
assert "Expires in 5m.]" in marker
|
|
entry = store.retrieve(result.cache_key)
|
|
assert entry is not None
|
|
assert entry.original_content == code
|
|
finally:
|
|
clear_request_compression_store()
|
|
|
|
|
|
# =============================================================================
|
|
# TestCodeCompressorConfig
|
|
# =============================================================================
|
|
|
|
|
|
class TestCodeCompressorConfig:
|
|
"""Tests for CodeCompressorConfig dataclass."""
|
|
|
|
def test_default_values(self):
|
|
"""Default config values are sensible."""
|
|
config = CodeCompressorConfig()
|
|
|
|
assert config.preserve_imports is True
|
|
assert config.preserve_signatures is True
|
|
assert config.preserve_type_annotations is True
|
|
assert config.preserve_decorators is True
|
|
assert config.docstring_mode == DocstringMode.FIRST_LINE
|
|
assert config.target_compression_rate == 0.2
|
|
assert config.max_body_lines == 5
|
|
assert config.min_tokens_for_compression == 100
|
|
assert config.enable_ccr is True
|
|
|
|
def test_custom_values(self):
|
|
"""Custom config values are applied."""
|
|
config = CodeCompressorConfig(
|
|
preserve_imports=False,
|
|
preserve_signatures=True,
|
|
docstring_mode=DocstringMode.FULL,
|
|
target_compression_rate=0.3,
|
|
max_body_lines=10,
|
|
min_tokens_for_compression=50,
|
|
)
|
|
|
|
assert config.preserve_imports is False
|
|
assert config.preserve_signatures is True
|
|
assert config.docstring_mode == DocstringMode.FULL
|
|
assert config.target_compression_rate == 0.3
|
|
assert config.max_body_lines == 10
|
|
assert config.min_tokens_for_compression == 50
|
|
|
|
def test_docstring_modes(self):
|
|
"""All docstring modes are valid."""
|
|
for mode in DocstringMode:
|
|
config = CodeCompressorConfig(docstring_mode=mode)
|
|
assert config.docstring_mode == mode
|
|
|
|
|
|
# =============================================================================
|
|
# TestCodeCompressionResult
|
|
# =============================================================================
|
|
|
|
|
|
class TestCodeCompressionResult:
|
|
"""Tests for CodeCompressionResult dataclass."""
|
|
|
|
def test_tokens_saved(self):
|
|
"""tokens_saved property calculates correctly."""
|
|
result = CodeCompressionResult(
|
|
compressed="short",
|
|
original="long content here",
|
|
original_tokens=100,
|
|
compressed_tokens=30,
|
|
compression_ratio=0.3,
|
|
language=CodeLanguage.PYTHON,
|
|
syntax_valid=True,
|
|
)
|
|
|
|
assert result.tokens_saved == 70
|
|
|
|
def test_tokens_saved_no_negative(self):
|
|
"""tokens_saved never returns negative."""
|
|
result = CodeCompressionResult(
|
|
compressed="expanded",
|
|
original="short",
|
|
original_tokens=10,
|
|
compressed_tokens=20,
|
|
compression_ratio=2.0,
|
|
language=CodeLanguage.PYTHON,
|
|
syntax_valid=True,
|
|
)
|
|
|
|
assert result.tokens_saved == 0
|
|
|
|
def test_savings_percentage(self):
|
|
"""savings_percentage property calculates correctly."""
|
|
result = CodeCompressionResult(
|
|
compressed="short",
|
|
original="long content",
|
|
original_tokens=100,
|
|
compressed_tokens=25,
|
|
compression_ratio=0.25,
|
|
language=CodeLanguage.PYTHON,
|
|
syntax_valid=True,
|
|
)
|
|
|
|
assert result.savings_percentage == 75.0
|
|
|
|
def test_savings_percentage_zero_original(self):
|
|
"""savings_percentage handles zero original tokens."""
|
|
result = CodeCompressionResult(
|
|
compressed="",
|
|
original="",
|
|
original_tokens=0,
|
|
compressed_tokens=0,
|
|
compression_ratio=1.0,
|
|
language=CodeLanguage.UNKNOWN,
|
|
syntax_valid=True,
|
|
)
|
|
|
|
assert result.savings_percentage == 0.0
|
|
|
|
|
|
# =============================================================================
|
|
# TestCodeLanguage
|
|
# =============================================================================
|
|
|
|
|
|
class TestCodeLanguage:
|
|
"""Tests for CodeLanguage enum and detection."""
|
|
|
|
def test_all_language_values_are_unique(self):
|
|
"""All language enum values are unique."""
|
|
values = [lang.value for lang in CodeLanguage]
|
|
assert len(values) == len(set(values))
|
|
|
|
def test_detect_python_language(self):
|
|
"""Python language is detected from code patterns."""
|
|
|
|
code = """
|
|
import os
|
|
from typing import List
|
|
|
|
def function(arg: str) -> str:
|
|
return arg
|
|
|
|
class MyClass:
|
|
pass
|
|
"""
|
|
lang, confidence = detect_language(code)
|
|
assert lang == CodeLanguage.PYTHON
|
|
assert confidence > 0.5
|
|
|
|
def test_detect_javascript_language(self):
|
|
"""JavaScript language is detected from code patterns."""
|
|
|
|
code = """
|
|
const express = require('express');
|
|
import { something } from 'module';
|
|
|
|
function handler(req, res) {
|
|
return res.json({ status: 'ok' });
|
|
}
|
|
|
|
export default handler;
|
|
"""
|
|
lang, confidence = detect_language(code)
|
|
assert lang in (CodeLanguage.JAVASCRIPT, CodeLanguage.TYPESCRIPT)
|
|
assert confidence > 0.3
|
|
|
|
def test_detect_go_language(self):
|
|
"""Go language is detected from code patterns."""
|
|
|
|
code = """
|
|
package main
|
|
|
|
import "fmt"
|
|
|
|
func main() {
|
|
fmt.Println("Hello")
|
|
}
|
|
"""
|
|
lang, confidence = detect_language(code)
|
|
assert lang == CodeLanguage.GO
|
|
assert confidence > 0.3
|
|
|
|
|
|
# =============================================================================
|
|
# TestCodeAwareCompressor
|
|
# =============================================================================
|
|
|
|
|
|
class TestCodeAwareCompressor:
|
|
"""Tests for CodeAwareCompressor core functionality."""
|
|
|
|
def test_init_with_default_config(self):
|
|
"""Compressor initializes with default config."""
|
|
compressor = CodeAwareCompressor()
|
|
|
|
assert compressor.config is not None
|
|
assert compressor.config.preserve_imports is True
|
|
|
|
def test_init_with_custom_config(self, default_config):
|
|
"""Compressor initializes with custom config."""
|
|
compressor = CodeAwareCompressor(default_config)
|
|
|
|
assert compressor.config == default_config
|
|
|
|
def test_compress_skips_small_content(self, compressor):
|
|
"""Small content is not compressed."""
|
|
small_code = "def f(): pass"
|
|
result = compressor.compress(small_code)
|
|
|
|
assert result.compressed == small_code
|
|
assert result.compression_ratio == 1.0
|
|
|
|
def test_compress_handles_empty_content(self, compressor):
|
|
"""Empty content returns empty result."""
|
|
result = compressor.compress("")
|
|
|
|
assert result.compressed == ""
|
|
assert result.compression_ratio == 1.0
|
|
assert result.syntax_valid is True
|
|
|
|
def test_compress_with_explicit_language(self, compressor):
|
|
"""Language can be specified explicitly."""
|
|
code = generate_python_code(2)
|
|
result = compressor.compress(code, language="python")
|
|
|
|
# Should detect or use the specified language
|
|
assert result.language == CodeLanguage.PYTHON or result.language == CodeLanguage.UNKNOWN
|
|
|
|
def test_compress_auto_detects_python(self, compressor):
|
|
"""Python code is auto-detected during compression."""
|
|
code = """
|
|
import os
|
|
from typing import List
|
|
|
|
def function(arg: str) -> List[str]:
|
|
return [arg]
|
|
|
|
class MyClass:
|
|
pass
|
|
"""
|
|
result = compressor.compress(code)
|
|
# Should detect Python (if tree-sitter available) or return UNKNOWN
|
|
assert result.language in (CodeLanguage.PYTHON, CodeLanguage.UNKNOWN)
|
|
|
|
def test_compress_auto_detects_javascript(self, compressor):
|
|
"""JavaScript code is auto-detected during compression."""
|
|
code = """
|
|
const express = require('express');
|
|
import { something } from 'module';
|
|
|
|
function handler(req, res) {
|
|
return res.json({ status: 'ok' });
|
|
}
|
|
|
|
export default handler;
|
|
"""
|
|
result = compressor.compress(code)
|
|
assert result.language in (
|
|
CodeLanguage.JAVASCRIPT,
|
|
CodeLanguage.TYPESCRIPT,
|
|
CodeLanguage.UNKNOWN,
|
|
)
|
|
|
|
def test_compress_auto_detects_go(self, compressor):
|
|
"""Go code is auto-detected during compression."""
|
|
code = """
|
|
package main
|
|
|
|
import "fmt"
|
|
|
|
func main() {
|
|
fmt.Println("Hello")
|
|
}
|
|
"""
|
|
result = compressor.compress(code)
|
|
assert result.language in (CodeLanguage.GO, CodeLanguage.UNKNOWN)
|
|
|
|
|
|
# =============================================================================
|
|
# TestFallbackCompression
|
|
# =============================================================================
|
|
|
|
|
|
class TestFallbackCompression:
|
|
"""Tests for fallback compression when tree-sitter unavailable."""
|
|
|
|
def test_fallback_when_tree_sitter_unavailable(self, default_config):
|
|
"""Uses fallback compression when tree-sitter is not installed."""
|
|
with patch(
|
|
"headroom.transforms.code_compressor._check_tree_sitter_available",
|
|
return_value=False,
|
|
):
|
|
compressor = CodeAwareCompressor(default_config)
|
|
code = generate_python_code(5)
|
|
|
|
result = compressor.compress(code)
|
|
|
|
# Should still return a result (fallback compression)
|
|
assert result is not None
|
|
# Kompress fallback does NOT guarantee syntax validity
|
|
# If Kompress is unavailable, returns original (valid)
|
|
# If Kompress IS available, syntax_valid=False (cannot guarantee)
|
|
|
|
def test_fallback_preserves_structure(self, default_config):
|
|
"""Fallback compression preserves basic structure when no compressor available.
|
|
|
|
When both tree-sitter and Kompress are unavailable, the fallback
|
|
returns the original code unchanged - preserving all structure.
|
|
"""
|
|
with (
|
|
patch(
|
|
"headroom.transforms.code_compressor._check_tree_sitter_available",
|
|
return_value=False,
|
|
),
|
|
patch(
|
|
"headroom.transforms.kompress_compressor.is_kompress_available",
|
|
return_value=False,
|
|
),
|
|
):
|
|
compressor = CodeAwareCompressor(default_config)
|
|
code = generate_python_code(3)
|
|
|
|
result = compressor.compress(code)
|
|
|
|
# With no compressor available, original code is returned unchanged
|
|
# This preserves all imports and class/function signatures
|
|
assert "import os" in result.compressed
|
|
assert "def function_" in result.compressed
|
|
# Compression ratio should be 1.0 (no compression)
|
|
assert result.compression_ratio == 1.0
|
|
|
|
|
|
# =============================================================================
|
|
# TestTransformInterface
|
|
# =============================================================================
|
|
|
|
|
|
class TestTransformInterface:
|
|
"""Tests for Transform interface (apply, should_apply)."""
|
|
|
|
def test_should_apply_returns_false_for_small_content(self, default_config, tokenizer):
|
|
"""should_apply returns False for small content."""
|
|
config = CodeCompressorConfig(min_tokens_for_compression=1000)
|
|
compressor = CodeAwareCompressor(config)
|
|
messages = [{"role": "user", "content": "def f(): pass"}]
|
|
|
|
assert not compressor.should_apply(messages, tokenizer)
|
|
|
|
def test_should_apply_returns_bool_for_large_code(self, default_config, tokenizer):
|
|
"""should_apply returns boolean for large code content."""
|
|
compressor = CodeAwareCompressor(default_config)
|
|
code = generate_python_code(20)
|
|
messages = [{"role": "tool", "tool_call_id": "call_1", "content": code}]
|
|
|
|
# Should return True if there's code content to process
|
|
result = compressor.should_apply(messages, tokenizer)
|
|
assert isinstance(result, bool)
|
|
|
|
def test_apply_returns_transform_result(self, default_config, tokenizer):
|
|
"""apply() returns proper TransformResult."""
|
|
compressor = CodeAwareCompressor(default_config)
|
|
code = generate_python_code(10)
|
|
messages = [{"role": "tool", "tool_call_id": "call_1", "content": code}]
|
|
|
|
result = compressor.apply(messages, tokenizer)
|
|
|
|
assert result is not None
|
|
assert result.tokens_before > 0
|
|
assert len(result.messages) == 1
|
|
|
|
def test_apply_passes_through_non_code_messages(self, default_config, tokenizer):
|
|
"""apply() passes through non-code messages unchanged."""
|
|
compressor = CodeAwareCompressor(default_config)
|
|
messages = [
|
|
{"role": "user", "content": "Hello"},
|
|
{"role": "assistant", "content": "Hi there!"},
|
|
]
|
|
|
|
result = compressor.apply(messages, tokenizer)
|
|
|
|
assert result.messages[0]["content"] == "Hello"
|
|
assert result.messages[1]["content"] == "Hi there!"
|
|
|
|
def test_name_property(self, compressor):
|
|
"""Compressor has correct name."""
|
|
assert compressor.name == "code_aware_compressor"
|
|
|
|
|
|
# =============================================================================
|
|
# TestEdgeCases
|
|
# =============================================================================
|
|
|
|
|
|
class TestEdgeCases:
|
|
"""Edge case tests for CodeAwareCompressor."""
|
|
|
|
def test_whitespace_only_content(self, compressor):
|
|
"""Whitespace-only content is handled gracefully."""
|
|
result = compressor.compress(" \n\t\n ")
|
|
|
|
assert result.compression_ratio == 1.0
|
|
assert result.syntax_valid is True
|
|
|
|
def test_unicode_content(self, default_config):
|
|
"""Unicode in code is handled correctly."""
|
|
compressor = CodeAwareCompressor(default_config)
|
|
code = '''
|
|
def greet(name: str) -> str:
|
|
"""Greet the user in multiple languages."""
|
|
return f"Hello, {name}! \u4f60\u597d! \u3053\u3093\u306b\u3061\u306f!"
|
|
'''
|
|
result = compressor.compress(code)
|
|
|
|
# Should handle unicode without crashing
|
|
assert result is not None
|
|
|
|
def test_very_long_function(self, default_config):
|
|
"""Very long functions are compressed."""
|
|
compressor = CodeAwareCompressor(default_config)
|
|
lines = ["def very_long_function():"]
|
|
lines.append(' """A very long function."""')
|
|
for i in range(100):
|
|
lines.append(f" x_{i} = {i}")
|
|
lines.append(" return x_99")
|
|
code = "\n".join(lines)
|
|
|
|
result = compressor.compress(code)
|
|
|
|
# Should compress the long function body
|
|
assert result.compression_ratio < 1.0 or "tree_sitter" not in str(
|
|
is_tree_sitter_available()
|
|
)
|
|
|
|
def test_nested_functions(self, default_config):
|
|
"""Nested functions are handled."""
|
|
compressor = CodeAwareCompressor(default_config)
|
|
code = """
|
|
def outer():
|
|
def inner():
|
|
return "inner"
|
|
return inner()
|
|
"""
|
|
result = compressor.compress(code)
|
|
|
|
assert result is not None
|
|
# syntax_valid requires tree-sitter; without it, validation is skipped
|
|
if is_tree_sitter_available():
|
|
assert result.syntax_valid is True
|
|
|
|
def test_syntax_errors_in_input(self, default_config):
|
|
"""Syntax errors in input don't crash the compressor."""
|
|
compressor = CodeAwareCompressor(default_config)
|
|
# Invalid Python syntax
|
|
code = """
|
|
def broken(
|
|
# Missing closing paren
|
|
"""
|
|
# Should not raise
|
|
result = compressor.compress(code, language="python")
|
|
assert result is not None
|
|
|
|
def test_mixed_language_content(self, default_config):
|
|
"""Mixed language content (like markdown with code) is handled."""
|
|
compressor = CodeAwareCompressor(default_config)
|
|
content = """
|
|
# Documentation
|
|
|
|
Here is some code:
|
|
|
|
```python
|
|
def example():
|
|
pass
|
|
```
|
|
|
|
And some more text.
|
|
"""
|
|
# Should not crash
|
|
result = compressor.compress(content)
|
|
assert result is not None
|
|
|
|
|
|
# =============================================================================
|
|
# TestMemoryManagement
|
|
# =============================================================================
|
|
|
|
|
|
class TestMemoryManagement:
|
|
"""Tests for memory management functions."""
|
|
|
|
def test_is_tree_sitter_available_returns_bool(self):
|
|
"""is_tree_sitter_available returns a boolean."""
|
|
result = is_tree_sitter_available()
|
|
assert isinstance(result, bool)
|
|
|
|
def test_is_tree_sitter_loaded_returns_false_initially(self):
|
|
"""is_tree_sitter_loaded returns False when no parsers loaded."""
|
|
# Clear any loaded parsers first
|
|
unload_tree_sitter()
|
|
assert is_tree_sitter_loaded() is False
|
|
|
|
def test_unload_returns_false_when_nothing_loaded(self):
|
|
"""unload_tree_sitter returns False when nothing to unload."""
|
|
# Ensure nothing is loaded
|
|
unload_tree_sitter()
|
|
result = unload_tree_sitter()
|
|
assert result is False
|
|
|
|
|
|
# =============================================================================
|
|
# Integration Tests (only run if tree-sitter is installed)
|
|
# =============================================================================
|
|
|
|
|
|
@pytest.mark.skipif(not TREE_SITTER_INSTALLED, reason="tree-sitter-languages not installed")
|
|
class TestTreeSitterIntegration:
|
|
"""Integration tests that require actual tree-sitter installation.
|
|
|
|
These tests verify actual AST parsing and compression behavior.
|
|
"""
|
|
|
|
def test_actual_python_compression(self):
|
|
"""Test actual compression of Python code."""
|
|
config = CodeCompressorConfig(
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
code = generate_python_code(5)
|
|
|
|
result = compressor.compress(code, language="python")
|
|
|
|
# Should achieve compression
|
|
assert result.compression_ratio < 1.0
|
|
assert result.syntax_valid is True
|
|
assert result.language == CodeLanguage.PYTHON
|
|
|
|
def test_actual_javascript_compression(self):
|
|
"""Test actual compression of JavaScript code."""
|
|
config = CodeCompressorConfig(
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
code = generate_javascript_code(5)
|
|
|
|
result = compressor.compress(code, language="javascript")
|
|
|
|
assert result.compression_ratio < 1.0
|
|
assert result.syntax_valid is True
|
|
assert result.language == CodeLanguage.JAVASCRIPT
|
|
|
|
def test_actual_go_compression(self):
|
|
"""Test actual compression of Go code."""
|
|
config = CodeCompressorConfig(
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
code = generate_go_code(3)
|
|
|
|
result = compressor.compress(code, language="go")
|
|
|
|
assert result.compression_ratio < 1.0
|
|
assert result.syntax_valid is True
|
|
assert result.language == CodeLanguage.GO
|
|
|
|
@pytest.mark.parametrize(
|
|
(
|
|
"language",
|
|
"code",
|
|
"expected_signature",
|
|
"expected_omitted_lines",
|
|
"expected_removed_line",
|
|
"expected_closing",
|
|
),
|
|
[
|
|
(
|
|
"javascript",
|
|
(
|
|
"class Calc {\n"
|
|
" compute(x) {\n"
|
|
" let a = x + 1;\n"
|
|
" let b = a * 2;\n"
|
|
" let c = b - 3;\n"
|
|
" return c;\n"
|
|
" }\n"
|
|
"}\n"
|
|
),
|
|
"compute(x) {",
|
|
3,
|
|
"return c;",
|
|
"}\n}",
|
|
),
|
|
(
|
|
"typescript",
|
|
(
|
|
"class Calc {\n"
|
|
" compute(x: number): number {\n"
|
|
" let a = x + 1;\n"
|
|
" let b = a * 2;\n"
|
|
" let c = b - 3;\n"
|
|
" return c;\n"
|
|
" }\n"
|
|
"}\n"
|
|
),
|
|
"compute(x: number): number {",
|
|
3,
|
|
"return c;",
|
|
"}\n}",
|
|
),
|
|
(
|
|
"java",
|
|
(
|
|
"public class Calc {\n"
|
|
" public int compute(int x) {\n"
|
|
" int a = x + 1;\n"
|
|
" int b = a * 2;\n"
|
|
" int c = b - 3;\n"
|
|
" int d = c / 4;\n"
|
|
" int e = d + 5;\n"
|
|
" return e;\n"
|
|
" }\n"
|
|
"}\n"
|
|
),
|
|
"public int compute(int x) {",
|
|
5,
|
|
"return e;",
|
|
"}\n}",
|
|
),
|
|
(
|
|
"cpp",
|
|
(
|
|
"class Calc {\n"
|
|
"public:\n"
|
|
" int compute(int x) {\n"
|
|
" int a = x + 1;\n"
|
|
" int b = a * 2;\n"
|
|
" int c = b - 3;\n"
|
|
" int d = c / 4;\n"
|
|
" int e = d + 5;\n"
|
|
" return e;\n"
|
|
" }\n"
|
|
"};\n"
|
|
),
|
|
"int compute(int x) {",
|
|
5,
|
|
"return e;",
|
|
"};",
|
|
),
|
|
(
|
|
"rust",
|
|
(
|
|
"impl Calc {\n"
|
|
" pub fn compute(&self, x: i32) -> i32 {\n"
|
|
" let a = x + 1;\n"
|
|
" let b = a * 2;\n"
|
|
" let c = b - 3;\n"
|
|
" let d = c / 4;\n"
|
|
" let e = d + 5;\n"
|
|
" e\n"
|
|
" }\n"
|
|
"}\n"
|
|
),
|
|
"pub fn compute(&self, x: i32) -> i32 {",
|
|
5,
|
|
" e\n",
|
|
"}\n}",
|
|
),
|
|
(
|
|
"csharp",
|
|
(
|
|
"namespace Acme\n"
|
|
"{\n"
|
|
" public class Calc\n"
|
|
" {\n"
|
|
" public int Compute(int x)\n"
|
|
" {\n"
|
|
" int a = x + 1;\n"
|
|
" int b = a * 2;\n"
|
|
" int c = b - 3;\n"
|
|
" int d = c / 4;\n"
|
|
" int e = d + 5;\n"
|
|
" return e;\n"
|
|
" }\n"
|
|
" }\n"
|
|
"}\n"
|
|
),
|
|
"public int Compute(int x)",
|
|
5,
|
|
"return e;",
|
|
"}\n}",
|
|
),
|
|
],
|
|
)
|
|
def test_compresses_methods_inside_class_member_containers(
|
|
self,
|
|
language,
|
|
code,
|
|
expected_signature,
|
|
expected_omitted_lines,
|
|
expected_removed_line,
|
|
expected_closing,
|
|
):
|
|
"""Class/impl member containers are distinct from executable method bodies."""
|
|
config = CodeCompressorConfig(
|
|
min_tokens_for_compression=1,
|
|
max_body_lines=1,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
|
|
result = compressor.compress(code, language=language)
|
|
|
|
assert result.language == CodeLanguage(language)
|
|
assert result.syntax_valid is True
|
|
assert result.compression_ratio < 1.0
|
|
assert expected_signature in result.compressed
|
|
assert f"// [{expected_omitted_lines} lines omitted]" in result.compressed
|
|
assert expected_removed_line not in result.compressed
|
|
assert result.compressed.endswith(expected_closing)
|
|
|
|
def test_imports_preserved(self):
|
|
"""Imports are preserved in compressed output."""
|
|
config = CodeCompressorConfig(
|
|
preserve_imports=True,
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
code = generate_python_code(5)
|
|
|
|
result = compressor.compress(code, language="python")
|
|
|
|
assert "import os" in result.compressed
|
|
assert "from typing import" in result.compressed
|
|
|
|
def test_signatures_preserved(self):
|
|
"""Function signatures are preserved."""
|
|
config = CodeCompressorConfig(
|
|
preserve_signatures=True,
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
code = generate_python_code(3)
|
|
|
|
result = compressor.compress(code, language="python")
|
|
|
|
# Should preserve function signatures
|
|
assert "def function_" in result.compressed
|
|
assert "arg:" in result.compressed or "(arg" in result.compressed
|
|
|
|
def test_error_handlers_preserved(self):
|
|
"""Module-level try/except blocks are preserved."""
|
|
config = CodeCompressorConfig(
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
# Code with module-level try/except (not inside functions)
|
|
code = '''
|
|
import os
|
|
|
|
def setup():
|
|
"""Setup function."""
|
|
pass
|
|
|
|
try:
|
|
from optional_module import feature
|
|
except ImportError:
|
|
feature = None
|
|
|
|
def main():
|
|
"""Main function with long body."""
|
|
result = []
|
|
for i in range(100):
|
|
result.append(i)
|
|
return result
|
|
'''
|
|
result = compressor.compress(code, language="python")
|
|
|
|
# Module-level error handlers should be preserved
|
|
assert "try:" in result.compressed or "except" in result.compressed
|
|
|
|
def test_syntax_verification(self):
|
|
"""Output syntax is verified as valid."""
|
|
config = CodeCompressorConfig(
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
code = generate_python_code(5)
|
|
|
|
result = compressor.compress(code, language="python")
|
|
|
|
# Verify the compressed output is valid Python
|
|
assert result.syntax_valid is True
|
|
|
|
# Should be parseable
|
|
try:
|
|
compile(result.compressed, "<test>", "exec")
|
|
except SyntaxError:
|
|
pytest.fail("Compressed output has invalid Python syntax")
|
|
|
|
def test_python_future_import_stays_at_module_start(self):
|
|
"""Compressed Python keeps future imports before executable statements."""
|
|
config = CodeCompressorConfig(
|
|
min_tokens_for_compression=10,
|
|
target_compression_rate=0.2,
|
|
max_body_lines=3,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
code = textwrap.dedent(
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from dataclasses import dataclass
|
|
from typing import Any, Callable, Iterable
|
|
|
|
|
|
def traced(label: str) -> Callable[[Callable[..., Any]], Callable[..., Any]]:
|
|
def decorate(fn: Callable[..., Any]) -> Callable[..., Any]:
|
|
async def wrapper(*args: Any, **kwargs: Any) -> Any:
|
|
return await fn(*args, **kwargs)
|
|
|
|
return wrapper
|
|
|
|
return decorate
|
|
|
|
|
|
@dataclass(slots=True)
|
|
class Event:
|
|
kind: str
|
|
payload: dict[str, Any]
|
|
retries: int = 0
|
|
|
|
@property
|
|
def important(self) -> bool:
|
|
return self.kind in {"error", "retry"} or self.retries > 2
|
|
|
|
|
|
class EventRouter:
|
|
def __init__(self, sinks: dict[str, Callable[[Event], Any]]) -> None:
|
|
self.sinks = sinks
|
|
self.history: list[tuple[str, bool]] = []
|
|
|
|
@traced("route")
|
|
async def route(self, events: Iterable[Event]) -> list[str]:
|
|
accepted: list[str] = []
|
|
for event in events:
|
|
match event:
|
|
case Event(kind="error", payload={"code": code, "message": msg}, retries=r) if r > 1:
|
|
destination = "pager"
|
|
accepted.append(f"{destination}:{code}:{msg}")
|
|
case Event(kind=kind, payload=payload) if (route := payload.get("route")):
|
|
destination = str(route)
|
|
accepted.append(f"{destination}:{kind}")
|
|
case _:
|
|
destination = "dead_letter"
|
|
accepted.append(destination)
|
|
|
|
self.history.append((destination, event.important))
|
|
|
|
return [item for item in accepted if item]
|
|
"""
|
|
)
|
|
|
|
result = compressor.compress(code, language="python")
|
|
|
|
assert result.syntax_valid is True
|
|
future_import_index = result.compressed.index("from __future__ import annotations")
|
|
first_executable_index = min(
|
|
result.compressed.index("@dataclass"),
|
|
result.compressed.index("def traced"),
|
|
result.compressed.index("class EventRouter"),
|
|
)
|
|
assert future_import_index < first_executable_index
|
|
try:
|
|
compile(result.compressed, "<test>", "exec")
|
|
except SyntaxError as exc:
|
|
pytest.fail(f"Compressed output has invalid Python syntax: {exc}\n{result.compressed}")
|
|
|
|
def test_tree_sitter_loaded_after_compression(self):
|
|
"""Parser is loaded after compression."""
|
|
config = CodeCompressorConfig(
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
|
|
# Ensure clean state
|
|
unload_tree_sitter()
|
|
assert is_tree_sitter_loaded() is False
|
|
|
|
# Compress should load parser
|
|
code = generate_python_code(3)
|
|
compressor.compress(code, language="python")
|
|
|
|
assert is_tree_sitter_loaded() is True
|
|
|
|
def test_unload_clears_parsers(self):
|
|
"""unload_tree_sitter clears loaded parsers."""
|
|
config = CodeCompressorConfig(
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
|
|
# Load a parser
|
|
code = generate_python_code(3)
|
|
compressor.compress(code, language="python")
|
|
assert is_tree_sitter_loaded() is True
|
|
|
|
# Unload
|
|
result = unload_tree_sitter()
|
|
assert result is True
|
|
assert is_tree_sitter_loaded() is False
|
|
|
|
|
|
# =============================================================================
|
|
# TestDocstringModes
|
|
# =============================================================================
|
|
|
|
|
|
@pytest.mark.skipif(not TREE_SITTER_INSTALLED, reason="tree-sitter-languages not installed")
|
|
class TestDocstringModes:
|
|
"""Tests for different docstring handling modes."""
|
|
|
|
def test_docstring_mode_full(self):
|
|
"""FULL mode preserves entire docstrings."""
|
|
config = CodeCompressorConfig(
|
|
docstring_mode=DocstringMode.FULL,
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
code = generate_python_code(2)
|
|
|
|
result = compressor.compress(code, language="python")
|
|
|
|
# Should preserve full docstrings
|
|
assert "Args:" in result.compressed or "Returns:" in result.compressed
|
|
|
|
def test_docstring_mode_first_line(self):
|
|
"""FIRST_LINE mode keeps only first line of docstring."""
|
|
config = CodeCompressorConfig(
|
|
docstring_mode=DocstringMode.FIRST_LINE,
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
code = generate_python_code(2)
|
|
|
|
result = compressor.compress(code, language="python")
|
|
|
|
# Multi-line docstring details should be removed
|
|
# This is implementation-dependent
|
|
assert result.compressed is not None
|
|
|
|
def test_docstring_mode_remove(self):
|
|
"""REMOVE mode removes all docstrings."""
|
|
config = CodeCompressorConfig(
|
|
docstring_mode=DocstringMode.REMOVE,
|
|
min_tokens_for_compression=10,
|
|
max_body_lines=2, # Low threshold to trigger compression
|
|
enable_ccr=False,
|
|
)
|
|
compressor = CodeAwareCompressor(config)
|
|
# Larger function to trigger body compression
|
|
code = '''
|
|
def example():
|
|
"""This docstring should be removed."""
|
|
x = 1
|
|
y = 2
|
|
z = 3
|
|
result = x + y + z
|
|
for i in range(10):
|
|
result += i
|
|
return result
|
|
'''
|
|
result = compressor.compress(code, language="python")
|
|
|
|
# Docstring should be removed when REMOVE mode is active
|
|
assert "This docstring should be removed" not in result.compressed
|
|
|
|
|
|
# =============================================================================
|
|
# TestSemanticSymbolImportance
|
|
# =============================================================================
|
|
|
|
|
|
def _payment_processing_code() -> str:
|
|
"""Python code with varying symbol importance for testing."""
|
|
return '''
|
|
import os
|
|
from typing import List, Optional
|
|
|
|
def process_payment(order, config):
|
|
"""Process a payment through the pipeline."""
|
|
validated = validate_order(order)
|
|
if not validated.is_valid:
|
|
return PaymentResult(status='failed')
|
|
charge = charge_customer(order.customer, order.total)
|
|
receipt = generate_receipt(charge)
|
|
send_confirmation(order.customer.email, receipt)
|
|
update_inventory(order.items)
|
|
log_transaction(charge.transaction_id)
|
|
notify_warehouse(order)
|
|
return PaymentResult(status='success', receipt=receipt)
|
|
|
|
def validate_order(order):
|
|
"""Validate an order before processing."""
|
|
if not order.items:
|
|
return ValidationResult(False, ['No items'])
|
|
total = sum(item.price for item in order.items)
|
|
if total <= 0:
|
|
return ValidationResult(False, ['Invalid total'])
|
|
if not order.customer:
|
|
return ValidationResult(False, ['No customer'])
|
|
return ValidationResult(True, [])
|
|
|
|
def charge_customer(customer, amount):
|
|
"""Charge the customer."""
|
|
gateway = get_payment_gateway()
|
|
response = gateway.charge(customer.card, amount)
|
|
if not response.success:
|
|
raise PaymentError(response.error)
|
|
return response
|
|
|
|
def generate_receipt(charge):
|
|
"""Generate a receipt for the charge."""
|
|
template = load_template('receipt')
|
|
return template.render(charge=charge)
|
|
|
|
def _format_log_entry(entry):
|
|
"""Format a log entry for internal use. Never called."""
|
|
timestamp = entry.get('ts', '')
|
|
level = entry.get('level', 'INFO')
|
|
message = entry.get('msg', '')
|
|
source = entry.get('source', 'unknown')
|
|
formatted = f'[{timestamp}] {level}: {message} ({source})'
|
|
return formatted.strip()
|
|
|
|
def _dead_helper():
|
|
"""Never called anywhere in this file."""
|
|
x = 1
|
|
y = 2
|
|
z = 3
|
|
result = x + y + z
|
|
for i in range(100):
|
|
result += i
|
|
return result
|
|
'''
|
|
|
|
|
|
@pytest.mark.skipif(not TREE_SITTER_INSTALLED, reason="tree-sitter-languages not installed")
|
|
class TestSemanticSymbolImportance:
|
|
"""Tests for semantic symbol importance analysis and variable compression."""
|
|
|
|
def _make_compressor(self, **overrides):
|
|
defaults = {
|
|
"min_tokens_for_compression": 10,
|
|
"max_body_lines": 3,
|
|
"enable_ccr": False,
|
|
"semantic_analysis": True,
|
|
}
|
|
defaults.update(overrides)
|
|
return CodeAwareCompressor(CodeCompressorConfig(**defaults))
|
|
|
|
def test_symbol_scores_populated(self):
|
|
"""Compression result includes symbol importance scores."""
|
|
compressor = self._make_compressor()
|
|
result = compressor.compress(_payment_processing_code(), language="python")
|
|
|
|
assert result.symbol_scores
|
|
assert "process_payment" in result.symbol_scores
|
|
assert "validate_order" in result.symbol_scores
|
|
assert "_dead_helper" in result.symbol_scores
|
|
|
|
def test_called_functions_score_higher_than_dead_code(self):
|
|
"""Functions called by others score higher than unused functions."""
|
|
compressor = self._make_compressor()
|
|
result = compressor.compress(_payment_processing_code(), language="python")
|
|
|
|
# validate_order is called by process_payment — should score higher
|
|
assert result.symbol_scores["validate_order"] > result.symbol_scores["_dead_helper"]
|
|
assert result.symbol_scores["charge_customer"] > result.symbol_scores["_dead_helper"]
|
|
|
|
def test_public_symbols_score_higher_than_private(self):
|
|
"""Public functions (no leading _) score higher than private ones."""
|
|
compressor = self._make_compressor()
|
|
code = '''
|
|
def public_func():
|
|
"""A public function."""
|
|
x = 1
|
|
y = 2
|
|
z = 3
|
|
result = x + y + z
|
|
for i in range(10):
|
|
result += i
|
|
return result
|
|
|
|
def _private_func():
|
|
"""A private function."""
|
|
x = 1
|
|
y = 2
|
|
z = 3
|
|
result = x + y + z
|
|
for i in range(10):
|
|
result += i
|
|
return result
|
|
'''
|
|
result = compressor.compress(code, language="python")
|
|
|
|
assert result.symbol_scores["public_func"] > result.symbol_scores["_private_func"]
|
|
|
|
def test_dead_code_compressed_to_signature_only(self):
|
|
"""Functions with score < 0.1 are compressed to signature + docstring only."""
|
|
compressor = self._make_compressor()
|
|
result = compressor.compress(_payment_processing_code(), language="python")
|
|
|
|
# _dead_helper has 0 references, private → score 0.0
|
|
assert result.symbol_scores["_dead_helper"] < 0.1
|
|
|
|
# Body should be fully omitted
|
|
assert "_dead_helper" in result.compressed
|
|
# Should NOT contain body content
|
|
assert "range(100)" not in result.compressed
|
|
|
|
def test_referenced_functions_keep_more_body(self):
|
|
"""Higher-scored functions get more body lines from the budget."""
|
|
# Use a generous target rate so there IS budget to distribute
|
|
compressor = self._make_compressor(target_compression_rate=0.7)
|
|
result = compressor.compress(_payment_processing_code(), language="python")
|
|
|
|
compressed = result.compressed
|
|
# With 70% target, high-scoring functions should retain body
|
|
# while low-scoring ones get less. validate_order is referenced
|
|
# and public (high score) so should keep some body.
|
|
# _dead_helper has lowest score so should get least body.
|
|
# Count body lines per function as a proxy for retention
|
|
lines = compressed.split("\n")
|
|
in_validate = False
|
|
in_dead = False
|
|
validate_body = 0
|
|
dead_body = 0
|
|
for line in lines:
|
|
if "def validate_order" in line:
|
|
in_validate = True
|
|
in_dead = False
|
|
continue
|
|
elif "def _dead_helper" in line:
|
|
in_dead = True
|
|
in_validate = False
|
|
continue
|
|
elif line.startswith("def ") or (line.startswith("class ") and ":" in line):
|
|
in_validate = False
|
|
in_dead = False
|
|
continue
|
|
if in_validate and line.strip() and not line.strip().startswith('"""'):
|
|
validate_body += 1
|
|
if in_dead and line.strip() and not line.strip().startswith('"""'):
|
|
dead_body += 1
|
|
|
|
assert validate_body >= dead_body
|
|
|
|
def test_omitted_comment_includes_calls(self):
|
|
"""Omitted comment includes call information when available."""
|
|
compressor = self._make_compressor()
|
|
result = compressor.compress(_payment_processing_code(), language="python")
|
|
|
|
# process_payment calls validate_order, charge_customer, generate_receipt
|
|
# These should appear in the omitted comment
|
|
compressed = result.compressed
|
|
if "lines omitted" in compressed:
|
|
# Find omitted comments and check for calls info
|
|
for line in compressed.split("\n"):
|
|
if "process_payment" not in line and "lines omitted" in line:
|
|
continue
|
|
if "lines omitted; calls:" in line:
|
|
assert "validate_order" in line or "charge_customer" in line
|
|
break
|
|
|
|
def test_semantic_analysis_disabled(self):
|
|
"""When semantic_analysis=False, all functions get uniform compression."""
|
|
compressor_with = self._make_compressor(semantic_analysis=True)
|
|
compressor_without = self._make_compressor(semantic_analysis=False)
|
|
|
|
code = _payment_processing_code()
|
|
result_with = compressor_with.compress(code, language="python")
|
|
result_without = compressor_without.compress(code, language="python")
|
|
|
|
# Without semantic analysis, no symbol scores
|
|
assert result_without.symbol_scores == {}
|
|
|
|
# With semantic analysis, dead code is compressed more aggressively
|
|
# _dead_helper body should NOT appear with semantic analysis
|
|
assert "range(100)" not in result_with.compressed
|
|
# But with uniform compression (no semantic), body lines ARE kept
|
|
assert "x = 1" in result_without.compressed
|
|
|
|
def test_summary_includes_semantic_info(self):
|
|
"""Summary includes semantic analysis information."""
|
|
compressor = self._make_compressor()
|
|
result = compressor.compress(_payment_processing_code(), language="python")
|
|
|
|
summary = result.summary
|
|
if result.symbol_scores:
|
|
low_count = sum(1 for s in result.symbol_scores.values() if s < 0.1)
|
|
if low_count > 0:
|
|
assert "low-importance" in summary
|
|
|
|
def test_dunder_methods_get_boost(self):
|
|
"""Dunder methods (__init__, etc.) get importance boost."""
|
|
compressor = self._make_compressor()
|
|
code = '''
|
|
class MyClass:
|
|
"""A class."""
|
|
def __init__(self, value):
|
|
"""Initialize."""
|
|
self.value = value
|
|
self.processed = False
|
|
self.results = []
|
|
self.cache = {}
|
|
self.errors = []
|
|
for i in range(10):
|
|
self.results.append(i)
|
|
|
|
def _setup_cache(self):
|
|
"""Internal setup."""
|
|
x = 1
|
|
y = 2
|
|
z = 3
|
|
result = x + y + z
|
|
for i in range(10):
|
|
result += i
|
|
return result
|
|
'''
|
|
result = compressor.compress(code, language="python")
|
|
|
|
# __init__ should score higher than _setup_cache
|
|
if "__init__" in result.symbol_scores and "_setup_cache" in result.symbol_scores:
|
|
assert result.symbol_scores["__init__"] > result.symbol_scores["_setup_cache"]
|
|
|
|
def test_javascript_importance(self):
|
|
"""Symbol importance works for JavaScript code."""
|
|
compressor = self._make_compressor()
|
|
code = """
|
|
import { db } from './database';
|
|
|
|
function processUser(userId) {
|
|
const user = fetchUser(userId);
|
|
const profile = buildProfile(user);
|
|
sendNotification(user.email, profile);
|
|
logAction('process', userId);
|
|
updateMetrics('user_processed');
|
|
return { user, profile };
|
|
}
|
|
|
|
function fetchUser(id) {
|
|
const result = db.query('SELECT * FROM users WHERE id = ?', [id]);
|
|
if (!result) {
|
|
throw new Error('User not found');
|
|
}
|
|
return result;
|
|
}
|
|
|
|
function buildProfile(user) {
|
|
const prefs = loadPreferences(user.id);
|
|
return { ...user, preferences: prefs };
|
|
}
|
|
|
|
function _internalDebug(msg) {
|
|
const ts = Date.now();
|
|
const formatted = `[${ts}] DEBUG: ${msg}`;
|
|
console.log(formatted);
|
|
return formatted;
|
|
}
|
|
"""
|
|
result = compressor.compress(code, language="javascript")
|
|
|
|
assert result.symbol_scores
|
|
# fetchUser is called by processUser — should score higher than _internalDebug
|
|
if "fetchUser" in result.symbol_scores and "_internalDebug" in result.symbol_scores:
|
|
assert result.symbol_scores["fetchUser"] > result.symbol_scores["_internalDebug"]
|
|
|
|
def test_syntax_still_valid_with_importance(self):
|
|
"""Compressed output with importance remains syntactically valid."""
|
|
compressor = self._make_compressor()
|
|
result = compressor.compress(_payment_processing_code(), language="python")
|
|
|
|
assert result.syntax_valid is True
|
|
|
|
# Should be parseable as Python
|
|
try:
|
|
compile(result.compressed, "<test>", "exec")
|
|
except SyntaxError:
|
|
pytest.fail("Semantic compression produced invalid Python syntax")
|
|
|
|
def test_empty_code_no_crash(self):
|
|
"""Importance analysis handles empty code gracefully."""
|
|
compressor = self._make_compressor()
|
|
result = compressor.compress("", language="python")
|
|
|
|
assert result.symbol_scores == {}
|
|
|
|
def test_config_default_semantic_analysis_enabled(self):
|
|
"""semantic_analysis is True by default in config."""
|
|
config = CodeCompressorConfig()
|
|
assert config.semantic_analysis is True
|
|
|
|
|
|
# =============================================================================
|
|
# Regression: tree-sitter ABI mismatch (real AST must run, no silent fallback)
|
|
# =============================================================================
|
|
|
|
|
|
@pytest.mark.skipif(not TREE_SITTER_INSTALLED, reason="tree-sitter grammar pack not installed")
|
|
class TestRealASTRuns:
|
|
"""Guards against the regression where the code-aware compressor silently
|
|
fell back to a lossy stripper because ``_get_parser`` built a stock
|
|
``tree_sitter.Parser`` and assigned it a foreign grammar-pack ``Language``
|
|
(raising ``TypeError`` that was swallowed into a fallback).
|
|
"""
|
|
|
|
def _compressor(self):
|
|
return CodeAwareCompressor(
|
|
CodeCompressorConfig(
|
|
min_tokens_for_compression=10,
|
|
enable_ccr=False,
|
|
)
|
|
)
|
|
|
|
def _recovery_compressor(self, **overrides):
|
|
defaults = {
|
|
"min_tokens_for_compression": 1,
|
|
"max_body_lines": 2,
|
|
"enable_ccr": False,
|
|
"semantic_analysis": False,
|
|
}
|
|
defaults.update(overrides)
|
|
return CodeAwareCompressor(CodeCompressorConfig(**defaults))
|
|
|
|
def _python_recovery_fixture(self) -> str:
|
|
return textwrap.dedent(
|
|
"""\
|
|
from pathlib import Path
|
|
|
|
def expand_search_roots(user_root: str) -> list[Path]:
|
|
root = Path(user_root)
|
|
candidates = [root]
|
|
for child in root.iterdir():
|
|
candidates.append(child.resolve())
|
|
return candidates
|
|
|
|
def load_user_overrides(config_path: str) -> dict[str, str]:
|
|
config: dict[str, str] = {}
|
|
for line in Path(config_path).read_text().splitlines():
|
|
if "=" in line:
|
|
key, value = line.split("=", 1)
|
|
config[key.strip()] = value.strip()
|
|
return config
|
|
"""
|
|
)
|
|
|
|
def test_get_parser_returns_stock_node_api(self):
|
|
"""The parser must yield nodes with the stock tree_sitter property API
|
|
that the tree-walking code relies on (``.type``/``.children``/...)."""
|
|
from headroom.transforms.code_compressor import _get_parser
|
|
|
|
parser = _get_parser("python")
|
|
tree = parser.parse(b"def foo(x):\n return x + 1\n")
|
|
root = tree.root_node
|
|
|
|
# Property access (NOT method calls) — the old pack binding exposed
|
|
# methods like ``.kind()`` which would break every call site.
|
|
assert root.type == "module"
|
|
assert root.child_count >= 1
|
|
assert isinstance(root.children, list)
|
|
|
|
func = root.children[0]
|
|
assert func.type == "function_definition"
|
|
assert isinstance(func.start_byte, int)
|
|
assert isinstance(func.end_byte, int)
|
|
# start_point must be index-able like a (row, col) tuple.
|
|
assert func.start_point[0] == 0
|
|
assert b"def foo" in func.text
|
|
|
|
def test_check_tree_sitter_available_verifies_real_parse(self):
|
|
"""``_check_tree_sitter_available`` must only return True when an actual
|
|
parse succeeds — not merely when the package imports."""
|
|
import headroom.transforms.code_compressor as cc
|
|
|
|
cc._tree_sitter_available = None # reset memoized result
|
|
assert cc._check_tree_sitter_available() is True
|
|
|
|
def test_check_tree_sitter_available_false_when_parse_broken(self):
|
|
"""If parsing raises (e.g. the old foreign-Language bug), availability
|
|
must report False instead of green-lighting the broken path."""
|
|
import headroom.transforms.code_compressor as cc
|
|
|
|
cc._tree_sitter_available = None
|
|
with patch.object(cc, "_get_parser", side_effect=TypeError("boom")):
|
|
assert cc._check_tree_sitter_available() is False
|
|
cc._tree_sitter_available = None # reset for other tests
|
|
|
|
def test_ast_runs_for_python_no_fallback(self):
|
|
"""A supported language must be compressed via real AST, not the
|
|
UNKNOWN-language Kompress fallback."""
|
|
result = self._compressor().compress(_payment_processing_code(), language="python")
|
|
|
|
# The fallback path forces language=UNKNOWN and syntax_valid=False.
|
|
# Real AST keeps the detected language and guarantees valid syntax.
|
|
assert result.language == CodeLanguage.PYTHON
|
|
assert result.syntax_valid is True
|
|
compile(result.compressed, "<test>", "exec")
|
|
|
|
def test_ast_preserves_structure_for_python(self):
|
|
"""AST output retains signatures/scopes/imports (unlike the old
|
|
whitespace garble)."""
|
|
code = (
|
|
"import math\n"
|
|
"\n"
|
|
"def compute(values):\n"
|
|
" total = 0\n"
|
|
" for v in values:\n"
|
|
" total += v * v\n"
|
|
" total -= 1\n"
|
|
" total *= 2\n"
|
|
" total //= 3\n"
|
|
" return math.sqrt(total)\n"
|
|
)
|
|
result = self._compressor().compress(code, language="python")
|
|
|
|
assert result.language == CodeLanguage.PYTHON
|
|
assert result.syntax_valid is True
|
|
# Structure markers survive compression.
|
|
assert "import math" in result.compressed
|
|
assert "def compute(values):" in result.compressed
|
|
# Output is still valid Python.
|
|
compile(result.compressed, "<test>", "exec")
|
|
|
|
def test_python_invalid_node_falls_back_locally(self):
|
|
"""One invalid Python rewrite must preserve only that definition."""
|
|
compressor = self._recovery_compressor()
|
|
code = self._python_recovery_fixture()
|
|
original_compress_function_ast = compressor._compress_function_ast
|
|
|
|
def _patched(node, code_text, language, lang_config, body_limits, analysis):
|
|
name = cc._get_definition_name(node)
|
|
if name == "expand_search_roots":
|
|
return "def expand_search_roots(user_root: str) -> list[Path]:\n if True\n"
|
|
if name == "load_user_overrides":
|
|
return (
|
|
"def load_user_overrides(config_path: str) -> dict[str, str]:\n"
|
|
" config: dict[str, str] = {}\n"
|
|
" # [4 lines omitted]\n"
|
|
" return config"
|
|
)
|
|
return original_compress_function_ast(
|
|
node,
|
|
code_text,
|
|
language,
|
|
lang_config,
|
|
body_limits,
|
|
analysis,
|
|
)
|
|
|
|
with patch.object(compressor, "_compress_function_ast", side_effect=_patched):
|
|
result = compressor.compress(code, language="python")
|
|
|
|
assert result.syntax_valid is True
|
|
assert result.compression_ratio < 1.0
|
|
assert result.compressed != code
|
|
assert "if True" not in result.compressed
|
|
assert "for child in root.iterdir():" in result.compressed
|
|
assert "# [4 lines omitted]" in result.compressed
|
|
compile(result.compressed, "<test>", "exec")
|
|
|
|
def test_python_recovery_does_not_block_valid_modern_syntax(self):
|
|
"""Valid decorators, nested defs, and match syntax still compress."""
|
|
compressor = self._recovery_compressor(max_body_lines=4)
|
|
code = textwrap.dedent(
|
|
"""\
|
|
from dataclasses import dataclass
|
|
|
|
def traced(fn):
|
|
return fn
|
|
|
|
@dataclass
|
|
class Command:
|
|
name: str
|
|
payload: dict[str, int]
|
|
|
|
@traced
|
|
def route_command(command: Command) -> str:
|
|
def normalize(value: str) -> str:
|
|
return value.strip().lower()
|
|
|
|
match normalize(command.name):
|
|
case "ping":
|
|
return "pong"
|
|
case "echo":
|
|
return str(command.payload)
|
|
case _:
|
|
return "unknown"
|
|
"""
|
|
)
|
|
|
|
result = compressor.compress(code, language="python")
|
|
|
|
assert result.syntax_valid is True
|
|
assert result.compression_ratio < 1.0
|
|
assert "@traced" in result.compressed
|
|
assert "class Command:" in result.compressed
|
|
assert "def route_command(command: Command) -> str:" in result.compressed
|
|
compile(result.compressed, "<test>", "exec")
|
|
|
|
def test_python_recovery_still_returns_original_when_all_candidates_invalid(self):
|
|
"""Recovery keeps the terminal whole-file fail-safe."""
|
|
compressor = self._recovery_compressor()
|
|
code = self._python_recovery_fixture()
|
|
|
|
def _patched(node, _code_text, _language, _lang_config, _body_limits, _analysis):
|
|
name = cc._get_definition_name(node) or "broken"
|
|
return f"def {name}(:\n pass"
|
|
|
|
with patch.object(compressor, "_compress_function_ast", side_effect=_patched):
|
|
result = compressor.compress(code, language="python")
|
|
|
|
assert result.syntax_valid is True
|
|
assert result.compression_ratio == 1.0
|
|
assert "(:\n" not in result.compressed
|
|
assert "for child in root.iterdir():" in result.compressed
|
|
assert 'key, value = line.split("=", 1)' in result.compressed
|
|
compile(result.compressed, "<test>", "exec")
|
|
|
|
def test_get_node_text_uses_utf8_byte_offsets(self):
|
|
"""tree-sitter byte offsets must not be sliced as Python str indexes."""
|
|
from headroom.transforms.code_compressor import _get_node_text, _get_parser
|
|
|
|
code = 'def first():\n """中文占位"""\n return 1\n\ndef second():\n return 2\n'
|
|
root = _get_parser("python").parse(code.encode("utf-8")).root_node
|
|
functions = [node for node in root.children if node.type == "function_definition"]
|
|
|
|
assert _get_node_text(functions[1], code) == "def second():\n return 2"
|
|
|
|
def test_ast_compresses_python_after_non_ascii_source(self):
|
|
"""CJK/emoji before a later function must not corrupt downstream slices."""
|
|
compressor = CodeAwareCompressor(
|
|
CodeCompressorConfig(
|
|
min_tokens_for_compression=1,
|
|
max_body_lines=2,
|
|
enable_ccr=False,
|
|
semantic_analysis=False,
|
|
)
|
|
)
|
|
code = (
|
|
"def first():\n"
|
|
' """中文占位 with emoji 🔥."""\n'
|
|
" return 1\n"
|
|
"\n"
|
|
"def second():\n"
|
|
" values = []\n"
|
|
" for i in range(10):\n"
|
|
" values.append(i)\n"
|
|
" values.append(i * 2)\n"
|
|
" values.append(i * 3)\n"
|
|
" values.append(i * 4)\n"
|
|
" return sum(values)\n"
|
|
)
|
|
|
|
result = compressor.compress(code, language="python")
|
|
|
|
assert result.language == CodeLanguage.PYTHON
|
|
assert result.syntax_valid is True
|
|
assert result.compression_ratio < 1.0
|
|
assert "def second():" in result.compressed
|
|
assert "中文占位" in result.compressed
|
|
compile(result.compressed, "<test>", "exec")
|
|
|
|
def test_ast_runs_for_rust_no_fallback(self):
|
|
"""A second supported language (Rust) also runs through real AST."""
|
|
code = (
|
|
"pub fn add(a: i64, b: i64) -> i64 {\n"
|
|
" let mut acc = a;\n"
|
|
" acc += b;\n"
|
|
" acc -= 0;\n"
|
|
" acc\n"
|
|
"}\n"
|
|
)
|
|
result = self._compressor().compress(code, language="rust")
|
|
|
|
assert result.language == CodeLanguage.RUST
|
|
assert result.syntax_valid is True
|
|
# Signature is preserved verbatim.
|
|
assert "pub fn add(a: i64, b: i64) -> i64" in result.compressed
|
|
|
|
|
|
@pytest.mark.skipif(not TREE_SITTER_INSTALLED, reason="tree-sitter grammar pack not installed")
|
|
class TestMultiLineClassHeaders:
|
|
"""Keep the opening brace when a multi-line signature shares its last
|
|
line with it (``impl<T> Foo<T> where T: Debug {``, ``class Foo
|
|
implements Bar {``). Dropping that line leaves the rebuilt header
|
|
without a brace, so the compressed output no longer parses and the
|
|
compressor falls back to the original.
|
|
"""
|
|
|
|
def _compressor(self):
|
|
return CodeAwareCompressor(
|
|
CodeCompressorConfig(
|
|
min_tokens_for_compression=1,
|
|
max_body_lines=2,
|
|
enable_ccr=False,
|
|
semantic_analysis=False,
|
|
)
|
|
)
|
|
|
|
@pytest.mark.parametrize(
|
|
("language", "code", "expected_header"),
|
|
[
|
|
(
|
|
"rust",
|
|
(
|
|
"impl<T> Wrapper<T>\n"
|
|
"where T: Debug + Clone {\n"
|
|
" pub fn total(&self) -> u64 {\n"
|
|
" let mut acc = 0;\n"
|
|
" for item in &self.items {\n"
|
|
" acc += item.id;\n"
|
|
" }\n"
|
|
" acc\n"
|
|
" }\n"
|
|
"}\n"
|
|
),
|
|
"where T: Debug + Clone {",
|
|
),
|
|
(
|
|
"java",
|
|
(
|
|
"public class Calc\n"
|
|
" extends Base\n"
|
|
" implements Runnable {\n"
|
|
" public int compute(int x) {\n"
|
|
" int a = x + 1;\n"
|
|
" int b = a * 2;\n"
|
|
" int c = b - 3;\n"
|
|
" return c;\n"
|
|
" }\n"
|
|
"}\n"
|
|
),
|
|
"implements Runnable {",
|
|
),
|
|
],
|
|
)
|
|
def test_opening_brace_line_survives_compression(self, language, code, expected_header):
|
|
result = self._compressor().compress(code, language=language)
|
|
|
|
assert result.language == CodeLanguage(language)
|
|
# A ratio of 1.0 means the syntax check rejected the output and the
|
|
# compressor returned the original untouched.
|
|
assert result.compression_ratio < 1.0
|
|
assert expected_header in result.compressed
|
|
|
|
|
|
@pytest.mark.skipif(not TREE_SITTER_INSTALLED, reason="tree-sitter grammar pack not installed")
|
|
class TestCSharpSupport:
|
|
"""C# (``csharp`` grammar) parity with Java/C++/Rust: signatures preserved
|
|
verbatim, method/constructor bodies compressed, block-scoped namespaces
|
|
routed through container compression (no verbatim re-dump), file-scoped
|
|
namespace headers kept ahead of types, malformed input passed through.
|
|
"""
|
|
|
|
def _compressor(self):
|
|
return CodeAwareCompressor(
|
|
CodeCompressorConfig(
|
|
min_tokens_for_compression=1,
|
|
max_body_lines=1,
|
|
enable_ccr=False,
|
|
)
|
|
)
|
|
|
|
def test_block_scoped_namespace_compresses_without_redumping(self):
|
|
"""Block-scoped ``namespace { }`` wraps types in a declaration_list; the
|
|
container must compress its members and NOT be re-emitted verbatim
|
|
(which would duplicate the class and defeat compression — the C#
|
|
analogue of the #1318 class-member trap, one nesting level deeper)."""
|
|
code = (
|
|
"using System;\n"
|
|
"\n"
|
|
"namespace Acme.Widgets\n"
|
|
"{\n"
|
|
" public class WidgetService\n"
|
|
" {\n"
|
|
" public int Process(int x)\n"
|
|
" {\n"
|
|
" var a = x + 1;\n"
|
|
" var b = a + 2;\n"
|
|
" var c = b + 3;\n"
|
|
" Console.WriteLine(c);\n"
|
|
" return c;\n"
|
|
" }\n"
|
|
" }\n"
|
|
"}\n"
|
|
)
|
|
result = self._compressor().compress(code, language="csharp")
|
|
|
|
assert result.language == CodeLanguage.CSHARP
|
|
assert result.syntax_valid is True
|
|
assert result.compression_ratio < 1.0
|
|
# namespace + type headers preserved verbatim
|
|
assert "namespace Acme.Widgets" in result.compressed
|
|
assert "public class WidgetService" in result.compressed
|
|
assert "public int Process(int x)" in result.compressed
|
|
# method body actually compressed
|
|
assert "lines omitted" in result.compressed
|
|
assert "return c;" not in result.compressed
|
|
# the class is emitted exactly once (no re-dump / duplication)
|
|
assert result.compressed.count("class WidgetService") == 1
|
|
# block-namespace wrapper still closes
|
|
assert result.compressed.rstrip().endswith("}")
|
|
|
|
def test_file_scoped_namespace_directive_precedes_types(self):
|
|
"""A file-scoped ``namespace X;`` must stay ahead of the type
|
|
declarations; emitting it after a type would not be valid C#."""
|
|
code = (
|
|
"using System;\n"
|
|
"\n"
|
|
"namespace Acme.Tools;\n"
|
|
"\n"
|
|
"public class Helper\n"
|
|
"{\n"
|
|
" public int Add(int a, int b)\n"
|
|
" {\n"
|
|
" var s = a + b;\n"
|
|
" var t = s + 1;\n"
|
|
" var u = t + 2;\n"
|
|
" return u;\n"
|
|
" }\n"
|
|
"}\n"
|
|
)
|
|
result = self._compressor().compress(code, language="csharp")
|
|
|
|
assert result.language == CodeLanguage.CSHARP
|
|
assert result.syntax_valid is True
|
|
assert result.compression_ratio < 1.0
|
|
assert "namespace Acme.Tools;" in result.compressed
|
|
assert "public class Helper" in result.compressed
|
|
assert result.compressed.index("namespace Acme.Tools;") < result.compressed.index(
|
|
"class Helper"
|
|
)
|
|
|
|
def test_record_preserves_positional_parameters(self):
|
|
"""Record primary-constructor parameters are part of the signature and
|
|
must be preserved verbatim while the method body compresses."""
|
|
code = (
|
|
"namespace Acme.Models;\n"
|
|
"\n"
|
|
"public record Point(int X, int Y)\n"
|
|
"{\n"
|
|
" public double Dist()\n"
|
|
" {\n"
|
|
" var sq = X * X + Y * Y;\n"
|
|
" var r = System.Math.Sqrt(sq);\n"
|
|
" System.Console.WriteLine(r);\n"
|
|
" return r;\n"
|
|
" }\n"
|
|
"}\n"
|
|
)
|
|
result = self._compressor().compress(code, language="csharp")
|
|
|
|
assert result.language == CodeLanguage.CSHARP
|
|
assert result.syntax_valid is True
|
|
assert result.compression_ratio < 1.0
|
|
assert "public record Point(int X, int Y)" in result.compressed
|
|
assert "public double Dist()" in result.compressed
|
|
assert "lines omitted" in result.compressed
|
|
|
|
def test_preprocessor_wrapped_file_is_not_duplicated(self):
|
|
"""A file wrapped in ``#if``/``#endif`` (idiomatic in multi-targeted
|
|
.NET code) must not have its content emitted twice. The visitor
|
|
recurses into ``preproc_if`` and captures the declarations, but the
|
|
top-level pass used to re-emit the uncaptured wrapper verbatim,
|
|
duplicating the whole file (observed on real repos: output up to
|
|
~1.9x the input). Opaque handling keeps it verbatim exactly once."""
|
|
code = (
|
|
"#if !HAVE_TRACE_WRITER\n"
|
|
"using System;\n"
|
|
"\n"
|
|
"namespace Acme\n"
|
|
"{\n"
|
|
" public enum TraceLevel\n"
|
|
" {\n"
|
|
" Off = 0,\n"
|
|
" Error = 1,\n"
|
|
" }\n"
|
|
"}\n"
|
|
"#endif\n"
|
|
)
|
|
result = self._compressor().compress(code, language="csharp")
|
|
|
|
assert result.syntax_valid is True
|
|
# Content appears exactly once — no wrapper re-dump.
|
|
assert result.compressed.count("enum TraceLevel") == 1
|
|
assert result.compressed.count("using System;") == 1
|
|
# Never larger than the input (verbatim pass-through is acceptable).
|
|
assert result.compression_ratio <= 1.0
|
|
# Conditional directives stay balanced.
|
|
assert result.compressed.count("#if") == result.compressed.count("#endif")
|
|
|
|
def test_preprocessor_wrapped_usings_stay_ahead_of_types(self):
|
|
"""``#if``-wrapped using directives (idiomatic for multi-targeting)
|
|
must be emitted with the imports. Appending them as trailing top-level
|
|
code puts usings after type declarations — invalid C#, which trips the
|
|
output syntax check and falls back to no compression at all (observed
|
|
on real repos: hundreds of files silently uncompressed)."""
|
|
code = (
|
|
"using System;\n"
|
|
"#if HAVE_BIG_INTEGER\n"
|
|
"using System.Numerics;\n"
|
|
"#endif\n"
|
|
"\n"
|
|
"namespace Acme\n"
|
|
"{\n"
|
|
" public class Calc\n"
|
|
" {\n"
|
|
" public int Compute(int x)\n"
|
|
" {\n"
|
|
" var a = x + 1;\n"
|
|
" var b = a * 2;\n"
|
|
" var c = b - 3;\n"
|
|
" var d = c / 4;\n"
|
|
" return d;\n"
|
|
" }\n"
|
|
" }\n"
|
|
"}\n"
|
|
)
|
|
result = self._compressor().compress(code, language="csharp")
|
|
|
|
assert result.syntax_valid is True
|
|
# Compression must actually happen (no silent fallback).
|
|
assert result.compression_ratio < 1.0
|
|
assert "lines omitted" in result.compressed
|
|
# The conditional import block survives verbatim, ahead of the types.
|
|
assert "#if HAVE_BIG_INTEGER" in result.compressed
|
|
assert "using System.Numerics;" in result.compressed
|
|
assert result.compressed.index("#endif") < result.compressed.index("class Calc")
|
|
|
|
def test_region_markers_inside_class_compress_cleanly(self):
|
|
"""``#region``/``#endregion`` markers span their trailing newline, so
|
|
their end_point sits at column 0 of the NEXT line. Line-based child
|
|
extraction used to swallow that line — duplicating the following
|
|
method's signature and the class's closing brace (invalid output →
|
|
silent fallback to no compression on real repos)."""
|
|
code = (
|
|
"namespace Acme\n"
|
|
"{\n"
|
|
" public class Svc\n"
|
|
" {\n"
|
|
" #region Api\n"
|
|
" public int F(int x)\n"
|
|
" {\n"
|
|
" var a = x + 1;\n"
|
|
" var b = a * 2;\n"
|
|
" var c = b - 3;\n"
|
|
" return c;\n"
|
|
" }\n"
|
|
" #endregion\n"
|
|
" }\n"
|
|
"}\n"
|
|
)
|
|
result = self._compressor().compress(code, language="csharp")
|
|
|
|
assert result.syntax_valid is True
|
|
assert result.compression_ratio < 1.0
|
|
assert "lines omitted" in result.compressed
|
|
# Signature emitted exactly once (regions used to duplicate it).
|
|
assert result.compressed.count("public int F(int x)") == 1
|
|
# Region markers survive, braces stay balanced.
|
|
assert "#region Api" in result.compressed
|
|
assert "#endregion" in result.compressed
|
|
assert result.compressed.count("{") == result.compressed.count("}")
|
|
|
|
def test_license_region_header_stays_on_top(self):
|
|
"""A ``#region License`` banner above the usings (near-universal in
|
|
real .NET code) must stay at the top of the file. Relocating it after
|
|
the type declarations — the old behavior for uncaptured top-level
|
|
nodes — is rejected by tree-sitter-c-sharp, failing output validation
|
|
and silently forfeiting compression for the whole file."""
|
|
code = (
|
|
"#region License\n"
|
|
"// Copyright (c) 2007 Example Corp.\n"
|
|
"// Licensed under the MIT license.\n"
|
|
"#endregion\n"
|
|
"\n"
|
|
"using System;\n"
|
|
"\n"
|
|
"namespace Acme\n"
|
|
"{\n"
|
|
" public class Svc\n"
|
|
" {\n"
|
|
" public int F(int x)\n"
|
|
" {\n"
|
|
" var a = x + 1;\n"
|
|
" var b = a * 2;\n"
|
|
" var c = b - 3;\n"
|
|
" return c;\n"
|
|
" }\n"
|
|
" }\n"
|
|
"}\n"
|
|
)
|
|
result = self._compressor().compress(code, language="csharp")
|
|
|
|
assert result.syntax_valid is True
|
|
assert result.compression_ratio < 1.0
|
|
assert "lines omitted" in result.compressed
|
|
# Header block survives, in order, ahead of everything else.
|
|
assert result.compressed.index("#region License") < result.compressed.index("using System;")
|
|
assert result.compressed.index("// Copyright") < result.compressed.index("#endregion")
|
|
assert result.compressed.index("#endregion") < result.compressed.index("class Svc")
|
|
|
|
def test_malformed_csharp_passes_through_unchanged(self):
|
|
"""Malformed C# must pass through unchanged (no data loss; prefer false
|
|
negatives over serving broken or altered code)."""
|
|
code = (
|
|
"namespace Broken\n"
|
|
"{\n"
|
|
" public class Oops\n"
|
|
" {\n"
|
|
" public int F(int x)\n"
|
|
" {\n"
|
|
" var a = x + 1\n" # missing ';' and unclosed braces
|
|
" return a\n"
|
|
)
|
|
result = self._compressor().compress(code, language="csharp")
|
|
|
|
assert result.compressed == code
|
|
assert result.compression_ratio == 1.0
|
|
|
|
def test_detect_language_identifies_csharp(self):
|
|
"""Auto-detection (no explicit language hint) recognizes C#."""
|
|
code = (
|
|
"using System;\n"
|
|
"\n"
|
|
"namespace Acme\n"
|
|
"{\n"
|
|
" public class Svc\n"
|
|
" {\n"
|
|
" public int Total { get; set; }\n"
|
|
" public int Add(int a, int b)\n"
|
|
" {\n"
|
|
" return a + b;\n"
|
|
" }\n"
|
|
" }\n"
|
|
"}\n"
|
|
)
|
|
lang, confidence = detect_language(code)
|
|
assert lang == CodeLanguage.CSHARP
|
|
assert confidence > 0.0
|
|
|
|
|
|
@pytest.mark.skipif(not TREE_SITTER_INSTALLED, reason="tree-sitter grammar pack not installed")
|
|
class TestPhpSupport:
|
|
"""PHP (``php`` grammar) parity with C#: signatures preserved verbatim,
|
|
function/method bodies compressed, ``<?php`` tag and ``namespace``/``use``
|
|
header order preserved, Perl sigil-overlap disambiguated in detection,
|
|
malformed input passed through.
|
|
"""
|
|
|
|
def _compressor(self):
|
|
return CodeAwareCompressor(
|
|
CodeCompressorConfig(
|
|
min_tokens_for_compression=1,
|
|
max_body_lines=1,
|
|
enable_ccr=False,
|
|
)
|
|
)
|
|
|
|
def test_class_methods_compress_signatures_preserved(self):
|
|
code = (
|
|
"<?php\n"
|
|
"namespace App\\Service;\n"
|
|
"\n"
|
|
"use App\\Model\\User;\n"
|
|
"\n"
|
|
"final class UserService {\n"
|
|
" private $logger;\n"
|
|
"\n"
|
|
" public function process(User $u): bool {\n"
|
|
" $name = strtolower(trim($u->getName()));\n"
|
|
" $tags = [];\n"
|
|
" foreach ($u->getTags() as $tag) {\n"
|
|
" $tags[] = $tag->normalize();\n"
|
|
" }\n"
|
|
" $this->logger->info($name);\n"
|
|
" return true;\n"
|
|
" }\n"
|
|
"}\n"
|
|
)
|
|
result = self._compressor().compress(code, language="php")
|
|
|
|
assert result.language == CodeLanguage.PHP
|
|
assert result.syntax_valid is True
|
|
assert result.compression_ratio < 1.0
|
|
# signature + class header preserved verbatim
|
|
assert "final class UserService" in result.compressed
|
|
assert "public function process(User $u): bool" in result.compressed
|
|
# method body actually compressed
|
|
assert "lines omitted" in result.compressed
|
|
assert "$tag->normalize()" not in result.compressed
|
|
# the class is emitted exactly once
|
|
assert result.compressed.count("class UserService") == 1
|
|
|
|
def test_php_tag_and_namespace_precede_uses_and_types(self):
|
|
"""``<?php`` must stay first and ``namespace X;`` must precede the
|
|
``use`` imports and type declarations — any other order is not valid
|
|
PHP."""
|
|
code = (
|
|
"<?php\n"
|
|
"namespace App\\Tools;\n"
|
|
"\n"
|
|
"use App\\Model\\Item;\n"
|
|
"\n"
|
|
"function helper(int $x): int {\n"
|
|
" $acc = 0;\n"
|
|
" for ($i = 0; $i < $x; $i++) {\n"
|
|
" $acc += $i;\n"
|
|
" $acc -= 1;\n"
|
|
" }\n"
|
|
" return $acc;\n"
|
|
"}\n"
|
|
)
|
|
result = self._compressor().compress(code, language="php")
|
|
|
|
assert result.language == CodeLanguage.PHP
|
|
assert result.syntax_valid is True
|
|
assert result.compression_ratio < 1.0
|
|
compressed = result.compressed
|
|
assert compressed.lstrip().startswith("<?php")
|
|
assert compressed.index("<?php") < compressed.index("namespace App\\Tools;")
|
|
assert compressed.index("namespace App\\Tools;") < compressed.index("use App\\Model\\Item;")
|
|
assert compressed.index("use App\\Model\\Item;") < compressed.index("function helper")
|
|
assert "lines omitted" in compressed
|
|
|
|
def test_detect_language_identifies_php(self):
|
|
"""Auto-detection recognizes PHP despite the Perl sigil overlap
|
|
(``$var`` matches Perl's prefilter; the ``<?php`` tag disambiguates)."""
|
|
code = (
|
|
"<?php\n"
|
|
"namespace Acme;\n"
|
|
"\n"
|
|
"use Acme\\Widget;\n"
|
|
"\n"
|
|
"class Svc {\n"
|
|
" public function add(int $a, int $b): int {\n"
|
|
" $sum = $a + $b;\n"
|
|
" return $sum;\n"
|
|
" }\n"
|
|
"}\n"
|
|
)
|
|
lang, confidence = detect_language(code)
|
|
assert lang == CodeLanguage.PHP
|
|
assert confidence > 0.0
|
|
|
|
def test_phtml_alias_coerces_to_php(self):
|
|
assert coerce_language("phtml") == CodeLanguage.PHP
|
|
assert coerce_language("php8") == CodeLanguage.PHP
|
|
|
|
def test_malformed_php_passes_through_unchanged(self):
|
|
code = "<?php\nclass Broken {\n public function oops( {\n"
|
|
result = self._compressor().compress(code, language="php")
|
|
assert result.compressed == code
|
|
|
|
|
|
@pytest.mark.skipif(not TREE_SITTER_INSTALLED, reason="tree-sitter grammar pack not installed")
|
|
class TestRustAttributeAttachment:
|
|
"""Rust outer attributes (``#[derive(...)]``, ``#[inline]``, ``#[cfg(...)]``)
|
|
are siblings of the item they decorate, like doc comments. Left
|
|
uncaptured, they fall through to the leftover sweep and drift away from
|
|
the item — commonly after the compressed function, which rustc rejects
|
|
with ``expected item after attributes``.
|
|
"""
|
|
|
|
def _compressor(self):
|
|
return CodeAwareCompressor(
|
|
CodeCompressorConfig(
|
|
min_tokens_for_compression=1,
|
|
max_body_lines=1,
|
|
enable_ccr=False,
|
|
semantic_analysis=False,
|
|
)
|
|
)
|
|
|
|
def test_attributes_stay_attached_to_their_items(self):
|
|
code = (
|
|
"#[derive(Debug, Clone, PartialEq)]\n"
|
|
"pub struct Item {\n"
|
|
" pub id: u64,\n"
|
|
" pub name: String,\n"
|
|
"}\n"
|
|
"\n"
|
|
"#[inline]\n"
|
|
'#[cfg(feature = "extra")]\n'
|
|
"pub fn log_item(item: &Item, sink: &mut Vec<u64>) {\n"
|
|
" sink.push(item.id);\n"
|
|
" sink.push(item.id + 1);\n"
|
|
" sink.push(item.id + 2);\n"
|
|
" sink.push(item.id + 3);\n"
|
|
"}\n"
|
|
)
|
|
|
|
result = self._compressor().compress(code, language="rust")
|
|
|
|
assert result.compression_ratio < 1.0
|
|
assert "#[derive(Debug, Clone, PartialEq)]\npub struct Item {" in result.compressed
|
|
assert '#[inline]\n#[cfg(feature = "extra")]\npub fn log_item' in result.compressed
|
|
|
|
def test_doc_comment_stays_ahead_of_attributes(self):
|
|
code = (
|
|
"/// Records one item.\n"
|
|
"#[inline]\n"
|
|
"pub fn record(item: &Item, sink: &mut Vec<u64>) {\n"
|
|
" sink.push(item.id);\n"
|
|
" sink.push(item.id + 1);\n"
|
|
" sink.push(item.id + 2);\n"
|
|
"}\n"
|
|
)
|
|
|
|
result = self._compressor().compress(code, language="rust")
|
|
|
|
assert result.compression_ratio < 1.0
|
|
assert "/// Records one item.\n#[inline]\npub fn record" in result.compressed
|
|
|
|
def test_attribute_attaches_across_blank_line(self):
|
|
"""Rust allows whitespace between an outer attribute and its item."""
|
|
code = (
|
|
"pub struct Item { pub id: u64 }\n"
|
|
"\n"
|
|
"#[inline]\n"
|
|
"\n"
|
|
"pub fn record(item: &Item, sink: &mut Vec<u64>) {\n"
|
|
" sink.push(item.id);\n"
|
|
" sink.push(item.id + 1);\n"
|
|
" sink.push(item.id + 2);\n"
|
|
"}\n"
|
|
)
|
|
|
|
result = self._compressor().compress(code, language="rust")
|
|
|
|
assert result.compression_ratio < 1.0
|
|
assert result.syntax_valid is True
|
|
assert "#[inline]\npub fn record" in result.compressed
|
|
assert not result.compressed.rstrip().endswith("#[inline]")
|
|
|
|
def test_attribute_attaches_through_detached_comment(self):
|
|
"""A comment between an attribute and its item must not detach it."""
|
|
code = (
|
|
"pub struct Item { pub id: u64 }\n"
|
|
"\n"
|
|
"#[inline]\n"
|
|
"\n"
|
|
"// Keep with the function below.\n"
|
|
"\n"
|
|
"pub fn record(item: &Item, sink: &mut Vec<u64>) {\n"
|
|
" sink.push(item.id);\n"
|
|
" sink.push(item.id + 1);\n"
|
|
" sink.push(item.id + 2);\n"
|
|
" sink.push(item.id + 3);\n"
|
|
"}\n"
|
|
)
|
|
|
|
result = self._compressor().compress(code, language="rust")
|
|
|
|
assert result.compression_ratio < 1.0
|
|
assert result.syntax_valid is True
|
|
assert "#[inline]\npub fn record" in result.compressed
|
|
assert not result.compressed.rstrip().endswith("#[inline]")
|
|
|
|
|
|
@pytest.mark.skipif(not TREE_SITTER_INSTALLED, reason="tree-sitter-languages not installed")
|
|
class TestSemicolonSeparatedStatements:
|
|
"""``a = 1; b = 2`` must not corrupt or duplicate the compressed output."""
|
|
|
|
def _compressor(self):
|
|
return CodeAwareCompressor(
|
|
CodeCompressorConfig(
|
|
min_tokens_for_compression=1,
|
|
max_body_lines=5,
|
|
enable_ccr=False,
|
|
)
|
|
)
|
|
|
|
def test_top_level_separator_does_not_become_a_bare_line(self):
|
|
"""Python exposes the ``;`` as a root-level token; emitting it as a
|
|
standalone fragment puts a bare ``;`` on its own line and makes the
|
|
whole result invalid, so compression falls back to the original."""
|
|
code = (
|
|
"import json\n"
|
|
"B='/tmp/'\n"
|
|
"v=json.load(open(B+'v.json')); m=json.load(open(B+'m.json'))\n"
|
|
"print('oe', m['order_expire']); print('pay', m['pay_url'])\n"
|
|
)
|
|
result = self._compressor().compress(code, language="python")
|
|
|
|
assert result.syntax_valid is True
|
|
assert result.compressed != code
|
|
assert [line for line in result.compressed.splitlines() if line.strip() == ";"] == []
|
|
assert "v=json.load(open(B+'v.json'))" in result.compressed
|
|
assert "m=json.load(open(B+'m.json'))" in result.compressed
|
|
|
|
def test_class_body_statement_is_not_duplicated(self):
|
|
code = "class C:\n x=1; y=2\n z=3\n"
|
|
result = self._compressor().compress(code, language="python")
|
|
|
|
assert result.syntax_valid is True
|
|
assert result.compressed.count("x=1; y=2") == 1
|
|
|
|
def test_class_body_statement_continuation_is_kept(self):
|
|
"""The second statement starts on the already-emitted row and
|
|
continues on the next one; trimming the shared row must not drop
|
|
the continuation line."""
|
|
code = "class C:\n x=1; y=(2 +\n 3)\n z=4\n"
|
|
result = self._compressor().compress(code, language="python")
|
|
|
|
assert result.syntax_valid is True
|
|
assert result.compressed.count("x=1; y=(2 +") == 1
|
|
assert result.compressed.count(" 3)") == 1
|
|
assert result.compressed.count("z=4") == 1
|
|
|
|
def test_import_after_statement_keeps_source_order(self, tmp_path):
|
|
"""Imports are hoisted to the top of the output, but an import that
|
|
follows executable code must not jump above it: importing the plugin
|
|
before the path update fails at runtime."""
|
|
vendor = tmp_path / "vendor"
|
|
vendor.mkdir()
|
|
module_name = "semicolon_order_plugin"
|
|
(vendor / f"{module_name}.py").write_text("VALUE = 42\n")
|
|
code = f"import sys; sys.path.insert(0, {str(vendor)!r}); import {module_name}\n"
|
|
|
|
result = self._compressor().compress(code, language="python")
|
|
|
|
assert result.syntax_valid is True
|
|
assert result.compressed != code
|
|
assert (
|
|
result.compressed.index("import sys")
|
|
< result.compressed.index("sys.path.insert")
|
|
< result.compressed.index(f"import {module_name}")
|
|
)
|
|
|
|
saved_path = list(sys.path)
|
|
saved_modules = set(sys.modules)
|
|
try:
|
|
exec(result.compressed, {})
|
|
assert module_name in sys.modules
|
|
finally:
|
|
sys.path[:] = saved_path
|
|
for name in set(sys.modules) - saved_modules:
|
|
del sys.modules[name]
|
|
|
|
def test_declaration_after_top_level_code_keeps_source_order(self):
|
|
"""Declarations are assembled before the top-level code bucket, so a
|
|
class defined after executable code would move ahead of it. Safe
|
|
reassembly cannot be guaranteed, so the original snippet is kept."""
|
|
code = "import sys; marker = 1; import decimal\nclass Money(decimal.Decimal):\n pass\n"
|
|
result = self._compressor().compress(code, language="python")
|
|
|
|
assert result.syntax_valid is True
|
|
assert result.compressed == code
|
|
namespace: dict[str, object] = {}
|
|
exec(result.compressed, namespace)
|
|
assert issubclass(namespace["Money"], decimal.Decimal)
|
|
|
|
def test_decorator_default_after_top_level_code_keeps_source_order(self):
|
|
"""A decorator and a default expression are evaluated when the
|
|
function is defined, so they cannot move ahead of the import they
|
|
use."""
|
|
code = (
|
|
"import sys; marker = 1; import functools\n"
|
|
"@functools.lru_cache(maxsize=None)\n"
|
|
"def compute(value=marker):\n"
|
|
" return value\n"
|
|
)
|
|
result = self._compressor().compress(code, language="python")
|
|
|
|
assert result.syntax_valid is True
|
|
assert result.compressed == code
|
|
namespace: dict[str, object] = {}
|
|
exec(result.compressed, namespace)
|
|
assert namespace["compute"]() == 1
|
|
|
|
def test_function_body_keeps_each_line_once(self):
|
|
"""Both statements share one physical line. The line slice already
|
|
carries the whole line, so adding the second statement's range again
|
|
duplicates it; dropping it instead would lose a continuation line."""
|
|
code = (
|
|
"def f():\n"
|
|
" x=1; y=(2 +\n"
|
|
" 3)\n"
|
|
" a=3\n"
|
|
" b=4\n"
|
|
" c=5\n"
|
|
" d=6\n"
|
|
" e=7\n"
|
|
" return x+y\n"
|
|
)
|
|
compressor = self._compressor()
|
|
parser = cc._get_parser("python")
|
|
root = parser.parse(code.encode("utf-8")).root_node
|
|
analysis = compressor._analyze_symbol_importance(root, code, CodeLanguage.PYTHON, "")
|
|
compressed = compressor._compress_function_ast(
|
|
root.children[0],
|
|
code,
|
|
CodeLanguage.PYTHON,
|
|
cc._LANG_CONFIGS[CodeLanguage.PYTHON],
|
|
{},
|
|
analysis,
|
|
)
|
|
|
|
assert compressed.count("x=1; y=(2 +") == 1
|
|
assert compressed.count(" 3)") == 1
|