Files
headroom/tests/test_compression/test_code_handler.py
Abhay Singh d5e5534b47 perf(compression/code): avoid a UTF-8 copy per code block in byte-to-char mapping (#3169)
## Description

`CodeStructureHandler._byte_spans_to_char_spans` converts tree-sitter's
byte offsets to character offsets for each compressed code block. Its
ASCII fast path was:

```python
n_bytes = len(content.encode("utf-8"))
if n_bytes == len(content):
    return spans
```

`content.encode("utf-8")` allocates a full UTF-8 byte copy of the whole
code block, which is then discarded — just to compare its length against
`len(content)`. For the common case (ASCII source, byte == char),
`content.isascii()` answers the same question in C with no allocation.

The non-ASCII branch additionally re-encoded **every character** in the
loop (`len(ch.encode("utf-8"))`) to get its UTF-8 width — one small
allocation per character. A character's width (1-4 bytes) is fully
determined by its code point, so it can be derived with `ord(ch)` and no
encoding.

This runs per code block on the compression path, so for code-heavy
traffic the discarded per-block allocation adds up. Output is
byte-for-byte identical.

Benchmark (`_byte_spans_to_char_spans` on a ~20 KB ASCII block, mean of
20000 calls):

```
old (len(encode) == len) : 0.45 us/call  (+ a 20 KB bytes allocation)
new (isascii())          : 0.06 us/call  (~7x faster, no allocation)
```

Correctness was verified by comparing the old and new implementations on
mixed content (`café 🚀 语言` — 2/4/3-byte characters): the remapped span
offsets are identical.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/code_handler.py`
(`_byte_spans_to_char_spans`): use `content.isascii()` for the fast-path
check (no UTF-8 allocation), and derive each character's UTF-8 width
from its code point instead of re-encoding it per character.
- `tests/test_compression/test_code_handler.py`: added
`TestByteSpanToCharSpan` — ASCII spans returned unchanged, and non-ASCII
byte offsets (café / rocket / CJK) remapped to the correct character
offsets including the exclusive-end boundary.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_compression/test_code_handler.py  ->  24 passed, 8 skipped (tree-sitter not installed)
uvx ruff@0.16.2 check headroom/compression/handlers/code_handler.py tests/test_compression/test_code_handler.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/compression/handlers/code_handler.py  ->  Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: ran the old and new implementations side by
side on ASCII and on mixed non-ASCII content (`café 🚀 语言`) and confirmed
identical remapped span offsets; benchmarked the ASCII path (0.45us ->
0.06us, ~7x, and the new path allocates nothing). Added the regression
test and ran the file.
- Observed result: identical span mapping; the per-block UTF-8
allocation on the ASCII fast path is gone, and the non-ASCII loop no
longer allocates per character.
- Not tested: an end-to-end code-compression request through
`UniversalCompressor`; the static helper is exercised directly with
byte-offset spans, which is exactly what tree-sitter feeds it.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is an internal offset-mapping
helper in the code structure handler, not a rollout-channel-gated
runtime feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. The returned spans are identical
for ASCII and non-ASCII content; only the work done to compute them is
cheaper.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: less per-block allocation/CPU when mapping code
spans; identical output.
- Rollback path: revert this PR; the helper returns to encoding the
content for the length check and per-character.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal helper, output unchanged)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Scope is deliberately small and behavior-preserving: `content.isascii()`
is exactly equivalent to "every UTF-8 byte is one character," and the
code-point width table (`<0x80 -> 1`, `<0x800 -> 2`, `<0x10000 -> 3`,
else `4`) matches Python's UTF-8 encoding for every `str` character.
2026-09-30 11:28:39 -05:00

393 lines
15 KiB
Python

"""Tests for code structure handler."""
from unittest.mock import patch
import pytest
from headroom.compression.handlers.code_handler import (
CodeStructureHandler,
_check_tree_sitter,
is_tree_sitter_available,
)
requires_tree_sitter = pytest.mark.skipif(
not is_tree_sitter_available(),
reason="tree-sitter-language-pack not installed",
)
class TestCanHandle:
@pytest.fixture
def handler(self):
return CodeStructureHandler()
def test_detects_python(self, handler):
assert handler.can_handle("def foo():\n pass\n") is True
def test_detects_javascript(self, handler):
assert handler.can_handle("function foo() { return 1; }") is True
def test_rejects_prose(self, handler):
assert handler.can_handle("This is a plain sentence.") is False
class TestRegexFallback:
"""Regex path runs regardless of tree-sitter availability."""
@pytest.fixture
def handler(self):
return CodeStructureHandler(use_tree_sitter=False)
def test_python_signature_preserved_body_compressible(self, handler):
code = "def hello(name: str) -> str:\n message = name\n return message\n"
result = handler.get_mask(code, language="python")
assert result.metadata["parser"] == "regex"
sig = "def hello(name: str) -> str:"
start = code.index(sig)
assert all(result.mask.mask[i] for i in range(start, start + len(sig)))
body_char = code.index("message = name")
assert result.mask.mask[body_char] is False
def test_python_import_preserved(self, handler):
code = "import os\n\nx = 1\n"
result = handler.get_mask(code, language="python")
assert all(result.mask.mask[i] for i in range(len("import os")))
class TestLanguageDetection:
@pytest.fixture
def handler(self):
return CodeStructureHandler()
def test_detects_python(self, handler):
code = "import os\n\nclass Foo:\n def method(self):\n pass\n"
assert handler._detect_language(code) == "python"
def test_detects_go(self, handler):
code = 'package main\n\nimport (\n\t"fmt"\n)\n\nfunc main() {\n}\n'
assert handler._detect_language(code) == "go"
def test_detects_rust(self, handler):
code = "use std::io;\n\npub fn main() {\n let mut x = 1;\n}\n"
assert handler._detect_language(code) == "rust"
def test_detects_perl(self, handler):
code = "use strict;\npackage Foo;\n\nsub greet {\n my $name = shift;\n return $name;\n}\n"
assert handler._detect_language(code) == "perl"
def test_falls_back_to_default(self):
handler = CodeStructureHandler(default_language="javascript")
assert handler._detect_language("plain words only here") == "javascript"
class TestRegexFallbackLanguages:
"""Signature/import preservation on the regex path across languages."""
@pytest.fixture
def handler(self):
return CodeStructureHandler(use_tree_sitter=False)
def test_go_func_signature_preserved(self, handler):
code = "func Add(a int, b int) int {\n\treturn a + b\n}\n"
result = handler.get_mask(code, language="go")
sig = "func Add(a int, b int)"
start = code.index(sig)
assert all(result.mask.mask[i] for i in range(start, start + len(sig)))
def test_rust_fn_signature_preserved(self, handler):
code = "pub fn add(a: i32, b: i32) -> i32 {\n a + b\n}\n"
result = handler.get_mask(code, language="rust")
sig = "pub fn add(a: i32, b: i32)"
start = code.index(sig)
assert all(result.mask.mask[i] for i in range(start, start + len(sig)))
def test_typescript_interface_preserved(self, handler):
code = "interface Shape {\n area(): number;\n}\n\nconst x = 1;\n"
result = handler.get_mask(code, language="typescript")
sig = "interface Shape"
start = code.index(sig)
assert all(result.mask.mask[i] for i in range(start, start + len(sig)))
def test_javascript_arrow_function_preserved(self, handler):
code = "const add = (a, b) => {\n return a + b;\n};\n"
result = handler.get_mask(code, language="javascript")
sig = "const add = (a, b) =>"
start = code.index(sig)
assert all(result.mask.mask[i] for i in range(start, start + len(sig)))
def test_perl_sub_signature_preserved(self, handler):
code = "sub add {\n my ($a, $b) = @_;\n return $a + $b;\n}\n"
result = handler.get_mask(code, language="perl")
sig = "sub add"
start = code.index(sig)
assert all(result.mask.mask[i] for i in range(start, start + len(sig)))
def test_perl_use_import_preserved(self, handler):
code = "use strict;\nuse warnings;\n\nmy $x = 1;\n"
result = handler.get_mask(code, language="perl")
assert all(result.mask.mask[i] for i in range(len("use strict")))
def test_regex_confidence_lower_than_tree_sitter(self, handler):
result = handler.get_mask("def f():\n pass\n", language="python")
assert result.confidence == 0.7
class TestAvailabilityProbe:
"""_check_tree_sitter must exercise a real parse, not just an import."""
def test_abi_mismatch_returns_false(self):
import types
import headroom.compression.handlers.code_handler as mod
mod._tree_sitter_available = None
fake_ts = types.ModuleType("tree_sitter")
class FakeParser:
def __setattr__(self, name, value):
if name == "language":
raise RuntimeError("ABI mismatch")
super().__setattr__(name, value)
fake_ts.Parser = FakeParser
fake_pack = types.ModuleType("tree_sitter_language_pack")
fake_pack.get_language = lambda name: object()
with patch.dict(
"sys.modules",
{
"tree_sitter": fake_ts,
"tree_sitter_language_pack": fake_pack,
},
):
result = _check_tree_sitter()
assert result is False
mod._tree_sitter_available = None
@requires_tree_sitter
def test_healthy_install_returns_true(self):
import headroom.compression.handlers.code_handler as mod
mod._tree_sitter_available = None
assert _check_tree_sitter() is True
mod._tree_sitter_available = None
class TestEdgeCases:
@pytest.fixture
def handler(self):
return CodeStructureHandler()
def test_empty_content(self, handler):
result = handler.get_mask("")
assert result.confidence == 0.0
assert result.metadata.get("empty") is True
def test_whitespace_only_content(self, handler):
result = handler.get_mask(" \n\n ")
assert result.metadata.get("empty") is True
def test_unknown_language_regex_no_patterns(self):
"""A language with no regex patterns yields an all-compressible
mask rather than raising."""
handler = CodeStructureHandler(use_tree_sitter=False)
code = "BEGIN\n WRITELN('hello')\nEND.\n"
result = handler.get_mask(code, language="pascal")
assert not any(result.mask.mask)
def test_mask_length_matches_content(self, handler):
code = "def f():\n return 1\n"
result = handler.get_mask(code, language="python")
assert len(result.mask.mask) == len(code)
@requires_tree_sitter
class TestTreeSitterContainers:
"""Container bodies must stay compressible (signature-only spans).
Regression: class_definition / decorated_definition / impl_item were
marked structural over their FULL span, so every method body inside a
class (i.e. most real code) was preserved and compression no-opped at
confidence 0.95.
"""
@pytest.fixture
def handler(self):
return CodeStructureHandler()
def test_class_method_bodies_compressible(self, handler):
code = (
"class Foo:\n"
" def method_a(self):\n"
" body_line_a = 1\n"
" return body_line_a\n"
"\n"
" def method_b(self):\n"
" body_line_b = 2\n"
" return body_line_b\n"
)
result = handler.get_mask(code, language="python")
assert result.metadata["parser"] == "tree-sitter"
# Class signature and method signatures preserved
assert all(result.mask.mask[i] for i in range(len("class Foo:")))
sig = "def method_a(self):"
start = code.index(sig)
assert all(result.mask.mask[i] for i in range(start, start + len(sig)))
# Method bodies compressible
for body in ("body_line_a = 1", "body_line_b = 2"):
start = code.index(body)
assert not any(result.mask.mask[i] for i in range(start, start + len(body))), (
f"method body {body!r} must be compressible"
)
def test_decorated_function_body_compressible(self, handler):
code = "@decorator\ndef decorated():\n body_line = 4\n return body_line\n"
result = handler.get_mask(code, language="python")
# Decorator and signature preserved
assert all(result.mask.mask[i] for i in range(len("@decorator")))
sig = "def decorated():"
start = code.index(sig)
assert all(result.mask.mask[i] for i in range(start, start + len(sig)))
# Body compressible
start = code.index("body_line = 4")
assert not any(result.mask.mask[i] for i in range(start, start + len("body_line = 4"))), (
"decorated function body must be compressible"
)
def test_module_function_body_compressible(self, handler):
code = "def standalone():\n body_line = 3\n return body_line\n"
result = handler.get_mask(code, language="python")
start = code.index("body_line = 3")
assert not any(result.mask.mask[i] for i in range(start, start + len("body_line = 3")))
def test_rust_impl_method_bodies_compressible(self, handler):
code = (
"struct Foo { x: i32 }\n"
"impl Foo {\n"
" fn method(&self) -> i32 {\n"
" let body_line = 5;\n"
" body_line\n"
" }\n"
"}\n"
)
result = handler.get_mask(code, language="rust")
# impl signature preserved
start = code.index("impl Foo")
assert all(result.mask.mask[i] for i in range(start, start + len("impl Foo")))
# method body compressible
start = code.index("let body_line = 5;")
assert not any(
result.mask.mask[i] for i in range(start, start + len("let body_line = 5;"))
), "impl method body must be compressible"
def test_concurrent_parsing_uses_tree_sitter(self, handler):
"""Parsers must be thread-local.
Regression: parsers were cached in a process-global dict and
shared across threads. tree-sitter Parser objects are pyo3
unsendable — touching one from a non-creator thread panics (or
raises, dropping the handler to the regex fallback). Parsing
from a thread pool must succeed on the tree-sitter path in
every thread.
"""
from concurrent.futures import ThreadPoolExecutor
code = "class Foo:\n def m(self):\n x = 1\n return x\n"
def work(_: int) -> str:
result = handler.get_mask(code, language="python")
return str(result.metadata["parser"])
with ThreadPoolExecutor(max_workers=4) as pool:
parsers = list(pool.map(work, range(16)))
assert parsers == ["tree-sitter"] * 16, (
f"all threads must parse via tree-sitter, got: {set(parsers)}"
)
def test_non_ascii_content_mask_alignment(self, handler):
"""Byte offsets must be converted to char offsets.
Regression: tree-sitter reports byte offsets into the UTF-8
encoding, but the mask is char-indexed. Multi-byte characters
(here: accents + an emoji, 9 extra bytes) shifted every later
span, preserving the wrong characters.
"""
code = (
"# café münü 🎉 comment\n"
"def target(x: int) -> int:\n"
" body_value = 9\n"
" return body_value\n"
)
result = handler.get_mask(code, language="python")
sig = "def target(x: int) -> int:"
start = code.index(sig)
assert all(result.mask.mask[i] for i in range(start, start + len(sig))), (
"signature after non-ASCII content must be exactly preserved"
)
bstart = code.index("body_value = 9")
assert not any(
result.mask.mask[i] for i in range(bstart, bstart + len("body_value = 9"))
), "body after non-ASCII content must stay compressible"
def test_preservation_ratio_sane_for_class_code(self, handler):
"""A class with substantial method bodies should NOT preserve
everything — the whole point of the handler."""
body = "\n".join(f" line_{i} = {i}" for i in range(20))
code = f"class Big:\n def method(self):\n{body}\n return 0\n"
result = handler.get_mask(code, language="python")
assert result.preservation_ratio < 0.5, (
f"class code preserved {result.preservation_ratio:.0%} — "
"container bodies are leaking into the structural mask"
)
class TestByteSpanToCharSpan:
"""_byte_spans_to_char_spans must map tree-sitter byte offsets to character
offsets. The ASCII fast path returns spans unchanged (byte == char) without
allocating a UTF-8 copy; the non-ASCII path derives each character's width
from its code point instead of re-encoding it."""
def test_ascii_returns_spans_unchanged(self):
from headroom.compression.handlers.code_handler import CodeSpan
spans = [
CodeSpan(start=0, end=5, role="sig", is_structural=True),
CodeSpan(start=6, end=11, role="body", is_structural=False),
]
out = CodeStructureHandler._byte_spans_to_char_spans(spans, "def f():\n pass\n")
assert [(s.start, s.end) for s in out] == [(0, 5), (6, 11)]
def test_non_ascii_remaps_byte_offsets_to_char_offsets(self):
from headroom.compression.handlers.code_handler import CodeSpan
# café (é = 2 bytes), 🚀 (4 bytes), 语言 (3 bytes each).
content = "x = 'café 🚀 语言'\n"
enc = content.encode("utf-8")
# A structural span covering the whole line, plus one covering 'return'-like
# region: give byte offsets and expect character offsets back.
whole = CodeSpan(start=0, end=len(enc), role="body", is_structural=True)
# byte offset of the closing quote vs its character offset
quote_byte = enc.rindex(b"'")
quote_char = content.rindex("'")
marker = CodeSpan(start=quote_byte, end=len(enc), role="sig", is_structural=True)
out = CodeStructureHandler._byte_spans_to_char_spans([whole, marker], content)
assert out[0].start == 0
assert out[0].end == len(content) # exclusive end maps to char length
assert out[1].start == quote_char # byte offset correctly remapped
assert out[1].end == len(content)