Commit Graph
7 Commits
Author SHA1 Message Date
Abhay Singh d5e5534b47 perf(compression/code): avoid a UTF-8 copy per code block in byte-to-char mapping (#3169)
## Description

`CodeStructureHandler._byte_spans_to_char_spans` converts tree-sitter's
byte offsets to character offsets for each compressed code block. Its
ASCII fast path was:

```python
n_bytes = len(content.encode("utf-8"))
if n_bytes == len(content):
    return spans
```

`content.encode("utf-8")` allocates a full UTF-8 byte copy of the whole
code block, which is then discarded — just to compare its length against
`len(content)`. For the common case (ASCII source, byte == char),
`content.isascii()` answers the same question in C with no allocation.

The non-ASCII branch additionally re-encoded **every character** in the
loop (`len(ch.encode("utf-8"))`) to get its UTF-8 width — one small
allocation per character. A character's width (1-4 bytes) is fully
determined by its code point, so it can be derived with `ord(ch)` and no
encoding.

This runs per code block on the compression path, so for code-heavy
traffic the discarded per-block allocation adds up. Output is
byte-for-byte identical.

Benchmark (`_byte_spans_to_char_spans` on a ~20 KB ASCII block, mean of
20000 calls):

```
old (len(encode) == len) : 0.45 us/call  (+ a 20 KB bytes allocation)
new (isascii())          : 0.06 us/call  (~7x faster, no allocation)
```

Correctness was verified by comparing the old and new implementations on
mixed content (`café 🚀 语言` — 2/4/3-byte characters): the remapped span
offsets are identical.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/code_handler.py`
(`_byte_spans_to_char_spans`): use `content.isascii()` for the fast-path
check (no UTF-8 allocation), and derive each character's UTF-8 width
from its code point instead of re-encoding it per character.
- `tests/test_compression/test_code_handler.py`: added
`TestByteSpanToCharSpan` — ASCII spans returned unchanged, and non-ASCII
byte offsets (café / rocket / CJK) remapped to the correct character
offsets including the exclusive-end boundary.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_compression/test_code_handler.py  ->  24 passed, 8 skipped (tree-sitter not installed)
uvx ruff@0.16.2 check headroom/compression/handlers/code_handler.py tests/test_compression/test_code_handler.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/compression/handlers/code_handler.py  ->  Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: ran the old and new implementations side by
side on ASCII and on mixed non-ASCII content (`café 🚀 语言`) and confirmed
identical remapped span offsets; benchmarked the ASCII path (0.45us ->
0.06us, ~7x, and the new path allocates nothing). Added the regression
test and ran the file.
- Observed result: identical span mapping; the per-block UTF-8
allocation on the ASCII fast path is gone, and the non-ASCII loop no
longer allocates per character.
- Not tested: an end-to-end code-compression request through
`UniversalCompressor`; the static helper is exercised directly with
byte-offset spans, which is exactly what tree-sitter feeds it.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is an internal offset-mapping
helper in the code structure handler, not a rollout-channel-gated
runtime feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. The returned spans are identical
for ASCII and non-ASCII content; only the work done to compute them is
cheaper.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: less per-block allocation/CPU when mapping code
spans; identical output.
- Rollback path: revert this PR; the helper returns to encoding the
content for the length check and per-character.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal helper, output unchanged)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Scope is deliberately small and behavior-preserving: `content.isascii()`
is exactly equivalent to "every UTF-8 byte is one character," and the
code-point width table (`<0x80 -> 1`, `<0x800 -> 2`, `<0x10000 -> 3`,
else `4`) matches Python's UTF-8 encoding for every `str` character.
2026-09-30 11:28:39 -05:00
Shengbo_Wang 1de35e775f fix(code): parse-probe tree-sitter availability in code_handler (#1231) (#1300)
## Description

`_check_tree_sitter()` in
`headroom/compression/handlers/code_handler.py` only verified that
`tree_sitter_language_pack` could be imported. When tree-sitter core and
the language pack are built against different ABIs, the import succeeds
but `parser.language = get_language(...)` raises at request time,
silently falling back to the generic text compressor — with no warning,
while the banner still reports code-aware as enabled.

#1299 fixed the same class of bug in `transforms/code_compressor.py`.
This PR is the defensive follow-up tracked by #1231: it applies the same
parse probe to the compression **structure handler** so both code-aware
paths are consistent.

Closes #1231

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Replace the import-only probe in `code_handler._check_tree_sitter()`
with a real parse probe: construct a `Parser`, assign a Python language,
and parse `b"x = 1\n"` — if any step fails, mark unavailable
- Log a WARNING when import succeeds but parsing fails, so the downgrade
is visible
- Add `TestAvailabilityProbe` covering the simulated ABI mismatch (->
False) and healthy install (-> True) cases

> Note: an earlier revision of this PR also touched
`transforms/code_compressor.py`, but that fix landed independently via
#1299. After rebasing onto current `main`, this PR is scoped to the
remaining `code_handler.py` gap only.

## Testing

- [x] Unit tests pass (`pytest
tests/test_compression/test_code_handler.py` -> 22 passed, 8 skipped)
- [x] Linting / formatting pass (`ruff check .`, `ruff format --check
.`)
- [x] New tests added for new functionality

### Real Behavior Proof

- Setup: Windows 11 (GBK locale), Python 3.10, tree-sitter NOT installed
- Before fix: `_check_tree_sitter()` returns `True` on a
partial/ABI-mismatched install (import succeeds), then silently degrades
to the text compressor at request time with no warning
- After fix: the probe parses a trivial snippet; an ABI mismatch is
caught at probe time, `_check_tree_sitter()` returns `False`, and a
WARNING is logged.
`TestAvailabilityProbe::test_abi_mismatch_returns_false` reproduces this
with a fake Parser whose `language` setter raises.

Signed-off-by: RTCartist <wangshengb@buaa.edu.cn>
2026-07-09 14:06:29 -05:00
Veesh GoldmanandClaude Opus 4.8 f39858c233 feat(code): add Perl support to code-aware compressor (#1125)
## Description

Adds Perl as a supported language for `CodeAwareCompressor` /
`CodeStructureHandler`. Function bodies are compressed while
`use`/`require` imports, `sub`/`method` signatures, and
`package`/`class`/`role` declarations are preserved — bringing Perl up
to parity with the other Tier-2 languages.

No new dependencies: the Perl grammar already ships in
`tree-sitter-language-pack` (already a Headroom dependency), so this is
pure configuration.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `code_handler.py`: Perl entries in the four per-language tables —
`_STRUCTURAL_NODE_TYPES`, `_SIGNATURE_PATTERNS` (regex fallback),
`_LANGUAGE_MARKERS` (detection), `_IMPORT_PATTERNS`. The existing
`_CONTAINER_BODY_TYPES` already covers Perl's `block` body node, so no
change was needed there.
- `code_compressor.py`: `CodeLanguage.PERL` enum value, a data-driven
`LangConfig`, a `_LANGUAGE_PREFILTER` entry, and the supported-language
string in the parser error message.
- Node-type names (`subroutine_declaration_statement`,
`package_statement`, `signature`, `block`, …) are from the
`tree-sitter-perl/tree-sitter-perl` grammar (MIT).
- Tests: 1 detection test + 2 regex-path signature/import-preservation
tests.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/test_code_handler.py -q
collected 28 items
tests/test_compression/test_code_handler.py ............................ [100%]
======================== 28 passed, 1 warning in 2.53s =========================

$ ruff check headroom/compression/handlers/code_handler.py headroom/transforms/code_compressor.py tests/test_compression/test_code_handler.py
All checks passed!

$ mypy headroom/compression/handlers/code_handler.py headroom/transforms/code_compressor.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Python 3.12, `pip install -e ".[code]"`
(tree-sitter-language-pack installed, `is_tree_sitter_available() ==
True`).
- Exact command / steps: ran `CodeStructureHandler().get_mask(code,
language="perl")` on a real Perl module (package + two subs with
bodies).
- Observed result: detected as `perl`, parsed via the `tree-sitter` path
(not regex), and the preserved span was exactly the imports + package +
sub signatures, with both sub bodies marked compressible:

```text
tree-sitter available: True
parser: tree-sitter | detected: perl
--- PRESERVED (signatures/imports/structure) ---
use strict;use warnings;package Greeter;sub new sub greet
```

- Not tested: the full proxy/MCP server end-to-end path (out of scope —
this PR only touches the code compressor's language tables).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Docs/CHANGELOG left unchecked — happy to add a Perl line to either if
you'd like; I wasn't sure of your preferred location.
- *Disclosure: I maintain the upstream `tree-sitter-perl` grammar this
relies on. It's already a transitive dependency of Headroom via
`tree-sitter-language-pack` — this PR only adds config to use it, with
no dependency changes.*

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 18:47:41 -05:00
6cdb846200 fix(compression): use thread-local tree-sitter parsers in code handler (#893)
## Description

`CodeStructureHandler` cached tree-sitter parsers in a process-global
dict; the lock only guarded creation, while `parse()` ran unlocked on
any thread. tree-sitter `Parser` objects are pyo3 `unsendable` — using
one from a non-creator thread panics. The proxy invokes handlers from
executor pool threads, so a shared parser is an eventual crash. Same
class already fixed in `transforms/code_compressor.py` (#604). Stacked
on #892.

Closes # <!-- compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/code_handler.py`: one parser per
(thread, language) via `threading.local()`, porting the pattern from
`transforms/code_compressor.py`.
- `tests/test_compression/test_code_handler.py`: regression test parsing
from a 4-worker thread pool, asserting every call stays on the
tree-sitter path.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/ -q
94 passed, 8 skipped
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, tree-sitter-language-pack
1.8.1, branch `fix/code-threadlocal-parsers` (stacked on #892).
- Exact command / steps: `pytest tests/test_compression/ -q`.
- Observed result: 16 parses across a 4-worker pool all stay on the
tree-sitter path with no pyo3 panic; previously a shared parser would be
touched cross-thread.
- Not tested: Reproducing the original panic under production
concurrency (covered structurally by the thread-pool test).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library change. See Test Output.

## Additional Notes

Stacked on #892 — review the top commit until that merges. PR 5 of 7.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-06-15 16:38:08 -05:00
AshishandClaude Fable 5 615e1ed6f5 test(compression): fill code handler coverage gaps (#895)
## Description

`CodeStructureHandler` had zero dedicated tests before this series —
which is exactly why the P0 bugs in #890/#892/#893 went unnoticed. This
fills the remaining coverage gaps beyond the per-fix regression tests.
Stacked on #893.

Closes # <!-- compression-handler review -->

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `tests/test_compression/test_code_handler.py`: language detection
(python/go/rust + default fallback); regex-path signature/import
preservation across go/rust/typescript/javascript; regex confidence
value; empty/whitespace content; unknown language; mask-length
invariant.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/test_code_handler.py -q
25 passed
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, tree-sitter-language-pack
1.8.1, branch `test/code-handler-coverage` (stacked on #893).
- Exact command / steps: `pytest
tests/test_compression/test_code_handler.py -q`.
- Observed result: 25 tests pass; tree-sitter classes skip cleanly when
the pack is absent, regex-path tests always run.
- Not tested: N/A — this PR is tests only.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — tests only. See Test Output.

## Additional Notes

Stacked on #893 — review the top commit until that merges. PR 6 of 7.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 10:25:57 -05:00
AshishandClaude Fable 5 b1f700fc27 fix(compression): convert tree-sitter byte offsets to char offsets (#892)
## Description

tree-sitter reports node positions as byte offsets into the UTF-8
encoding, but `CodeStructureHandler` builds a character-indexed mask.
Any multi-byte character (accents, emoji, CJK in
docstrings/comments/strings) shifted every subsequent span, preserving
the wrong characters and leaking signature bytes into bodies. Stacked on
#890.

Closes # <!-- compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/code_handler.py`: remap spans through a
byte->char table before masking; pure-ASCII content (byte == char) skips
the conversion.
- `tests/test_compression/test_code_handler.py`: regression test with
`café münü 🎉` in a comment, asserting the following signature and body
are correctly aligned.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/ -q
93 passed, 8 skipped
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, tree-sitter-language-pack
1.8.1, branch `fix/code-byte-char-offsets` (stacked on #890).
- Exact command / steps: `pytest tests/test_compression/ -q`.
- Observed result: With 9 extra UTF-8 bytes ahead of it, a function
signature is exactly preserved and its body stays compressible; before,
the offsets were shifted.
- Not tested: End-to-end through the live proxy pipeline.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library change. See Test Output.

## Additional Notes

Stacked on #890 — review the top commit until that merges. PR 4 of 7.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 10:24:21 -05:00
AshishandClaude Fable 5 16ed73bca6 fix(compression): keep container bodies compressible in code handler (#890)
## Description

Two bugs in `CodeStructureHandler`'s tree-sitter path. (1) Container
nodes (class/impl/trait/decorated definitions) were marked structural
over their full span and `_spans_to_mask` never un-marks, so every
method body inside a class was preserved and compression silently
no-opped at confidence 0.95. (2) Discovered while testing:
`tree-sitter-language-pack >= 1.0` switched to a Rust binding (methods,
not attributes; `parse(str)`), so the handler raised `TypeError` on
every call and silently fell back to regex.

Closes # <!-- compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/code_handler.py`: containers emit a
signature-only span (start to body start); recursion gives nested
functions their own signature/body split; decorated definitions emit no
whole-node span.
- `headroom/compression/handlers/code_handler.py`: small compat shim
supporting both the classic attribute API and the new Rust-binding
method API.
- `tests/test_compression/test_code_handler.py`: new file (the handler
had zero dedicated tests) covering class/decorated/impl body
compressibility, regex fallback, and a preservation-ratio bound.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/ -q
92 passed, 8 skipped
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, tree-sitter-language-pack 1.8.1
installed, branch `fix/code-container-bodies`.
- Exact command / steps: `pytest tests/test_compression/ -q`.
- Observed result: Class method bodies are now compressible
(preservation ratio drops from ~1.0 to roughly the signature fraction);
the tree-sitter path runs instead of falling back to regex.
- Not tested: End-to-end through the live proxy pipeline.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library change. See Test Output.

## Additional Notes

PR 3 of 7; branched fresh from main (independent of #887/#889).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 21:31:32 -07:00