Commit Graph
15 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 fc8ff5a761 refactor(compression): deprecate the test-only UniversalCompressor package (#4095)
## Description

`headroom.compression` (UniversalCompressor, StructureMask and the
JSON/code structure handlers) is not called by any product code; only
`tests/test_compression/` uses it. The one-function API people use is
`headroom.compress()`, which does not touch it.

`headroom.compression.*` was an undocumented import path. It was never
in the published docs, only in the unpublished `wiki/compression.md`,
and no consumer repo imports it. Because it is an importable package
with an `__all__`, this PR deprecates it instead of deleting it.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- The package exports (`compress`, `UniversalCompressor`,
`UniversalCompressorConfig`, `CompressionResult`, `MagikaDetector`,
`ContentType`, `StructureMask`) are now lazy. The first access emits one
`DeprecationWarning` at the caller, pointing to `headroom.compress()` or
`ContentRouter`.
- The resolved object is cached in the package globals, so a `from`
import (which looks the name up twice) warns once, not twice.
- On `importlib.reload()` the package drops its cached exports, so they
resolve against the current submodules again.
- Importing `headroom.compression.universal` directly also warns.
- `import headroom.compression.detector` stays quiet, because
ContentRouter's startup Magika preload imports it.
- `tests/test_compression/test_deprecation.py` covers attribute, `from`
and wildcard imports, reload, the universal import and the quiet
detector import.
- It restores `sys.modules`, the parent-package attribute and the export
cache through monkeypatch, so test order does not matter.
- Three of these tests fail on the previous commit, which warned twice.
- Evidence: `rg -n
'headroom\.compression|UniversalCompressor|StructureMask|MagikaDetector'
headroom crates plugins sdk examples benchmarks scripts`, plus the
consumer repos, finds only the package itself, `content_router.py`'s
detector import and unrelated OTel metric names.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression tests/test_kompress_preload_deferral.py tests/test_transforms_content_router.py tests/test_compress_api.py tests/test_proxy_warmup.py
262 passed, 8 skipped, 1 warning in 39.82s
$ ruff check / format: All checks passed;  mypy: Success
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`, this branch's worktree
with the prebuilt `headroom._core`. Temp `HOME`,
`HEADROOM_WORKSPACE_DIR` and `CODEX_HOME`; nothing touched the real
home.
- Exact command / steps: (1) shared smoke harness: a real proxy from
this checkout against a local fake upstream, with one Anthropic and one
OpenAI request carrying a compressible tool result (`smoke.sh
<worktree>`); (2) `python -W always -c 'from headroom.compression import
compress; import headroom.compression.detector'`.
- Observed result: The proxy serves both requests (200) with savings and
logs no new warnings. The from-import emits exactly one
DeprecationWarning, attributed to the caller. The detector import is
silent. Output below.
- Not tested: Third-party code using `headroom.compression`; none is
known.

```text
branch: cleanup/compression-deprecate-universal-compressor @ 4a430674
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines ($SMOKE/ws/logs/proxy-52286.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: $SMOKE)
```

```text
$ python -W always -c "from headroom.compression import compress; import headroom.compression.detector; print(\"ok\")"
<string>:2: DeprecationWarning: headroom.compression.compress is deprecated and will be removed in a future release; use headroom.compress() for one-call compression, or ContentRouter from headroom.transforms.
ok
```

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Not applicable; no rollout-managed behaviour
is touched.
- Stable/default behavior changed: Only a DeprecationWarning when the
deprecated names are used.
- Kill switch / disable path: Not applicable; the warning is a standard
DeprecationWarning and can be filtered with `warnings`.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this single commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable.

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

Documentation changes are not in this PR. They are recorded in the
cleanup docs ledger for the single follow-up docs PR.

Deprecated (not removed): the seven `headroom.compression` package
exports and the module `headroom.compression.universal`. The handlers
and masks modules warn only through the package. In
`tests/test_proxy_debug_endpoints.py`,
`test_debug_endpoints_do_not_mutate_state` failed once in a combined run
and passed on two reruns (29/29). It looks flaky and is unrelated to
this change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:02:40 -07:00
Abhay Singh d5e5534b47 perf(compression/code): avoid a UTF-8 copy per code block in byte-to-char mapping (#3169)
## Description

`CodeStructureHandler._byte_spans_to_char_spans` converts tree-sitter's
byte offsets to character offsets for each compressed code block. Its
ASCII fast path was:

```python
n_bytes = len(content.encode("utf-8"))
if n_bytes == len(content):
    return spans
```

`content.encode("utf-8")` allocates a full UTF-8 byte copy of the whole
code block, which is then discarded — just to compare its length against
`len(content)`. For the common case (ASCII source, byte == char),
`content.isascii()` answers the same question in C with no allocation.

The non-ASCII branch additionally re-encoded **every character** in the
loop (`len(ch.encode("utf-8"))`) to get its UTF-8 width — one small
allocation per character. A character's width (1-4 bytes) is fully
determined by its code point, so it can be derived with `ord(ch)` and no
encoding.

This runs per code block on the compression path, so for code-heavy
traffic the discarded per-block allocation adds up. Output is
byte-for-byte identical.

Benchmark (`_byte_spans_to_char_spans` on a ~20 KB ASCII block, mean of
20000 calls):

```
old (len(encode) == len) : 0.45 us/call  (+ a 20 KB bytes allocation)
new (isascii())          : 0.06 us/call  (~7x faster, no allocation)
```

Correctness was verified by comparing the old and new implementations on
mixed content (`café 🚀 语言` — 2/4/3-byte characters): the remapped span
offsets are identical.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/code_handler.py`
(`_byte_spans_to_char_spans`): use `content.isascii()` for the fast-path
check (no UTF-8 allocation), and derive each character's UTF-8 width
from its code point instead of re-encoding it per character.
- `tests/test_compression/test_code_handler.py`: added
`TestByteSpanToCharSpan` — ASCII spans returned unchanged, and non-ASCII
byte offsets (café / rocket / CJK) remapped to the correct character
offsets including the exclusive-end boundary.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
tests/test_compression/test_code_handler.py  ->  24 passed, 8 skipped (tree-sitter not installed)
uvx ruff@0.16.2 check headroom/compression/handlers/code_handler.py tests/test_compression/test_code_handler.py  ->  All checks passed!
uvx mypy@1.20.2 headroom/compression/handlers/code_handler.py  ->  Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.16.2 and mypy 1.20.2 via uvx.
- Exact command / steps: ran the old and new implementations side by
side on ASCII and on mixed non-ASCII content (`café 🚀 语言`) and confirmed
identical remapped span offsets; benchmarked the ASCII path (0.45us ->
0.06us, ~7x, and the new path allocates nothing). Added the regression
test and ran the file.
- Observed result: identical span mapping; the per-block UTF-8
allocation on the ASCII fast path is gone, and the non-ASCII loop no
longer allocates per character.
- Not tested: an end-to-end code-compression request through
`UniversalCompressor`; the static helper is exercised directly with
byte-offset spans, which is exactly what tree-sitter feeds it.

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is an internal offset-mapping
helper in the code structure handler, not a rollout-channel-gated
runtime feature.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no. The returned spans are identical
for ASCII and non-ASCII content; only the work done to compute them is
cheaper.
- Kill switch / disable path: N/A.
- Unsafe override required: no.
- Qualification impact: less per-block allocation/CPU when mapping code
spans; identical output.
- Rollback path: revert this PR; the helper returns to encoding the
content for the length check and per-character.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A:
internal helper, output unchanged)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Scope is deliberately small and behavior-preserving: `content.isascii()`
is exactly equivalent to "every UTF-8 byte is one character," and the
code-point width table (`<0x80 -> 1`, `<0x800 -> 2`, `<0x10000 -> 3`,
else `4`) matches Python's UTF-8 encoding for every `str` character.
2026-09-30 11:28:39 -05:00
Shengbo_Wang 1de35e775f fix(code): parse-probe tree-sitter availability in code_handler (#1231) (#1300)
## Description

`_check_tree_sitter()` in
`headroom/compression/handlers/code_handler.py` only verified that
`tree_sitter_language_pack` could be imported. When tree-sitter core and
the language pack are built against different ABIs, the import succeeds
but `parser.language = get_language(...)` raises at request time,
silently falling back to the generic text compressor — with no warning,
while the banner still reports code-aware as enabled.

#1299 fixed the same class of bug in `transforms/code_compressor.py`.
This PR is the defensive follow-up tracked by #1231: it applies the same
parse probe to the compression **structure handler** so both code-aware
paths are consistent.

Closes #1231

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Replace the import-only probe in `code_handler._check_tree_sitter()`
with a real parse probe: construct a `Parser`, assign a Python language,
and parse `b"x = 1\n"` — if any step fails, mark unavailable
- Log a WARNING when import succeeds but parsing fails, so the downgrade
is visible
- Add `TestAvailabilityProbe` covering the simulated ABI mismatch (->
False) and healthy install (-> True) cases

> Note: an earlier revision of this PR also touched
`transforms/code_compressor.py`, but that fix landed independently via
#1299. After rebasing onto current `main`, this PR is scoped to the
remaining `code_handler.py` gap only.

## Testing

- [x] Unit tests pass (`pytest
tests/test_compression/test_code_handler.py` -> 22 passed, 8 skipped)
- [x] Linting / formatting pass (`ruff check .`, `ruff format --check
.`)
- [x] New tests added for new functionality

### Real Behavior Proof

- Setup: Windows 11 (GBK locale), Python 3.10, tree-sitter NOT installed
- Before fix: `_check_tree_sitter()` returns `True` on a
partial/ABI-mismatched install (import succeeds), then silently degrades
to the text compressor at request time with no warning
- After fix: the probe parses a trivial snippet; an ABI mismatch is
caught at probe time, `_check_tree_sitter()` returns `False`, and a
WARNING is logged.
`TestAvailabilityProbe::test_abi_mismatch_returns_false` reproduces this
with a fake Parser whose `language` setter raises.

Signed-off-by: RTCartist <wangshengb@buaa.edu.cn>
2026-07-09 14:06:29 -05:00
Tejas Chopra 840871cb96 fix(compression): repair entropy preservation + JSON-safe truncation fallback (#1536)
## Description

Reported by [@JoaoMarcos44](https://github.com/JoaoMarcos44) via an
independent security audit — thanks for the careful, well-documented
report.

Fixes two confirmed findings from a June 2026 security audit of
`headroom/compression/` (the `UniversalCompressor` utility). Both are
real defects in shipped, public, tested code; note that this module is
**not** on the proxy hot path (the proxy uses `headroom/transforms/`),
so real-world blast radius is module-local rather than proxy-wide.

- **SEC-01 (entropy bypass):** `use_entropy_preservation` was a silent
no-op. `compress()` tokenized content at character level
(`list(content)`) and fed single-char tokens to `compute_entropy_mask`,
whose `min_token_length` guard skipped every one — so high-entropy
secrets (API keys, OAuth tokens, UUIDs, hashes) were never preserved
despite the feature being enabled.
- **SEC-02 (JSON corruption):** the `_simple_compress` truncation
fallback (used when Kompress is unavailable or raises) inserted a
separator containing raw newlines. When that fallback ran on a span
inside a JSON string value it produced invalid JSON (RFC 8259 §7),
crashing downstream `json.loads()`.

The other three audited items need no code change and were verified, not
assumed: SEC-03 (surrogate DoS) is already caught by the `try/except` in
`code_handler._extract_mask` and falls back to regex — non-reproducible
even with `tree_sitter_language_pack` installed; SEC-04 (prompt
injection) is out of a compressor's scope; SEC-05 (SQLite race) is a
misread (`CompressionStore` defaults to `InMemoryBackend`; the SQLite
backend uses WAL + busy_timeout + a lock).

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Add `compute_entropy_mask_for_content()` (`masks.py`): scores
whitespace-delimited words and maps high-entropy ones back to character
positions, returning a char-aligned mask. The existing token-level
`compute_entropy_mask` is left intact.
- Introduce `SECRET_ENTROPY_MIN_LENGTH = 20` as the default word-length
floor. Normalized Shannon entropy rates short-but-diverse words (e.g.
"detailed") nearly as high as a real secret, so a length floor is the
discriminator; 20 matches the entropy-detection floor used by secret
scanners (trufflehog, detect-secrets) and prevents over-preserving prose
(which would otherwise block legitimate compression).
- Wire the content-level entropy pass into
`UniversalCompressor.compress()` (scores `content`, not the char-level
`tokens`).
- Replace the `_simple_compress` separator `"\n...[compressed]...\n"`
with the control-char-free `" ...[compressed]... "`.
- Add regression tests at the mask level and end-to-end.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/compression/
All checks passed!

$ mypy headroom/compression/masks.py headroom/compression/universal.py
Success: no issues found in 2 source files

$ pytest tests/test_compression/test_masks.py tests/test_compression/test_universal.py \
         tests/test_compression/test_json_handler.py tests/test_compression/test_code_handler.py -q
======================= 111 passed, 2 warnings in 10.76s =======================
```

## Real Behavior Proof

- Environment: macOS, Python 3.12 in repo `.venv`;
`tree_sitter_language_pack` and Kompress present.
- Exact command / steps: reproduced each finding by calling
`UniversalCompressor.compress()` directly before/after the fix — SEC-01:
`compute_entropy_mask(list("k="+secret))` preserved 0 of N tokens
(inert); after fix `compute_entropy_mask_for_content` preserves the
secret's char range and the end-to-end test shows a 43-char secret
dropped with preservation off / kept with it on. SEC-02:
`compress(json.dumps({...long value...}), content_type=JSON)` with
`use_kompress=False` raised `JSONDecodeError` before the fix and
round-trips through `json.loads()` after.
- Observed result: SEC-01 entropy preservation now functions; SEC-02
output is valid JSON on both the Kompress and fallback paths; the
previously-failing `test_compression_reduces_tokens` passes again (no
over-preservation).
- Not tested: `tests/test_compression/test_evals.py` and
`test_llm_eval.py` (require external API/model access); the
proxy/transforms live path is unaffected since it does not import
`UniversalCompressor`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

CHANGELOG not updated (handled by the release tooling). The audit also
flagged SEC-03/04/05 — left unchanged by design, with verification
rationale in the Description.
2026-06-28 10:39:02 -07:00
Veesh GoldmanandClaude Opus 4.8 f39858c233 feat(code): add Perl support to code-aware compressor (#1125)
## Description

Adds Perl as a supported language for `CodeAwareCompressor` /
`CodeStructureHandler`. Function bodies are compressed while
`use`/`require` imports, `sub`/`method` signatures, and
`package`/`class`/`role` declarations are preserved — bringing Perl up
to parity with the other Tier-2 languages.

No new dependencies: the Perl grammar already ships in
`tree-sitter-language-pack` (already a Headroom dependency), so this is
pure configuration.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `code_handler.py`: Perl entries in the four per-language tables —
`_STRUCTURAL_NODE_TYPES`, `_SIGNATURE_PATTERNS` (regex fallback),
`_LANGUAGE_MARKERS` (detection), `_IMPORT_PATTERNS`. The existing
`_CONTAINER_BODY_TYPES` already covers Perl's `block` body node, so no
change was needed there.
- `code_compressor.py`: `CodeLanguage.PERL` enum value, a data-driven
`LangConfig`, a `_LANGUAGE_PREFILTER` entry, and the supported-language
string in the parser error message.
- Node-type names (`subroutine_declaration_statement`,
`package_statement`, `signature`, `block`, …) are from the
`tree-sitter-perl/tree-sitter-perl` grammar (MIT).
- Tests: 1 detection test + 2 regex-path signature/import-preservation
tests.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/test_code_handler.py -q
collected 28 items
tests/test_compression/test_code_handler.py ............................ [100%]
======================== 28 passed, 1 warning in 2.53s =========================

$ ruff check headroom/compression/handlers/code_handler.py headroom/transforms/code_compressor.py tests/test_compression/test_code_handler.py
All checks passed!

$ mypy headroom/compression/handlers/code_handler.py headroom/transforms/code_compressor.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Python 3.12, `pip install -e ".[code]"`
(tree-sitter-language-pack installed, `is_tree_sitter_available() ==
True`).
- Exact command / steps: ran `CodeStructureHandler().get_mask(code,
language="perl")` on a real Perl module (package + two subs with
bodies).
- Observed result: detected as `perl`, parsed via the `tree-sitter` path
(not regex), and the preserved span was exactly the imports + package +
sub signatures, with both sub bodies marked compressible:

```text
tree-sitter available: True
parser: tree-sitter | detected: perl
--- PRESERVED (signatures/imports/structure) ---
use strict;use warnings;package Greeter;sub new sub greet
```

- Not tested: the full proxy/MCP server end-to-end path (out of scope —
this PR only touches the code compressor's language tables).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Docs/CHANGELOG left unchecked — happy to add a Perl line to either if
you'd like; I wasn't sure of your preferred location.
- *Disclosure: I maintain the upstream `tree-sitter-perl` grammar this
relies on. It's already a transitive dependency of Headroom via
`tree-sitter-language-pack` — this PR only adds config to use it, with
no dependency changes.*

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 18:47:41 -05:00
6cdb846200 fix(compression): use thread-local tree-sitter parsers in code handler (#893)
## Description

`CodeStructureHandler` cached tree-sitter parsers in a process-global
dict; the lock only guarded creation, while `parse()` ran unlocked on
any thread. tree-sitter `Parser` objects are pyo3 `unsendable` — using
one from a non-creator thread panics. The proxy invokes handlers from
executor pool threads, so a shared parser is an eventual crash. Same
class already fixed in `transforms/code_compressor.py` (#604). Stacked
on #892.

Closes # <!-- compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/code_handler.py`: one parser per
(thread, language) via `threading.local()`, porting the pattern from
`transforms/code_compressor.py`.
- `tests/test_compression/test_code_handler.py`: regression test parsing
from a 4-worker thread pool, asserting every call stays on the
tree-sitter path.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/ -q
94 passed, 8 skipped
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, tree-sitter-language-pack
1.8.1, branch `fix/code-threadlocal-parsers` (stacked on #892).
- Exact command / steps: `pytest tests/test_compression/ -q`.
- Observed result: 16 parses across a 4-worker pool all stay on the
tree-sitter path with no pyo3 panic; previously a shared parser would be
touched cross-thread.
- Not tested: Reproducing the original panic under production
concurrency (covered structurally by the thread-pool test).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library change. See Test Output.

## Additional Notes

Stacked on #892 — review the top commit until that merges. PR 5 of 7.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-06-15 16:38:08 -05:00
AshishandClaude Fable 5 615e1ed6f5 test(compression): fill code handler coverage gaps (#895)
## Description

`CodeStructureHandler` had zero dedicated tests before this series —
which is exactly why the P0 bugs in #890/#892/#893 went unnoticed. This
fills the remaining coverage gaps beyond the per-fix regression tests.
Stacked on #893.

Closes # <!-- compression-handler review -->

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `tests/test_compression/test_code_handler.py`: language detection
(python/go/rust + default fallback); regex-path signature/import
preservation across go/rust/typescript/javascript; regex confidence
value; empty/whitespace content; unknown language; mask-length
invariant.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/test_code_handler.py -q
25 passed
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, tree-sitter-language-pack
1.8.1, branch `test/code-handler-coverage` (stacked on #893).
- Exact command / steps: `pytest
tests/test_compression/test_code_handler.py -q`.
- Observed result: 25 tests pass; tree-sitter classes skip cleanly when
the pack is absent, regex-path tests always run.
- Not tested: N/A — this PR is tests only.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — tests only. See Test Output.

## Additional Notes

Stacked on #893 — review the top commit until that merges. PR 6 of 7.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 10:25:57 -05:00
AshishandClaude Fable 5 b1f700fc27 fix(compression): convert tree-sitter byte offsets to char offsets (#892)
## Description

tree-sitter reports node positions as byte offsets into the UTF-8
encoding, but `CodeStructureHandler` builds a character-indexed mask.
Any multi-byte character (accents, emoji, CJK in
docstrings/comments/strings) shifted every subsequent span, preserving
the wrong characters and leaking signature bytes into bodies. Stacked on
#890.

Closes # <!-- compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/code_handler.py`: remap spans through a
byte->char table before masking; pure-ASCII content (byte == char) skips
the conversion.
- `tests/test_compression/test_code_handler.py`: regression test with
`café münü 🎉` in a comment, asserting the following signature and body
are correctly aligned.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/ -q
93 passed, 8 skipped
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, tree-sitter-language-pack
1.8.1, branch `fix/code-byte-char-offsets` (stacked on #890).
- Exact command / steps: `pytest tests/test_compression/ -q`.
- Observed result: With 9 extra UTF-8 bytes ahead of it, a function
signature is exactly preserved and its body stays compressible; before,
the offsets were shifted.
- Not tested: End-to-end through the live proxy pipeline.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library change. See Test Output.

## Additional Notes

Stacked on #890 — review the top commit until that merges. PR 4 of 7.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 10:24:21 -05:00
AshishandClaude Fable 5 65b0e8c58d fix(compression): measure short-value threshold on payload, not token (#889)
## Description

`JSONStructureHandler._should_preserve_token` compared `len(token.text)`
— which includes both quote characters — against
`short_value_threshold`. A value of exactly threshold length was
rejected: the documented "20-char threshold" was effectively 18 chars of
payload. Stacked on #887.

Closes # <!-- compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/json_handler.py`: strip quotes once at
the top of the string-value branch and use the payload length for both
the short-value and entropy checks.
- `tests/test_compression/test_json_handler.py`: regression test for a
value of exactly `short_value_threshold` length.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/test_json_handler.py -q
33 passed
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, branch
`fix/json-quote-threshold` (stacked on #887).
- Exact command / steps: `pytest
tests/test_compression/test_json_handler.py -q`.
- Observed result: A 20-char value is preserved at a 20-char threshold;
previously it was dropped due to the +2 quote miscount.
- Not tested: End-to-end through the live proxy pipeline.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library change. See Test Output.

## Additional Notes

Stacked on #887 — review the top commit until that merges. PR 2 of 7.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 10:23:26 -05:00
AshishandClaude Fable 5 16ed73bca6 fix(compression): keep container bodies compressible in code handler (#890)
## Description

Two bugs in `CodeStructureHandler`'s tree-sitter path. (1) Container
nodes (class/impl/trait/decorated definitions) were marked structural
over their full span and `_spans_to_mask` never un-marks, so every
method body inside a class was preserved and compression silently
no-opped at confidence 0.95. (2) Discovered while testing:
`tree-sitter-language-pack >= 1.0` switched to a Rust binding (methods,
not attributes; `parse(str)`), so the handler raised `TypeError` on
every call and silently fell back to regex.

Closes # <!-- compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/code_handler.py`: containers emit a
signature-only span (start to body start); recursion gives nested
functions their own signature/body split; decorated definitions emit no
whole-node span.
- `headroom/compression/handlers/code_handler.py`: small compat shim
supporting both the classic attribute API and the new Rust-binding
method API.
- `tests/test_compression/test_code_handler.py`: new file (the handler
had zero dedicated tests) covering class/decorated/impl body
compressibility, regex fallback, and a preservation-ratio bound.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/ -q
92 passed, 8 skipped
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, tree-sitter-language-pack 1.8.1
installed, branch `fix/code-container-bodies`.
- Exact command / steps: `pytest tests/test_compression/ -q`.
- Observed result: Class method bodies are now compressible
(preservation ratio drops from ~1.0 to roughly the signature fraction);
the tree-sitter path runs instead of falling back to regex.
- Not tested: End-to-end through the live proxy pipeline.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library change. See Test Output.

## Additional Notes

PR 3 of 7; branched fresh from main (independent of #887/#889).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 21:31:32 -07:00
AshishandClaude Fable 5 d6f0f0f642 fix(compression): correct JSON array item counting and entropy gate (#887)
## Description

Two bugs in `JSONStructureHandler` that jointly defeated the "keep first
N array items fully" design. (1) Every comma under `array_depth > 0` was
counted as an array item separator — including commas *between keys
inside objects* — so for arrays of objects the first record's own keys
exhausted `max_array_items_full` and dropped values belonging to item 0.
(2) Fixing that unmasked a second bug: self-normalized Shannon entropy
scores English prose at 0.90+, above the 0.85 "identifier" threshold, so
every long description was preserved as a fake high-entropy identifier.

Closes # <!-- found during a compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/json_handler.py`: replace depth-keyed
comma counting with a container stack so only commas whose immediate
enclosing container is an array advance that array's item index.
- `headroom/compression/handlers/json_handler.py`: gate the entropy
preservation check on a no-spaces identifier signal, so UUIDs/hashes
still pass but prose compresses.
- `tests/test_compression/test_json_handler.py`: regression tests for
object-comma counting, items past the threshold, and prose-vs-identifier
entropy.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/test_json_handler.py -q
32 passed
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, branch
`fix/json-array-item-count`.
- Exact command / steps: `pytest
tests/test_compression/test_json_handler.py -q` plus an empirical mask
dump on `[{"a":1,"b":2,...}]`.
- Observed result: Values inside array item 0 are now preserved; long
prose values compress while UUIDs are retained (prose scored
0.906-0.929, UUID 0.956 — the threshold alone could not separate them).
- Not tested: End-to-end through the live proxy pipeline (the handler is
not yet wired into the proxy hot path).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library/compression change with no UI. See Test Output.

## Additional Notes

First of a 7-PR compression-handler review series.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 21:24:12 -07:00
chopratejas 3290a3d582 Remove LLMLingua: Kompress is the sole text compressor
LLMLingua was the original ML text compressor (BERT-based). Kompress
(ModernBERT, trained on 330K structured tool outputs) replaced it with
better compression quality and simpler architecture.

Removed across 35 files:
- Deleted headroom/transforms/llmlingua_compressor.py
- Deleted tests/test_transforms/test_llmlingua_compressor.py
- Deleted tests/test_proxy_llmlingua.py
- Removed all enable_llmlingua config, _get_llmlingua methods,
  LLMLingua fallback paths, LLMLINGUA strategy enum values
- Removed CLI flags, model configs, compression handler references
- Simplified ContentRouter: Kompress is primary and only text compressor
2026-03-26 11:11:00 -07:00
chopratejas 0adc39ab7a Fix CI: guard starlette imports, asyncio.run(), deprecate datetime.utcnow()
- Guard starlette imports in test_compress_api.py (skip ASGI tests without proxy deps)
- Replace asyncio.get_event_loop().run_until_complete() with asyncio.run() (Python 3.13)
- Replace datetime.utcnow() with datetime.now(timezone.utc).replace(tzinfo=None) everywhere
2026-02-19 11:03:24 -08:00
chopratejas 64a747d66e Fix flaky JavaScript signature preservation test threshold
Lower threshold from 70% to 60% - methods inside class bodies
may be compressed, which is expected behavior.
2026-01-15 15:38:25 -08:00
chopratejas 31aa72c885 Add universal compression module with ML-based content detection
- Add headroom.compression module with UniversalCompressor
- ML-based content detection using Magika (JSON, code, logs, text)
- Structure-preserving compression via handler protocol
- JSON handler: preserves keys, brackets, high-entropy values (UUIDs)
- Code handler: preserves imports, signatures, types (tree-sitter AST)
- Entropy-based preservation for identifiers and hashes
- CCR integration for reversible compression
- Comprehensive test suite with LLM eval tests
- Add docs/compression.md with full API documentation
2026-01-15 15:26:14 -08:00