Files
headroom/wiki/transforms.md
Yevhen KovalandJerrettDavis 76ec363a60 docs(readme): list C# and PHP as AST-compressed, drop quarantined Perl (#3225)
## Description

The README "What's inside" bullet for CodeCompressor (and the two
matching lists in `wiki/transforms.md` plus the module docstring of
`headroom/transforms/code_compressor.py`) still read `Python, JS/TS, Go,
Rust, Java, C/C++, Perl`. That list predates two changes on `main`:

- **C#** (#1664 → #1926) and **PHP** (#201 → #2423) are first-class
`LangConfig` entries in `CodeAwareCompressor` (`code_compressor.py:440`
and `:469`) but were never added to the lists.
- **Perl** has been quarantined from code-aware compression since #2204
(refs #2185): `_get_parser()` refuses the `perl` grammar
(`code_compressor.py:106`, `:136`) and `compress()` short-circuits Perl
to the Kompress fallback / passthrough (`code_compressor.py:1200`), so
"AST-aware for … Perl" is no longer true.

All four lists now match the canonical docs site
(`docs/content/docs/code-compression.mdx`, which already says `Go, Rust,
Java, C, C++, C#, PHP` for Tier 2 and does not list Perl).

Closes #201 — PHP support itself landed in #2423 (the PR description
mentioned the issue without a closing keyword, so it stayed open); this
PR makes it visible in the README.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `README.md` — CodeCompressor bullet: add C# and PHP, drop Perl.
- `wiki/transforms.md` — "Multi-language support" bullet and the Tier 2
row of the "Supported Languages" table: add C# and PHP.
- `headroom/transforms/code_compressor.py` — module docstring,
"Supported Languages (Tier 2)": add C# and PHP. Docstring only; no
runtime code changes.

## Testing

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ git rev-parse --short HEAD
5a323bd8

$ grep -n "CodeCompressor\*\* — AST-aware" README.md
368:- **CodeCompressor** — AST-aware for Python, JS/TS, Go, Rust, Java, C/C++, C#, PHP.

$ grep -n "Multi-language support\|^| 2 |" wiki/transforms.md
158:- **Multi-language support** — Python, JavaScript, TypeScript, Go, Rust, Java, C, C++, C#, PHP
231:| 2 | Go, Rust, Java, C, C++, C#, PHP | Function body compression |
354:| 2 | SmartCrusher | Compress JSON tool outputs |

$ sed -n "18,19p" headroom/transforms/code_compressor.py
Supported Languages (Tier 2):
- Go, Rust, Java, C, C++, C#, PHP

$ grep -n "Perl" README.md wiki/transforms.md || echo PERL_ABSENT_FROM_LISTS
PERL_ABSENT_FROM_LISTS

$ grep -n "_UNSAFE_TREE_SITTER_LANGUAGES" headroom/transforms/code_compressor.py
106:_UNSAFE_TREE_SITTER_LANGUAGES: frozenset[str] = frozenset({"perl"})
136:    if language in _UNSAFE_TREE_SITTER_LANGUAGES:

$ grep -n "CSHARP = \|PHP = \|CodeLanguage.CSHARP: LangConfig\|CodeLanguage.PHP: LangConfig" headroom/transforms/code_compressor.py
225:    CSHARP = "csharp"
226:    PHP = "php"
440:    CodeLanguage.CSHARP: LangConfig(
469:    CodeLanguage.PHP: LangConfig(

$ python3 -m py_compile headroom/transforms/code_compressor.py && echo PY_COMPILE_OK
PY_COMPILE_OK

$ git log -1 --format=%B | npx -y -p @commitlint/cli -p @commitlint/config-conventional commitlint --config .commitlintrc.json
(exit 0)

$ git diff --stat main..HEAD
 README.md                              | 2 +-
 headroom/transforms/code_compressor.py | 2 +-
 wiki/transforms.md                     | 4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)
```

## Real Behavior Proof

- Environment: Linux 6.18 (CachyOS), system Python 3.14.7, checkout of
`headroomlabs-ai/headroom` `main` at 34a55175 (v0.36.5), branch
`docs/readme-code-languages`. No `uv`/dev venv on this machine.
- Exact command / steps: the `grep` / `sed` / `py_compile` /
`commitlint` / `git diff --stat` commands pasted above, run after the
patch.
- Observed result: all four language lists name C# and PHP; Perl is
absent from README and wiki; the quarantine guard and the C#/PHP
`LangConfig` entries exist at the quoted lines on current `main`; the
module still byte-compiles; the commit message passes the repo's
commitlint config; diff is 4 lines across 3 files, docs/docstring only.
- Not tested: `uv run pytest` / `ruff` / `mypy` / `make ci-precheck` —
not run locally (no dev environment here); no runtime path is touched,
so the compressor's behavior is unchanged by construction. The docs site
(`docs/`) is untouched because it was already correct.

## Runtime Rollout Safety

- Rollout-managed feature(s): N/A (documentation only)
- Minimum rollout channel: N/A
- Stable/default behavior changed: No
- Kill switch / disable path: N/A
- Unsafe override required: No
- Qualification impact: None
- Rollback path: revert the commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- N/A checklist items: no code comments / tests to add for a docs-only
change; the test suite was not run locally (see "Not tested").
- `wiki/transforms.md`: per #2784 the wiki is unpublished markdown that
"needs no syncing". I still aligned its two lists because they were
factually wrong in the same way and the file was deliberately kept as a
migration backlog — happy to drop that hunk if you'd rather keep the PR
to README + docstring.
- Perl removal: if the quarantine is ever lifted (the upstream
tree-sitter-perl scanner wedge from #2185), Perl can be re-added to
these lists alongside the code change.
- Already correct and untouched: the `Supported: …` error string in
`_get_parser()` (`code_compressor.py:170`) and the docs site table.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-25 10:37:14 -05:00

373 lines
11 KiB
Markdown

# Transform Reference
Headroom provides several transforms that work together to optimize LLM context.
## SmartCrusher
Statistical compression for JSON tool outputs.
### How It Works
SmartCrusher analyzes JSON arrays and selectively keeps important items:
1. **First/Last items** - Context for pagination and recency
2. **Error items** - 100% preservation of error states
3. **Anomalies** - Statistical outliers (> 2 std dev from mean)
4. **Relevant items** - Matches to user's query via BM25/embeddings
5. **Change points** - Significant transitions in data
### Configuration
```python
from headroom import SmartCrusherConfig
config = SmartCrusherConfig(
min_tokens_to_crush=200, # Only compress if > 200 tokens
max_items_after_crush=50, # Keep at most 50 items
keep_first=3, # Always keep first 3 items
keep_last=2, # Always keep last 2 items
relevance_threshold=0.3, # Keep items with relevance > 0.3
anomaly_std_threshold=2.0, # Keep items > 2 std dev from mean
preserve_errors=True, # Always keep error items
)
```
### Example
```python
from headroom import SmartCrusher
crusher = SmartCrusher(config)
# Before: 1000 search results (45,000 tokens)
tool_output = {"results": [...1000 items...]}
# After: ~50 important items (4,500 tokens) - 90% reduction
compressed = crusher.crush(tool_output, query="user's question")
```
### What Gets Preserved
| Category | Preserved | Why |
|----------|-----------|-----|
| Errors | 100% | Critical for debugging |
| First N | 100% | Context/pagination |
| Last N | 100% | Recency |
| Anomalies | All | Unusual values matter |
| Relevant | Top K | Match user's query |
| Others | Sampled | Statistical representation |
---
## CacheAligner
Prefix stabilization for improved cache hit rates.
### The Problem
LLM providers cache request prefixes. But dynamic content breaks caching:
```
"You are helpful. Today is January 7, 2025." # Changes daily = no cache
```
### The Solution
CacheAligner extracts dynamic content to stabilize the prefix:
```python
from headroom import CacheAligner
aligner = CacheAligner()
result = aligner.align(messages)
# Static prefix (cacheable):
# "You are helpful."
# Dynamic content moved to end:
# [Current date context]
```
### Configuration
```python
from headroom import CacheAlignerConfig
config = CacheAlignerConfig(
extract_dates=True, # Move dates to dynamic section
normalize_whitespace=True, # Consistent spacing
stable_prefix_min_tokens=100, # Min prefix size for alignment
)
```
### Cache Hit Improvement
| Scenario | Before | After |
|----------|--------|-------|
| Daily date in prompt | 0% hits | ~95% hits |
| Dynamic user context | ~10% hits | ~80% hits |
| Consistent prompts | ~90% hits | ~95% hits |
---
## Context management
Context management is handled automatically inside the pipeline
(live-zone-only compression). Headroom **never** drops messages from the
conversation history and does not do position-based or score-based context
management. It compresses only the newest content blocks (the latest user
message and the latest tool result / tool output), type-aware and reversible
via CCR. The cache hot zone — system prompt, tools, and older turns — is never
mutated, which preserves provider prompt caching.
> The earlier position-based `RollingWindow` and score-based
> `IntelligentContextManager` transforms have been removed and are no longer
> part of Headroom.
---
## LLMLinguaCompressor — RETIRED
The earlier LLMLingua-2 integration (`LLMLinguaCompressor`,
`LLMLinguaConfig`, `is_llmlingua_model_loaded`, `unload_llmlingua_model`,
the `headroom-ai[llmlingua]` extra, and the `--llmlingua` proxy flag)
was retired in 0.9.x and replaced by **Kompress** (ModernBERT).
`pip install 'headroom-ai[llmlingua]'` no longer resolves; use the
`[ml]` extra instead. The Kompress transform shipped with the proxy
runs as Transform 4 in the live-zone pipeline (see
[ARCHITECTURE.md](ARCHITECTURE.md)).
---
## CodeAwareCompressor (Optional)
AST-based compression for source code using tree-sitter.
### When to Use
| Transform | Best For | Speed | Compression |
|-----------|----------|-------|-------------|
| SmartCrusher | JSON arrays | ~1ms | 70-90% |
| **CodeAwareCompressor** | Source code | ~10-50ms | 40-70% |
| Kompress (ML) | Any text | 50-200ms | 80-95% |
### Key Benefits
- **Syntax validity guaranteed** — Output always parses correctly
- **Preserves critical structure** — Imports, signatures, types, error handlers
- **Multi-language support** — Python, JavaScript, TypeScript, Go, Rust, Java, C, C++, C#, PHP
- **Lightweight** — ~50MB vs ~1GB for the ML compressor
### Installation
```bash
pip install "headroom-ai[code]" # Adds tree-sitter-language-pack
```
### Configuration
```python
from headroom.transforms import CodeAwareCompressor, CodeCompressorConfig, DocstringMode
config = CodeCompressorConfig(
preserve_imports=True, # Always keep imports
preserve_signatures=True, # Always keep function signatures
preserve_type_annotations=True, # Keep type hints
preserve_error_handlers=True, # Keep try/except blocks
preserve_decorators=True, # Keep decorators
docstring_mode=DocstringMode.FIRST_LINE, # FULL, FIRST_LINE, REMOVE
target_compression_rate=0.2, # Keep 20% of tokens
max_body_lines=5, # Lines to keep per function body
min_tokens_for_compression=100, # Skip small content
language_hint=None, # Auto-detect if None
)
compressor = CodeAwareCompressor(config)
```
### Example
```python
from headroom.transforms import CodeAwareCompressor
compressor = CodeAwareCompressor()
code = '''
import os
from typing import List
def process_items(items: List[str]) -> List[str]:
"""Process a list of items."""
results = []
for item in items:
if not item:
continue
processed = item.strip().lower()
results.append(processed)
return results
'''
result = compressor.compress(code, language="python")
print(result.compressed)
# import os
# from typing import List
#
# def process_items(items: List[str]) -> List[str]:
# """Process a list of items."""
# results = []
# for item in items:
# # ... (5 lines compressed)
# pass
print(f"Compression: {result.compression_ratio:.0%}") # ~55%
print(f"Syntax valid: {result.syntax_valid}") # True
```
### Supported Languages
| Tier | Languages | Support Level |
|------|-----------|---------------|
| 1 | Python, JavaScript, TypeScript | Full AST analysis |
| 2 | Go, Rust, Java, C, C++, C#, PHP | Function body compression |
### Memory Management
```python
from headroom.transforms import is_tree_sitter_available, unload_tree_sitter
# Check if tree-sitter is installed
print(is_tree_sitter_available()) # True/False
# Free memory when done (parsers are lazy-loaded)
unload_tree_sitter()
```
---
## ContentRouter
Intelligent compression orchestrator that routes content to the optimal compressor.
### How It Works
ContentRouter analyzes content and selects the best compression strategy:
1. **Detect content type** — JSON, code, logs, search results, plain text
2. **Consider source hints** — File paths, tool names for high-confidence routing
3. **Route to compressor** — SmartCrusher, CodeAwareCompressor, SearchCompressor, etc.
4. **Log decisions** — Transparent routing for debugging
### Configuration
```python
from headroom.transforms import ContentRouter, ContentRouterConfig, CompressionStrategy
config = ContentRouterConfig(
min_section_tokens=100, # Minimum tokens to compress
enable_code_aware=True, # Use CodeAwareCompressor for code
enable_search_compression=True, # Use SearchCompressor for grep output
enable_log_compression=True, # Use LogCompressor for logs
default_strategy=CompressionStrategy.TEXT, # Fallback strategy
)
router = ContentRouter(config)
```
### Example
```python
from headroom.transforms import ContentRouter
router = ContentRouter()
# Router auto-detects content type and routes to optimal compressor
result = router.compress(content)
print(result.strategy_used) # CompressionStrategy.CODE_AWARE, SMART_CRUSHER, etc.
print(result.routing_log) # List of routing decisions
```
### Compression Strategies
| Strategy | Used For | Compressor |
|----------|----------|------------|
| CODE_AWARE | Source code | CodeAwareCompressor |
| SMART_CRUSHER | JSON arrays | SmartCrusher |
| SEARCH | Grep/find output | SearchCompressor |
| LOG | Log files | LogCompressor |
| TEXT | Plain text | TextCompressor |
| PASSTHROUGH | Small content | None |
(The earlier `LLMLINGUA` strategy was retired with the LLMLingua integration; ML compression is now provided by Kompress.)
### Content Detection
The router automatically detects content types by analyzing the content itself:
- **Source code**: Detected by syntax patterns, indentation, keywords
- **JSON arrays**: Detected by JSON structure with array elements
- **Search results**: Detected by `file:line:` patterns
- **Log output**: Detected by timestamp and log level patterns
- **Plain text**: Fallback for prose content
No manual hints required - the router inspects content directly.
### TOIN Integration
ContentRouter records all compressions to TOIN (Tool Output Intelligence Network) for cross-user learning:
- **All strategies tracked**: Code, search, logs, text, and ML compressions are recorded
- **Retrieval feedback**: When users retrieve original content via CCR, TOIN learns which compressions need expansion
- **Pattern learning**: TOIN builds signatures for each content type to improve future compressions
This enables the feedback loop where compression decisions improve based on actual user behavior across all content types, not just JSON arrays.
---
## TransformPipeline
Combine transforms for optimal results.
```python
from headroom import TransformPipeline, SmartCrusher, CacheAligner
pipeline = TransformPipeline(
[
SmartCrusher(), # First: compress tool outputs
CacheAligner(), # Then: stabilize prefix
]
)
result = pipeline.transform(messages)
print(f"Saved {result.tokens_saved} tokens")
```
### With ML compression (Optional, Kompress)
The earlier hand-assembled `TransformPipeline([..., LLMLinguaCompressor(), ...])` recipe is no longer supported. ML compression now ships as part of the live-zone pipeline when the `[ml]` extra is installed; see [ARCHITECTURE.md](ARCHITECTURE.md) for the current placement.
### Recommended Order
| Order | Transform | Purpose |
|-------|-----------|---------|
| 1 | CacheAligner | Stabilize prefix for caching |
| 2 | SmartCrusher | Compress JSON tool outputs |
| 3 | Kompress (ML) | ML compression on remaining text (optional, `[ml]` extra) |
**Why this order?**
- CacheAligner first to maximize prefix stability
- SmartCrusher handles JSON arrays efficiently
- Kompress compresses remaining long text
---
## Safety Guarantees
All transforms follow strict safety rules:
1. **Never remove human content** - User/assistant text is sacred
2. **Never break tool ordering** - Calls and results stay paired
3. **Parse failures are no-ops** - Malformed content passes through
4. **Preserves recency** - Last N turns always kept
5. **100% error preservation** - Error items never dropped