mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description The README "What's inside" bullet for CodeCompressor (and the two matching lists in `wiki/transforms.md` plus the module docstring of `headroom/transforms/code_compressor.py`) still read `Python, JS/TS, Go, Rust, Java, C/C++, Perl`. That list predates two changes on `main`: - **C#** (#1664 → #1926) and **PHP** (#201 → #2423) are first-class `LangConfig` entries in `CodeAwareCompressor` (`code_compressor.py:440` and `:469`) but were never added to the lists. - **Perl** has been quarantined from code-aware compression since #2204 (refs #2185): `_get_parser()` refuses the `perl` grammar (`code_compressor.py:106`, `:136`) and `compress()` short-circuits Perl to the Kompress fallback / passthrough (`code_compressor.py:1200`), so "AST-aware for … Perl" is no longer true. All four lists now match the canonical docs site (`docs/content/docs/code-compression.mdx`, which already says `Go, Rust, Java, C, C++, C#, PHP` for Tier 2 and does not list Perl). Closes #201 — PHP support itself landed in #2423 (the PR description mentioned the issue without a closing keyword, so it stayed open); this PR makes it visible in the README. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `README.md` — CodeCompressor bullet: add C# and PHP, drop Perl. - `wiki/transforms.md` — "Multi-language support" bullet and the Tier 2 row of the "Supported Languages" table: add C# and PHP. - `headroom/transforms/code_compressor.py` — module docstring, "Supported Languages (Tier 2)": add C# and PHP. Docstring only; no runtime code changes. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ git rev-parse --short HEAD5a323bd8$ grep -n "CodeCompressor\*\* — AST-aware" README.md 368:- **CodeCompressor** — AST-aware for Python, JS/TS, Go, Rust, Java, C/C++, C#, PHP. $ grep -n "Multi-language support\|^| 2 |" wiki/transforms.md 158:- **Multi-language support** — Python, JavaScript, TypeScript, Go, Rust, Java, C, C++, C#, PHP 231:| 2 | Go, Rust, Java, C, C++, C#, PHP | Function body compression | 354:| 2 | SmartCrusher | Compress JSON tool outputs | $ sed -n "18,19p" headroom/transforms/code_compressor.py Supported Languages (Tier 2): - Go, Rust, Java, C, C++, C#, PHP $ grep -n "Perl" README.md wiki/transforms.md || echo PERL_ABSENT_FROM_LISTS PERL_ABSENT_FROM_LISTS $ grep -n "_UNSAFE_TREE_SITTER_LANGUAGES" headroom/transforms/code_compressor.py 106:_UNSAFE_TREE_SITTER_LANGUAGES: frozenset[str] = frozenset({"perl"}) 136: if language in _UNSAFE_TREE_SITTER_LANGUAGES: $ grep -n "CSHARP = \|PHP = \|CodeLanguage.CSHARP: LangConfig\|CodeLanguage.PHP: LangConfig" headroom/transforms/code_compressor.py 225: CSHARP = "csharp" 226: PHP = "php" 440: CodeLanguage.CSHARP: LangConfig( 469: CodeLanguage.PHP: LangConfig( $ python3 -m py_compile headroom/transforms/code_compressor.py && echo PY_COMPILE_OK PY_COMPILE_OK $ git log -1 --format=%B | npx -y -p @commitlint/cli -p @commitlint/config-conventional commitlint --config .commitlintrc.json (exit 0) $ git diff --stat main..HEAD README.md | 2 +- headroom/transforms/code_compressor.py | 2 +- wiki/transforms.md | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) ``` ## Real Behavior Proof - Environment: Linux 6.18 (CachyOS), system Python 3.14.7, checkout of `headroomlabs-ai/headroom` `main` at34a55175(v0.36.5), branch `docs/readme-code-languages`. No `uv`/dev venv on this machine. - Exact command / steps: the `grep` / `sed` / `py_compile` / `commitlint` / `git diff --stat` commands pasted above, run after the patch. - Observed result: all four language lists name C# and PHP; Perl is absent from README and wiki; the quarantine guard and the C#/PHP `LangConfig` entries exist at the quoted lines on current `main`; the module still byte-compiles; the commit message passes the repo's commitlint config; diff is 4 lines across 3 files, docs/docstring only. - Not tested: `uv run pytest` / `ruff` / `mypy` / `make ci-precheck` — not run locally (no dev environment here); no runtime path is touched, so the compressor's behavior is unchanged by construction. The docs site (`docs/`) is untouched because it was already correct. ## Runtime Rollout Safety - Rollout-managed feature(s): N/A (documentation only) - Minimum rollout channel: N/A - Stable/default behavior changed: No - Kill switch / disable path: N/A - Unsafe override required: No - Qualification impact: None - Rollback path: revert the commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - N/A checklist items: no code comments / tests to add for a docs-only change; the test suite was not run locally (see "Not tested"). - `wiki/transforms.md`: per #2784 the wiki is unpublished markdown that "needs no syncing". I still aligned its two lists because they were factually wrong in the same way and the file was deliberately kept as a migration backlog — happy to drop that hunk if you'd rather keep the PR to README + docstring. - Perl removal: if the quarantine is ever lifted (the upstream tree-sitter-perl scanner wedge from #2185), Perl can be re-added to these lists alongside the code change. - Already correct and untouched: the `Supported: …` error string in `_get_parser()` (`code_compressor.py:170`) and the docs site table. Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
373 lines
11 KiB
Markdown
373 lines
11 KiB
Markdown
# Transform Reference
|
|
|
|
Headroom provides several transforms that work together to optimize LLM context.
|
|
|
|
## SmartCrusher
|
|
|
|
Statistical compression for JSON tool outputs.
|
|
|
|
### How It Works
|
|
|
|
SmartCrusher analyzes JSON arrays and selectively keeps important items:
|
|
|
|
1. **First/Last items** - Context for pagination and recency
|
|
2. **Error items** - 100% preservation of error states
|
|
3. **Anomalies** - Statistical outliers (> 2 std dev from mean)
|
|
4. **Relevant items** - Matches to user's query via BM25/embeddings
|
|
5. **Change points** - Significant transitions in data
|
|
|
|
### Configuration
|
|
|
|
```python
|
|
from headroom import SmartCrusherConfig
|
|
|
|
config = SmartCrusherConfig(
|
|
min_tokens_to_crush=200, # Only compress if > 200 tokens
|
|
max_items_after_crush=50, # Keep at most 50 items
|
|
keep_first=3, # Always keep first 3 items
|
|
keep_last=2, # Always keep last 2 items
|
|
relevance_threshold=0.3, # Keep items with relevance > 0.3
|
|
anomaly_std_threshold=2.0, # Keep items > 2 std dev from mean
|
|
preserve_errors=True, # Always keep error items
|
|
)
|
|
```
|
|
|
|
### Example
|
|
|
|
```python
|
|
from headroom import SmartCrusher
|
|
|
|
crusher = SmartCrusher(config)
|
|
|
|
# Before: 1000 search results (45,000 tokens)
|
|
tool_output = {"results": [...1000 items...]}
|
|
|
|
# After: ~50 important items (4,500 tokens) - 90% reduction
|
|
compressed = crusher.crush(tool_output, query="user's question")
|
|
```
|
|
|
|
### What Gets Preserved
|
|
|
|
| Category | Preserved | Why |
|
|
|----------|-----------|-----|
|
|
| Errors | 100% | Critical for debugging |
|
|
| First N | 100% | Context/pagination |
|
|
| Last N | 100% | Recency |
|
|
| Anomalies | All | Unusual values matter |
|
|
| Relevant | Top K | Match user's query |
|
|
| Others | Sampled | Statistical representation |
|
|
|
|
---
|
|
|
|
## CacheAligner
|
|
|
|
Prefix stabilization for improved cache hit rates.
|
|
|
|
### The Problem
|
|
|
|
LLM providers cache request prefixes. But dynamic content breaks caching:
|
|
|
|
```
|
|
"You are helpful. Today is January 7, 2025." # Changes daily = no cache
|
|
```
|
|
|
|
### The Solution
|
|
|
|
CacheAligner extracts dynamic content to stabilize the prefix:
|
|
|
|
```python
|
|
from headroom import CacheAligner
|
|
|
|
aligner = CacheAligner()
|
|
result = aligner.align(messages)
|
|
|
|
# Static prefix (cacheable):
|
|
# "You are helpful."
|
|
|
|
# Dynamic content moved to end:
|
|
# [Current date context]
|
|
```
|
|
|
|
### Configuration
|
|
|
|
```python
|
|
from headroom import CacheAlignerConfig
|
|
|
|
config = CacheAlignerConfig(
|
|
extract_dates=True, # Move dates to dynamic section
|
|
normalize_whitespace=True, # Consistent spacing
|
|
stable_prefix_min_tokens=100, # Min prefix size for alignment
|
|
)
|
|
```
|
|
|
|
### Cache Hit Improvement
|
|
|
|
| Scenario | Before | After |
|
|
|----------|--------|-------|
|
|
| Daily date in prompt | 0% hits | ~95% hits |
|
|
| Dynamic user context | ~10% hits | ~80% hits |
|
|
| Consistent prompts | ~90% hits | ~95% hits |
|
|
|
|
---
|
|
|
|
## Context management
|
|
|
|
Context management is handled automatically inside the pipeline
|
|
(live-zone-only compression). Headroom **never** drops messages from the
|
|
conversation history and does not do position-based or score-based context
|
|
management. It compresses only the newest content blocks (the latest user
|
|
message and the latest tool result / tool output), type-aware and reversible
|
|
via CCR. The cache hot zone — system prompt, tools, and older turns — is never
|
|
mutated, which preserves provider prompt caching.
|
|
|
|
> The earlier position-based `RollingWindow` and score-based
|
|
> `IntelligentContextManager` transforms have been removed and are no longer
|
|
> part of Headroom.
|
|
|
|
---
|
|
|
|
## LLMLinguaCompressor — RETIRED
|
|
|
|
The earlier LLMLingua-2 integration (`LLMLinguaCompressor`,
|
|
`LLMLinguaConfig`, `is_llmlingua_model_loaded`, `unload_llmlingua_model`,
|
|
the `headroom-ai[llmlingua]` extra, and the `--llmlingua` proxy flag)
|
|
was retired in 0.9.x and replaced by **Kompress** (ModernBERT).
|
|
`pip install 'headroom-ai[llmlingua]'` no longer resolves; use the
|
|
`[ml]` extra instead. The Kompress transform shipped with the proxy
|
|
runs as Transform 4 in the live-zone pipeline (see
|
|
[ARCHITECTURE.md](ARCHITECTURE.md)).
|
|
|
|
---
|
|
|
|
## CodeAwareCompressor (Optional)
|
|
|
|
AST-based compression for source code using tree-sitter.
|
|
|
|
### When to Use
|
|
|
|
| Transform | Best For | Speed | Compression |
|
|
|-----------|----------|-------|-------------|
|
|
| SmartCrusher | JSON arrays | ~1ms | 70-90% |
|
|
| **CodeAwareCompressor** | Source code | ~10-50ms | 40-70% |
|
|
| Kompress (ML) | Any text | 50-200ms | 80-95% |
|
|
|
|
### Key Benefits
|
|
|
|
- **Syntax validity guaranteed** — Output always parses correctly
|
|
- **Preserves critical structure** — Imports, signatures, types, error handlers
|
|
- **Multi-language support** — Python, JavaScript, TypeScript, Go, Rust, Java, C, C++, C#, PHP
|
|
- **Lightweight** — ~50MB vs ~1GB for the ML compressor
|
|
|
|
### Installation
|
|
|
|
```bash
|
|
pip install "headroom-ai[code]" # Adds tree-sitter-language-pack
|
|
```
|
|
|
|
### Configuration
|
|
|
|
```python
|
|
from headroom.transforms import CodeAwareCompressor, CodeCompressorConfig, DocstringMode
|
|
|
|
config = CodeCompressorConfig(
|
|
preserve_imports=True, # Always keep imports
|
|
preserve_signatures=True, # Always keep function signatures
|
|
preserve_type_annotations=True, # Keep type hints
|
|
preserve_error_handlers=True, # Keep try/except blocks
|
|
preserve_decorators=True, # Keep decorators
|
|
docstring_mode=DocstringMode.FIRST_LINE, # FULL, FIRST_LINE, REMOVE
|
|
target_compression_rate=0.2, # Keep 20% of tokens
|
|
max_body_lines=5, # Lines to keep per function body
|
|
min_tokens_for_compression=100, # Skip small content
|
|
language_hint=None, # Auto-detect if None
|
|
)
|
|
|
|
compressor = CodeAwareCompressor(config)
|
|
```
|
|
|
|
### Example
|
|
|
|
```python
|
|
from headroom.transforms import CodeAwareCompressor
|
|
|
|
compressor = CodeAwareCompressor()
|
|
|
|
code = '''
|
|
import os
|
|
from typing import List
|
|
|
|
def process_items(items: List[str]) -> List[str]:
|
|
"""Process a list of items."""
|
|
results = []
|
|
for item in items:
|
|
if not item:
|
|
continue
|
|
processed = item.strip().lower()
|
|
results.append(processed)
|
|
return results
|
|
'''
|
|
|
|
result = compressor.compress(code, language="python")
|
|
print(result.compressed)
|
|
# import os
|
|
# from typing import List
|
|
#
|
|
# def process_items(items: List[str]) -> List[str]:
|
|
# """Process a list of items."""
|
|
# results = []
|
|
# for item in items:
|
|
# # ... (5 lines compressed)
|
|
# pass
|
|
|
|
print(f"Compression: {result.compression_ratio:.0%}") # ~55%
|
|
print(f"Syntax valid: {result.syntax_valid}") # True
|
|
```
|
|
|
|
### Supported Languages
|
|
|
|
| Tier | Languages | Support Level |
|
|
|------|-----------|---------------|
|
|
| 1 | Python, JavaScript, TypeScript | Full AST analysis |
|
|
| 2 | Go, Rust, Java, C, C++, C#, PHP | Function body compression |
|
|
|
|
### Memory Management
|
|
|
|
```python
|
|
from headroom.transforms import is_tree_sitter_available, unload_tree_sitter
|
|
|
|
# Check if tree-sitter is installed
|
|
print(is_tree_sitter_available()) # True/False
|
|
|
|
# Free memory when done (parsers are lazy-loaded)
|
|
unload_tree_sitter()
|
|
```
|
|
|
|
---
|
|
|
|
## ContentRouter
|
|
|
|
Intelligent compression orchestrator that routes content to the optimal compressor.
|
|
|
|
### How It Works
|
|
|
|
ContentRouter analyzes content and selects the best compression strategy:
|
|
|
|
1. **Detect content type** — JSON, code, logs, search results, plain text
|
|
2. **Consider source hints** — File paths, tool names for high-confidence routing
|
|
3. **Route to compressor** — SmartCrusher, CodeAwareCompressor, SearchCompressor, etc.
|
|
4. **Log decisions** — Transparent routing for debugging
|
|
|
|
### Configuration
|
|
|
|
```python
|
|
from headroom.transforms import ContentRouter, ContentRouterConfig, CompressionStrategy
|
|
|
|
config = ContentRouterConfig(
|
|
min_section_tokens=100, # Minimum tokens to compress
|
|
enable_code_aware=True, # Use CodeAwareCompressor for code
|
|
enable_search_compression=True, # Use SearchCompressor for grep output
|
|
enable_log_compression=True, # Use LogCompressor for logs
|
|
default_strategy=CompressionStrategy.TEXT, # Fallback strategy
|
|
)
|
|
|
|
router = ContentRouter(config)
|
|
```
|
|
|
|
### Example
|
|
|
|
```python
|
|
from headroom.transforms import ContentRouter
|
|
|
|
router = ContentRouter()
|
|
|
|
# Router auto-detects content type and routes to optimal compressor
|
|
result = router.compress(content)
|
|
|
|
print(result.strategy_used) # CompressionStrategy.CODE_AWARE, SMART_CRUSHER, etc.
|
|
print(result.routing_log) # List of routing decisions
|
|
```
|
|
|
|
### Compression Strategies
|
|
|
|
| Strategy | Used For | Compressor |
|
|
|----------|----------|------------|
|
|
| CODE_AWARE | Source code | CodeAwareCompressor |
|
|
| SMART_CRUSHER | JSON arrays | SmartCrusher |
|
|
| SEARCH | Grep/find output | SearchCompressor |
|
|
| LOG | Log files | LogCompressor |
|
|
| TEXT | Plain text | TextCompressor |
|
|
| PASSTHROUGH | Small content | None |
|
|
|
|
(The earlier `LLMLINGUA` strategy was retired with the LLMLingua integration; ML compression is now provided by Kompress.)
|
|
|
|
### Content Detection
|
|
|
|
The router automatically detects content types by analyzing the content itself:
|
|
|
|
- **Source code**: Detected by syntax patterns, indentation, keywords
|
|
- **JSON arrays**: Detected by JSON structure with array elements
|
|
- **Search results**: Detected by `file:line:` patterns
|
|
- **Log output**: Detected by timestamp and log level patterns
|
|
- **Plain text**: Fallback for prose content
|
|
|
|
No manual hints required - the router inspects content directly.
|
|
|
|
### TOIN Integration
|
|
|
|
ContentRouter records all compressions to TOIN (Tool Output Intelligence Network) for cross-user learning:
|
|
|
|
- **All strategies tracked**: Code, search, logs, text, and ML compressions are recorded
|
|
- **Retrieval feedback**: When users retrieve original content via CCR, TOIN learns which compressions need expansion
|
|
- **Pattern learning**: TOIN builds signatures for each content type to improve future compressions
|
|
|
|
This enables the feedback loop where compression decisions improve based on actual user behavior across all content types, not just JSON arrays.
|
|
|
|
---
|
|
|
|
## TransformPipeline
|
|
|
|
Combine transforms for optimal results.
|
|
|
|
```python
|
|
from headroom import TransformPipeline, SmartCrusher, CacheAligner
|
|
|
|
pipeline = TransformPipeline(
|
|
[
|
|
SmartCrusher(), # First: compress tool outputs
|
|
CacheAligner(), # Then: stabilize prefix
|
|
]
|
|
)
|
|
|
|
result = pipeline.transform(messages)
|
|
print(f"Saved {result.tokens_saved} tokens")
|
|
```
|
|
|
|
### With ML compression (Optional, Kompress)
|
|
|
|
The earlier hand-assembled `TransformPipeline([..., LLMLinguaCompressor(), ...])` recipe is no longer supported. ML compression now ships as part of the live-zone pipeline when the `[ml]` extra is installed; see [ARCHITECTURE.md](ARCHITECTURE.md) for the current placement.
|
|
|
|
### Recommended Order
|
|
|
|
| Order | Transform | Purpose |
|
|
|-------|-----------|---------|
|
|
| 1 | CacheAligner | Stabilize prefix for caching |
|
|
| 2 | SmartCrusher | Compress JSON tool outputs |
|
|
| 3 | Kompress (ML) | ML compression on remaining text (optional, `[ml]` extra) |
|
|
|
|
**Why this order?**
|
|
- CacheAligner first to maximize prefix stability
|
|
- SmartCrusher handles JSON arrays efficiently
|
|
- Kompress compresses remaining long text
|
|
|
|
---
|
|
|
|
## Safety Guarantees
|
|
|
|
All transforms follow strict safety rules:
|
|
|
|
1. **Never remove human content** - User/assistant text is sacred
|
|
2. **Never break tool ordering** - Calls and results stay paired
|
|
3. **Parse failures are no-ops** - Malformed content passes through
|
|
4. **Preserves recency** - Last N turns always kept
|
|
5. **100% error preservation** - Error items never dropped
|