mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
7ff842da170b5bceb5d67048473eeb8a18e09a51
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
82384022bd |
fix(code): slice tree-sitter byte offsets as UTF-8 (#1332)
## Description CodeAwareCompressor was slicing Python strings with tree-sitter `start_byte` / `end_byte` offsets directly. That works for ASCII-only files, but it corrupts slices after non-ASCII source text such as CJK characters or emoji because tree-sitter offsets are UTF-8 byte offsets while Python string indexes are character offsets. This caused code-aware compression to produce invalid intermediate Python and then safely fall back to the original file, resulting in 0% compression on affected files. Closes #1319 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `_slice_code_bytes()` in `headroom/transforms/code_compressor.py` to slice source text using UTF-8 byte offsets. - Updated `_get_node_text()` to use byte-safe slicing. - Routed the other direct tree-sitter byte-offset slices through the same helper. - Added regression tests in `tests/test_transforms/test_code_compressor.py`: - `test_get_node_text_uses_utf8_byte_offsets` - `test_ast_compresses_python_after_non_ascii_source` ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py 68 passed, 1 warning $ .venv/bin/python -m ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py All checks passed! $ .venv/bin/python -m ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py 2 files already formatted $ git diff --check # no output $ /tmp/headroom-1319-venv/bin/python -m mypy headroom headroom/proxy/server.py:1186: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1257: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1261: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] pyproject.toml: note: unused section(s): module = ['mlx.*'] Success: no issues found in 394 source files ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.14.5, tree-sitter 0.25.2, tree-sitter-language-pack 0.13.0 - Exact command / steps: On `main`, ran a local reproducer with a Python source string containing a CJK docstring before a second function; called `_get_node_text()` on the second tree-sitter function node; ran a full `CodeAwareCompressor.compress(...)` repro with non-ASCII module text before an import and a compressible function; re-ran both repros on this branch. - Observed result: Before fix, `_get_node_text()` returned the wrong slice (`'nd():\n return 2\n'` instead of `'def second():\n return 2'`) and full compression fell back to the original file with `compression_ratio: 1.0`; after fix, `_get_node_text()` returns the full expected function slice and full compression succeeds with `compression_ratio < 1.0`, `syntax_valid: True`, and does not return the original. - Not tested: Full repository test suite; live proxy/provider integrations; Windows/Linux platform-specific behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes - Documentation was not updated because this is an internal bug fix with no user-facing API or behavior change beyond restoring intended compression. - `CHANGELOG.md` was not updated because the fix is narrow and issue-scoped; maintainers can advise if they want a changelog entry. - The fix is intentionally small and targeted: it only changes how tree-sitter byte offsets are converted back into Python source text, without changing compression heuristics or language behavior. |
||
|
|
c35af858ea |
fix(code): compress class member containers (#1334)
## Description CodeAwareCompressor used the same `body_node_types` config to find both executable function bodies and class/impl member containers. That works when those AST nodes happen to match, but it misses member containers such as Java `class_body`, C++ `field_declaration_list`, and Rust `declaration_list`, so class methods were returned essentially uncompressed. This adds an optional `class_body_node_types` override for class/impl member containers and uses it only in class compression. It also skips anonymous punctuation tokens while reconstructing class bodies and keeps same-line C++ class semicolons attached to the compressed class declaration. Closes #1318 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `LangConfig.class_body_node_types` for languages whose class/impl member container differs from executable method-body nodes. - Configured class member containers for JavaScript, TypeScript, Java, C++, and Rust. - Updated `_compress_class_ast` to use class-member containers, skip anonymous punctuation children, and preserve C++ `};` output without creating stray top-level semicolons. - Added regression coverage proving class/impl methods compress for JavaScript, TypeScript, Java, C++, and Rust. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ PYTHONPATH=. /tmp/headroom-1319-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q collected 71 items tests/test_transforms/test_code_compressor.py .......................... [ 36%] ............................................. [100%] 71 passed, 1 warning in 0.36s $ /tmp/headroom-1319-venv/bin/python -m ruff check . All checks passed! $ /tmp/headroom-1319-venv/bin/python -m ruff format --check . 965 files already formatted $ PYTHONPATH=. /tmp/headroom-1319-venv/bin/python -m mypy headroom headroom/proxy/server.py:1186: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1257: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1261: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] pyproject.toml: note: unused section(s): module = ['mlx.*'] Success: no issues found in 394 source files ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.14.5, branch `fix-code-compressor-class-members`, tree-sitter grammar pack installed in `/tmp/headroom-1319-venv`, repo imported with `PYTHONPATH=.`. - Exact command / steps: Reproduced class-method compression with `CodeAwareCompressor(CodeCompressorConfig(enable_ccr=False, min_tokens_for_compression=1, max_body_lines=1))` for Java/C++/Rust before the fix, then reran the pytest/ruff/mypy commands listed above after the patch. - Observed result: Java/C++/Rust class methods now compress below 1.0 while `syntax_valid` remains true; C++ output preserves `};`; regression coverage also verifies JavaScript/TypeScript class member containers. - Not tested: Full repository pytest suite; local `uv run` editable builds are blocked on this machine by native C++ header failures in optional/native dependencies (`hnswlib` / Rust `esaxx-rs`), so validation used a lightweight venv with `PYTHONPATH=.`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and CHANGELOG updates are not applicable for this narrow bug fix. The pytest warning shown above is from running without `pytest-asyncio` in the lightweight verification venv (`asyncio_mode` config is unknown there); it is unrelated to this change. |
||
|
|
cbd361de2a |
fix(code): validate Python compressed syntax (#1302)
## Description Fix a Python code-compression validity gap from #1233 where tree-sitter parsing could mark compressed output as syntactically valid even when Python compile-time syntax rules reject it. This keeps `from __future__ import ...` statements in the import-preservation bucket so they stay before executable definitions, and adds Python `compile(..., "exec")` verification after `ast.parse`. It also keeps the earlier conservative class-method decorator indentation hardening from this branch. Refs #1233. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Treat Python `future_import_statement` nodes as preserved imports. - Verify Python compressed output with both `ast.parse` and `compile(..., "exec")`. - Preserve original source-line indentation for decorators attached to class methods. - Add a regression fixture covering `from __future__ import annotations`, class decorators, property decorators, async methods, and `match` statements. - Add a direct regression assertion that future imports stay before executable definitions. - Document the user-visible fix in `CHANGELOG.md`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ /tmp/headroom-issue-1233-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py::TestTreeSitterIntegration::test_python_future_import_stays_at_module_start -q 1 passed, 1 warning $ /tmp/headroom-issue-1233-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q 61 passed, 1 warning $ uv run --extra dev ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py All checks passed! $ uv run --extra dev ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py 2 files already formatted $ git diff --check # no output ``` ## Real Behavior Proof - Environment: macOS, Python 3.11.14, local checkout with `[code]` dependencies installed in `/tmp/headroom-issue-1233-venv`. - Exact command / steps: added `test_python_future_import_stays_at_module_start`, ran it before the fix to confirm the compressed output failure, then reran the focused test and full `tests/test_transforms/test_code_compressor.py` after the patch. - Observed result: before this patch, the regression fixture produced compressed Python with `from __future__ import annotations` after class/function definitions. `result.syntax_valid` was `True`, but `compile(result.compressed, "<test>", "exec")` failed with `SyntaxError: from __future__ imports must occur at the beginning of the file`. After this patch, the focused regression and full code-compressor test file pass locally, and the regression now directly asserts that the future import appears before executable definitions. - Not tested: full repository pytest, `mypy headroom`, and a broad corpus run over third-party source files. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes This PR is now scoped to the stable compile-time failure path in #1233. The broader syntax-failure rate from the issue may still need corpus-level follow-up. Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
5e0bb69725 |
fix(code): verify a real parse in tree-sitter availability check (#1231) (#1299)
## Description `is_tree_sitter_available()` / `_check_tree_sitter_available()` in `headroom/transforms/code_compressor.py` return `True` based on importing `tree_sitter_language_pack` alone, without ever constructing a parser or attempting a parse. When the installed pack/parser combination is ABI-incompatible, `get_parser`/`parse` raises at runtime; the caller catches it and silently falls back to the lossy text compressor, while the availability flag and startup banner still report code-aware as on. This is the defensive half that the `<1.0` pin in #1234 does not cover: if that cap is ever lifted, the availability signal silently lies again. Follow-up to #1231. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Make `_check_tree_sitter_available()` construct a parser and parse a tiny snippet, returning `True` only if it yields a real `module` AST instead of trusting an import. - Add `_tree_sitter_importable()` for the cheap import-only probe, and use it to guard parser construction so the real-parse check cannot recurse. - Add tests asserting the check is `False` when parsing raises and `True` on a real parse, plus that AST compression runs for python/rust without falling back. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # pytest tests/test_transforms/test_code_compressor.py -> passed locally (tree-sitter-language-pack 0.13.0) # ruff check . and ruff format --check . pass locally on the rebased branch. # Full pytest suite / mypy not run locally; left to CI. ``` ## Real Behavior Proof - Environment: local repo on tree-sitter-language-pack 0.13.0, tree-sitter 0.25.2, Python 3.12, Linux - Exact command / steps: call `is_tree_sitter_available()`, then run `pytest tests/test_transforms/test_code_compressor.py` - Observed result: with a working pack the probe parses and returns `True` (code-aware runs, strategy `CODE_AWARE` rather than the kompress fallback); the new `test_check_tree_sitter_available_false_when_parse_broken` confirms that when parsing raises the check now returns `False` instead of the old import-only `True`, so the lossy fallback is no longer entered silently. - Not tested: reproducing the specific ABI-incompatible 1.x pack combo against a live install (covered instead by a mocked broken parse in the test) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable |
||
|
|
72eebd4fe6 |
Fix CI test, bump to 0.5.7
- test_nested_functions: guard syntax_valid assert behind is_tree_sitter_available() (CI doesn't have tree-sitter) - Bump version to 0.5.7 |
||
|
|
3290a3d582 |
Remove LLMLingua: Kompress is the sole text compressor
LLMLingua was the original ML text compressor (BERT-based). Kompress (ModernBERT, trained on 330K structured tool outputs) replaced it with better compression quality and simpler architecture. Removed across 35 files: - Deleted headroom/transforms/llmlingua_compressor.py - Deleted tests/test_transforms/test_llmlingua_compressor.py - Deleted tests/test_proxy_llmlingua.py - Removed all enable_llmlingua config, _get_llmlingua methods, LLMLingua fallback paths, LLMLINGUA strategy enum values - Removed CLI flags, model configs, compression handler references - Simplified ContentRouter: Kompress is primary and only text compressor |
||
|
|
4a655fbc6e |
feat: provider-aware prefix cache tracking and combined savings dashboard
Add per-provider prefix cache metrics (Anthropic/OpenAI/Google/Bedrock) with correct economics (read discounts, write premiums, bust detection). Model-aware bust detection excludes cold starts when switching models. Dashboard hero metric shows combined savings (compression + cache) with per-provider breakdown table, efficiency bar, and hit rate tracking. - Add _CACHE_ECONOMICS dict and _build_prefix_cache_stats() helper - Track cache_by_provider with per-model cold start awareness - Add _merge_cost_stats() to combine compression + cache savings - Dashboard: "Prefix Cache Impact" section with provider breakdown - Dashboard: hero "Total Savings" shows compress + cache breakdown - Fix ruff (unused var, quoted annotations) and mypy type errors - Refactor code_compressor to data-driven language config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
|
9c31b22bff |
feat(code): add semantic symbol importance to CodeAwareCompressor
CodeAwareCompressor now analyzes intra-file symbol relationships before compression, using tree-sitter AST walks to count references, map call graphs, and detect public/private visibility. This replaces uniform "keep first N body lines" compression with budget-based allocation driven by the existing target_compression_rate config. Key design decisions: - Distribution-based scoring (min-max normalized within each file) so it adapts to any file structure: utility libs, test files, orchestrators - Budget allocation: target_compression_rate determines total body line budget, distributed proportionally to importance × body size - max_body_lines respected as a hard cap over budget allocation - Context-aware: the existing `context` parameter now boosts symbols matching the user's task (word-boundary matching, not substring) - Qualified names (ClassName.method) internally to avoid collisions between identically-named methods in different classes - Omitted comments include call graph info from AST analysis - Zero new dependencies — uses tree-sitter already in headroom[code] - semantic_analysis=True by default, fully backward-compatible when False |
||
|
|
2ce26438a0 |
Integrate DynamicContentDetector into CacheAligner (Phase 1)
- Add DynamicContentDetector integration for comprehensive dynamic content detection (20+ patterns vs previous 4 date patterns) - New detection: UUIDs, API keys, JWT tokens, Unix timestamps, request/trace IDs, hex hashes (MD5/SHA1/SHA256), version numbers, high-entropy strings - Add CacheAlignerConfig options: use_dynamic_detector, detection_tiers, extra_dynamic_labels, entropy_threshold - Maintain backward compatibility with legacy date-only mode - Add 25 new comprehensive tests for Phase 1 functionality - Fix code compressor fallback test to properly mock LLMLingua availability Expected cache hit improvement: 30-50% by extracting more dynamic content |
||
|
|
905c229251 |
Add AST-based code compression and custom model configuration
CodeAwareCompressor: - Tree-sitter based AST parsing for Python, JS, TS, Go, Rust, Java, C, C++ - Preserves imports, signatures, type annotations, error handlers - Guarantees syntactically valid output - Uses tree-sitter-language-pack for broad language support ContentRouter: - Intelligent compression orchestrator - Auto-routes content to optimal compressor based on type detection - Source hint support for high-confidence routing Custom Model Configuration: - HEADROOM_MODEL_LIMITS env var and ~/.headroom/models.json support - Pattern-based inference for unknown models (opus/sonnet/haiku tiers) - Support for Claude 4.5, Claude 4, o3, o3-mini - Graceful fallback - never crashes on unknown models |