Commit Graph
18 Commits
Author SHA1 Message Date
XandJerrettDavis 54f48fb1c2 fix(code): keep semicolon-separated statements valid (#4630)
## Description

Python's grammar exposes the `;` between two statements on one physical
line as a root-level anonymous token. The leftover top-level sweep
captured it as a fragment, so assembly put a bare `;` on its own line:

```python
v=json.load(open(B+'v.json'))
;
m=json.load(open(B+'m.json'))
```

The result no longer parses, `compress()` discards it and returns the
original, and the proxy logs `Code compression produced invalid syntax
for python (...) returning original` for every such tool result. Two
live fenced blocks (220 and 117 tokens) hit this in the same proxy
session.

The line-based statement collection has a second symptom: when two
statements share one physical line (`x=1; y=2`), both AST nodes map to
the same row. Function bodies appended that row twice; class bodies
appended the full line twice, producing syntactically valid but wrong
output (`x=1; y=2` executed twice, compression ratio inflated to 1.429).

A third symptom surfaced once the output became valid: Python runs
top-level statements in order, but imports are always emitted first.
`import sys; sys.path.insert(0, 'vendor'); import plugin` now compressed
to valid code with `import plugin` hoisted above the path update, so the
plugin failed to load with `ModuleNotFoundError`.

Demoting such imports fixed ordering inside the top-level bucket, but
definitions are assembled before that bucket. `import sys; marker = 1;
import decimal` followed by `class Money(decimal.Decimal)` still moved
the class ahead of `import decimal` and raised `NameError` at definition
time.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Skip unnamed `;`/`,` children in the top-level leftover sweep so a
standalone separator can never become a code fragment.
- Merge overlapping row ranges in `_compress_function_ast` before the
budget loop. Taking the union (instead of dropping the later range)
preserves continuation lines of a multi-line statement that starts after
`;`.
- Track the last emitted row in `_compress_class_ast` and emit shared
rows once, trimming only rows already emitted so continuation lines
survive.
- For Python only, keep a root-level import behind executable top-level
code: the top-level sweep demotes such imports to the top-level code
stream (source order preserved) instead of hoisting them above the code
they depend on. Other languages are untouched because their imports/uses
are order-independent or must lead the file.
- For Python only, decline reassembly when executable top-level code
precedes any declaration. Definitions are emitted before the top-level
code bucket, so a class base, decorator, or default expression after
that code would be evaluated before the code or import it needs. Safe
reassembly cannot be guaranteed, so the original snippet is returned
unchanged. Snippets with no declarations keep compressing with demoted
imports in source order.
- Added `TestSemicolonSeparatedStatements` with seven regression tests:
top-level fallback, class-body duplication, class-body continuation,
function-body shared-line/continuation behavior, import-order
preservation that executes the compressed output, and executable
class-base and decorator/default ordering regressions.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run; this change only
touches untyped helpers
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q
109 passed in 5.22s

$ .venv/bin/ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!

$ .venv/bin/ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted

$ # same new tests against unpatched origin/main 5c8772b7 in a detached worktree
3 failed, 92 deselected in 0.69s
```

## Real Behavior Proof

- Environment: macOS 27.2 (Darwin, arm64), Python 3.12.11 from the repo
`.venv`, head `d581720d` based on main `31dcaf41`, both synced by the
reviewer without changing compressor source.
- Exact command / steps: ran `pytest
tests/test_transforms/test_code_compressor.py -q -k Semicolon` on a
detached worktree of unpatched `origin/main` and on this branch. Then
compressed the two real fenced Python tool results captured from
`~/.headroom/logs/proxy-8788.log` (220- and 117-token blocks) with
`CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=1))`,
and parsed each output with `ast.parse`. For the import-order case,
wrote a module to a temp `vendor/` directory, compressed `import sys;
sys.path.insert(0, '<vendor>'); import <module>`, and executed the
output. For the cross-bucket case, compressed the reviewer's `import
sys; marker = 1; import decimal\nclass Money(decimal.Decimal):\n pass\n`
and a decorator/default variant using `functools.lru_cache` with
`value=marker`, then executed both. Re-ran both counterexamples on the
same 0.42.0 compressor build carrying the import-demotion patch without
the fallback to confirm they reproduced.
- Observed result: on unpatched `5c8772b7` all three original semicolon
tests fail — top-level compression returns the unchanged original after
`Code compression produced invalid syntax for python (35 tokens)`, the
class sample emits `x=1; y=2` twice, and the function sample emits the
shared line twice. On `d581720d` all seven pass; full file 109 passed.
For the 220-token live sample: before the fix it fell back to the
original after the warning; after, ratio `0.995`, no bare `;` line,
`ast.parse` succeeds. The 117-token sample is valid with no bare `;`
(ratio `1.0`, so the router still skips it for low savings). Class
sample: `x=1; y=2` count `2 -> 1`, ratio `1.429 -> 1.0`. Function
sample: shared line count `2 -> 1` and continuation line count `2 -> 1`.
Import-order sample: before the import fix the output was `import sys /
import semicolon_order_plugin / (blank) / sys.path.insert(...)` and
`exec` raised `ModuleNotFoundError: No module named
'semicolon_order_plugin'`; after, the order is `import sys`,
`sys.path.insert(...)`, `import semicolon_order_plugin` and `exec`
succeeds. Cross-bucket samples: before the fallback the class variant
emitted `import sys / class Money(decimal.Decimal) / marker = 1 / import
decimal` and `exec` raised `NameError: name 'decimal' is not defined`;
the decorator variant raised `NameError: name 'functools' is not
defined` at decoration time. After the fallback both return the original
snippet byte-for-byte and `exec` succeeds (`Money` is a
`decimal.Decimal` subclass, `compute()` returns `1`). A semicolon
snippet without declarations (`import sys; sys.path.insert(...); import
json`) still compresses, emitting `import sys`, then the path update,
then `import json`.
- Not tested: the Rust core port
(`crates/headroom-core/src/transforms/code_compressor.rs`) has the same
structure and was not changed; end-to-end proxy traffic was not rerun on
this branch; non-Python fixtures for shared lines were not added (the
range merge is mechanical, but only Python exposes `;` as a root token,
and only Python demotes imports or applies the ordering fallback).

Function output after the fix (shared line and continuation each appear
once):

```python
def f():
    x=1; y=(2 +
            3)
    a=3
    b=4
    c=5
    # [3 lines omitted]
    pass
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: content with semicolon-separated
statements is compressed instead of falling back to the original; shared
lines are emitted once instead of duplicated; Python imports that follow
executable top-level code stay after it; Python snippets where
executable code precedes a definition are returned unchanged instead of
being reordered
- Kill switch / disable path: existing syntax validation and
`HEADROOM_CODE_SYNTAX_BREAKER=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

The top-level separator was only observed for Python: tree-sitter-python
exposes the assignments/calls on either side of `;` as direct module
children, with the `;` itself as an anonymous root sibling. The same
line-based collection is shared by the other grammars, so the range
merge also protects them, but no non-Python fixture was added.

Import demotion is Python-only and only applies after the sweep has
already emitted executable top-level code. Imports before any executable
code keep their place in the hoisted import block, so module docstrings
and license headers are unaffected. `_verify_syntax` also runs
`compile()`, which rejects misplaced `__future__` imports, so a demotion
that violated that rule would fall back to the original instead of
emitting broken ordering.

The ordering fallback is Python-only and is deliberately conservative:
any declaration after executable top-level code disables AST reassembly
for the snippet, even when the definition happens not to reference the
preceding code. The cost is lost compression for that shape; the
alternative is a semantically reordered result that syntax validation
cannot detect.

The Rust core port mirrors these functions (`extract_structure`, body
statement collection, class-body reconstruction). Parity there would
need its own fixtures and is out of scope for this PR.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-04 22:41:55 -06:00
XandJD Davis 7c150ec6b0 fix(code): keep Rust attributes attached to their items (#4624)
## Description

Rust outer attributes (`#[derive(...)]`, `#[inline]`, `#[cfg(...)]`) are
grammar siblings of the item they decorate, exactly like doc comments.
`_get_leading_comment_text` only collected comment node types, so
attributes fell through to the leftover top-level sweep and drifted away
from their item. On a mixed file the compressed output looked like this:

```rust
#[derive(Debug, Clone, PartialEq)]

pub struct Item {
    pub id: u64,
    pub name: String,
}

pub fn log_item(item: &Item, sink: &mut Vec<u64>) {
    sink.push(item.id);
    // [3 lines omitted]
}

/// Logs one item.
#[inline]
#[cfg(feature = "extra")]
```

The attribute block ends up after the last item, and `rustc` rejects the
file with `error: expected item after attributes`. Tree-sitter accepts
it, so the compressor's syntax check does not catch the corruption.

Rust line comments also include their trailing newline in the node text
(`/// doc\n`), so simply attaching them to the same preamble would
introduce blank lines.

Rust also allows blank lines between an outer attribute and the item it
decorates:

```rust
#[inline]

pub fn record(item: &Item) {}
```

The attribute is still attached to the function. Only ordinary comments
require row adjacency; attributes must survive the gap as well.

Comments can also sit between the attribute and its item:

```rust
#[inline]

// Keep with the function below.

pub fn record(item: &Item) {}
```

A blank line before that comment ended the preamble scan before the
attribute was reached. Detached comments are now skipped while scanning,
so the attribute behind one is still collected; the comments themselves
keep the ordinary adjacency policy.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `_ATTACHED_NODE_TYPES = {"attribute_item"}` and made the
leading-sibling collector treat those nodes like comments.
- Renamed the collected list to `preamble`, since it now holds comments
and attributes, and updated the docstring.
- Strip the trailing newline from each collected sibling before joining
(Rust line comments carry one in their node text).
- Keep the blank-line adjacency rule for ordinary comments only;
attributes always attach to the following item, since Rust permits
whitespace between an outer attribute and its item.
- Skip detached comments while scanning the preamble so an outer
attribute behind one is still collected, without capturing those
comments.
- Added `TestRustAttributeAttachment` with four regression tests,
including blank-line-separated attributes and attributes behind an
intervening comment.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run; this change only
touches an untyped helper
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_transforms/test_code_compressor.py -q
96 passed in 7.41s

$ uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!

$ uv run ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted
```

The new blank-line and intervening-comment tests fail on the unpatched
helper and pass with this change.

## Real Behavior Proof

- Environment: macOS 27.2 (Darwin, arm64), Python 3.12.11 from `uv sync
--extra dev --extra code`, `rustc 1.99.0`.
- Exact command / steps: compressed the samples below with
`CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=1,
max_body_lines=1, enable_ccr=False, semantic_analysis=False))`, wrote
each output to disk, then ran `rustc --edition 2021 --cfg
'feature="extra"' --crate-type lib`.
- Observed result: before the fix the compressed output ended with the
orphaned `/// Logs one item.` / `#[inline]` / `#[cfg(...)]` block and
`rustc` exited with `error: expected item after attributes`; with the
first fix ratio `0.808`, the preamble stayed attached, and `rustc` exit
0. For the blank-line sample, before the follow-up the attribute was
detached and `rustc` failed; after the follow-up the output emits
`#[inline]\npub fn record(...)`, ratio `0.795`, and `rustc` exit 0. For
the comment-between sample, before this change the attribute and comment
were emitted after the function and `rustc` failed; after the change the
output emits `#[inline]\npub fn record(...)`, ratio `0.746`, and `rustc`
exit 0. The detached comment keeps the ordinary policy and remains in
the leftover sweep.
- Not tested: attributes on `impl`/`trait`/`use` items (same sibling
mechanism, no dedicated fixture), other grammars (only Rust emits
`attribute_item`), and end-to-end proxy traffic.

Compressed output after the fix (compiled successfully):

```rust
#[derive(Debug, Clone, PartialEq)]
pub struct Item {
    pub id: u64,
    pub name: String,
}

/// Logs one item.
#[inline]
#[cfg(feature = "extra")]
pub fn log_item(item: &Item, sink: &mut Vec<u64>) {
    sink.push(item.id);
    // [3 lines omitted]
}
```

Blank-line attribute sample after the follow-up (compiled successfully):

```rust
pub struct Item { pub id: u64 }

#[inline]
pub fn record(item: &Item, sink: &mut Vec<u64>) {
    sink.push(item.id);
    // [2 lines omitted]
}
```

Comment-between sample after the follow-up (compiled successfully):

```rust
pub struct Item { pub id: u64 }

#[inline]
pub fn record(item: &Item, sink: &mut Vec<u64>) {
    sink.push(item.id);
    // [3 lines omitted]
}

// Keep with the function below.
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: Rust attributes are now emitted with
their item instead of being separated or orphaned
- Kill switch / disable path: existing syntax validation and
`HEADROOM_CODE_SYNTAX_BREAKER=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

The leftover path that caused this is shared by every language, but only
the Rust grammar emits `attribute_item` siblings today, so the behavior
change is scoped to Rust. The Rust core port
(`crates/headroom-core/src/transforms/code_compressor.rs`) has no
equivalent preamble collection; parity there is out of scope for this
PR.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 09:14:51 -06:00
XandJD Davis 0d0cda22b4 fix(code): keep multi-line class headers' opening brace (#4623)
## Description

The code-aware compressor rebuilds a class/impl header from the lines
before the member container, then re-adds the container's start line
only when that line begins with `{`. A multi-line signature can put the
opening brace on a continuation line shared with the signature tail:

```rust
impl<T> Wrapper<T>
where T: Value + Debug + Clone {
```

```java
public class Calc
    extends Base
    implements Runnable {
```

In that case the rebuilt header loses its `{`, the rebuilt source no
longer parses, the syntax check rejects the output, and the compressor
returns the input unchanged — so these containers are never compressed.
On a live proxy this shows up as repeated runtime warnings while
compressing Rust code:

```text
Code compression produced invalid syntax for rust (614 tokens), returning original
Code compression produced invalid syntax for rust (366 tokens), returning original
```

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `_compress_class_ast` keeps the member container's start line whenever
the container node begins with the opening brace, instead of requiring
the line to start with `{`.
- Colon-signature languages (Python) are unchanged: their block node
starts at the first body statement, which can itself begin with `{`, so
they stay on the colon path.
- Added `TestMultiLineClassHeaders` with a Rust `impl ... where ... {`
and a Java `extends`/`implements` header.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run; this change only
adds a branch to existing untyped helper logic
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_transforms/test_code_compressor.py -q
94 passed, 2 warnings in 24.25s

$ uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!

$ uv run ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted
```

Both new tests fail on `main` (the compressor logs `invalid syntax ...
returning original` and `compression_ratio == 1.0`) and pass with this
change.

## Real Behavior Proof

- Environment: macOS 27.2 (Darwin, arm64), Python 3.12.11 from `uv sync
--extra dev --extra code`, base commit `7e0094d`; `rustc 1.99.0`, `javac
17.0.21`.
- Exact command / steps: compressed the samples with
`CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=1,
max_body_lines=2, enable_ccr=False, semantic_analysis=False))`, wrote
the compressed output to disk, then compiled it with `rustc --edition
2021 --crate-type lib <out>.rs` and `javac <out>/Calc.java`.
- Observed result: before the fix both samples logged `Code compression
produced invalid syntax ... returning original` and `compression_ratio`
was `1.0`; after the fix Rust ratio `0.943` with `where T: Value + Debug
+ Clone {` retained and `rustc` exit 0, Java ratio `0.803` with
`implements Runnable {` retained and `javac` exit 0.
- Not tested: C++, JavaScript/TypeScript and C# multi-line class headers
(same code path, but no dedicated fixtures were added), and end-to-end
proxy traffic.

Compressed Rust output (compiled successfully):

```rust
impl<T> Wrapper<T>
where T: Value + Debug + Clone {
    pub fn record(&self, sink: &mut Vec<u64>) {
        for item in &self.items {
            sink.push(item.id());
        }
        // [2 lines omitted]
    }
}
```

Compressed Java header:

```java
public class Calc
    extends Base
    implements Runnable {
    public void compute(int x) {
```

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: stable
- Stable/default behavior changed: compression now succeeds for
containers that previously fell back to the original
- Kill switch / disable path: existing syntax validation and
`HEADROOM_CODE_SYNTAX_BREAKER=0`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

The same header logic exists in the Rust core port
(`crates/headroom-core/src/transforms/code_compressor.rs`), which
currently keeps only `node_lines[..sig_end]` and has the same gap. This
PR intentionally stays scoped to the Python implementation; the Rust
port can follow separately if maintainers want parity.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 09:14:25 -06:00
EndidandJD Davis ac8423965d fix(code): preserve syntax in CCR retrieval comments (#4583)
## Description

`CodeAwareCompressor.compress()` validates the AST-compressed source
before appending its CCR retrieval marker. The marker always started
with `#`, so JavaScript, TypeScript, Go and other C-family outputs
became invalid after validation while still reporting
`syntax_valid=True`.

Use the detected language's existing `LangConfig.comment_prefix` for
that marker. Python keeps `#`; C-family languages use `//`.

Addresses **item 4** of #4478. This deliberately does not close the
umbrella issue or change its other three items.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Reuse the existing per-language comment configuration when appending
CCR metadata.
- Exercise real compression with CCR enabled for Python, JavaScript,
TypeScript and Go. Parse the final output and verify that the recovery
hash retrieves the exact original source.
- No dependency, public API, rollout flag or changelog changes.

## Testing

- [x] New regression tests fail before the fix and pass after it.
- [x] Six relevant compressor suites pass: **149 passed**.
- [x] `ruff check .` passes; both changed files pass `ruff format
--check`.
- [x] Project pre-commit checks (including isolated mypy) and commitlint
pass.
- [x] Manual compression + Node syntax check + real in-memory CCR
retrieval performed.
- The full development environment's `mypy headroom
--ignore-missing-imports` reports two existing errors in
`memory/adapters/embedders.py:389` and
`integrations/langchain/chat_model.py:380`. A baseline run with the
unmodified compressor via `mypy --shadow-file` reports exactly the same
errors. These files are not changed here.

### Test Output

```text
Before: pytest tests/test_transforms/test_code_compressor.py -k ccr_marker_preserves -q
3 failed, 1 passed, 92 deselected
(JavaScript, TypeScript and Go fail the final-source syntax assertion.)

After: same command
4 passed, 92 deselected

pytest tests/test_transforms/test_code_compressor.py \
  tests/test_code_compressor_language_alias.py \
  tests/test_code_compressor_syntax_breaker.py \
  tests/test_code_compressor_bash.py \
  tests/test_code_compressor_thread_safety.py \
  tests/test_transforms/test_code_compressor_cjk.py -q
149 passed in 12.19s
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.12.14, Node 24.16.0,
tree-sitter-language-pack 0.13.0. Python source based on `1fe8549f`;
development/code dependencies installed from `uv.lock`, with the
official headroom-ai 0.42.0 wheel also installed. No provider or model
API call.
- Exact command / steps: run the script below from the checkout with
`HEADROOM_OFFLINE=1`, then run `node --check
/tmp/headroom-ccr-check.mjs`. Run once on the base revision and once
with this patch.

```python
from pathlib import Path
from headroom.cache.compression_store import CompressionStore, set_request_compression_store
from headroom.transforms.code_compressor import CodeAwareCompressor, CodeCompressorConfig
from tests.test_transforms.test_code_compressor import generate_javascript_code

store = CompressionStore()
set_request_compression_store(store)
code = generate_javascript_code()
result = CodeAwareCompressor(
    CodeCompressorConfig(min_tokens_for_compression=10)
).compress(code, language="javascript")
Path("/tmp/headroom-ccr-check.mjs").write_text(
    result.compressed, encoding="utf-8", newline="\n"
)
print(result.syntax_valid, result.compressed.splitlines()[-1])
assert result.cache_key
assert store.retrieve(result.cache_key).original_content == code
```

- Observed result: Before the fix, Headroom reports `True`, but Node
exits 1 with `SyntaxError: Invalid or unexpected token` at `# [200
tokens compressed. Retrieve more: hash=... Expires in 5m.]`.
- After: marker starts `// [200 tokens compressed...`; Node exits 0; CCR
retrieval equals the input byte for byte.
- Not tested: live provider/proxy traffic, Windows/Linux, full
repository test suite, freshly built Rust extension or the full Rust
workspace checks. `make ci-precheck` stops at `cargo fmt` because this
machine has no `cargo`; the Python change does not modify Rust code.

## Runtime Rollout Safety

- Rollout-managed feature(s): None; this fixes the existing code-aware
compressor, which is not gated by the runtime rollout registry.
- Minimum rollout channel: Stable; no beta/canary/dev channel or unsafe
override is required.
- Stable/default behavior changed: only the CCR comment delimiter on
languages whose configured delimiter is not `#`.
- Kill switch / disable path: Set
`CodeCompressorConfig(enable_ccr=False)` to disable the CCR marker and
original-content storage.
- Unsafe override required: no.
- Qualification impact: final source now remains parseable for the
covered languages; CCR payload/hash behavior is preserved.
- Rollback path: Revert commit
`69afcfa7e3acac52b99e638af2331538b3dadea6`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] The change follows project style and uses the existing language
configuration.
- [x] Regression tests cover final output syntax and original-content
recovery.
- [x] Relevant existing tests pass locally.
- [x] `CHANGELOG.md` is unchanged; the Conventional Commit title
supplies the release-note entry.
- [x] No user-facing documentation change is needed for this
syntax-preservation fix.

## Additional Notes

Prepared with AI assistance (Codex), with executed local regression
tests and a Node syntax-check reproduction. The scope is restricted to
#4478 item 4.

An independent Codex review found no blocking defects, checked language
coverage and marker consumers, and independently ran the main compressor
test file (96 passed), changed-file Ruff and `git diff --check`. This is
automated review, not maintainer approval.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4583"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 08:37:17 -06:00
6d5516dcb8 feat(code): add PHP support to CodeAwareCompressor (#2423)
## Description

Adds PHP to `CodeAwareCompressor`, fixing #201. PHP was already
*detected* as code (Magika labels in `headroom/compression/detector.py`
include `php`, and the Rust `magika_detector.rs` lists it too) but there
was no PHP `LangConfig`, so PHP content silently passed through
uncompressed. This wires PHP through the tree-sitter compression path
following the C# pattern (the most recently added, fully functional
language — deliberately not the quarantined Perl path).

A secondary detection bug is fixed along the way: PHP's `$variables`
match Perl's prefilter regex, and the existing Perl-dominance guard in
`detect_language` returned `UNKNOWN` for PHP files. An explicit `<?php`
open tag — which no Perl source contains — now drops Perl from the
candidate set before that guard runs.

## Type of Change

- [ ] Bug fix
- [x] New feature
- [ ] Documentation update
- [ ] Refactor
- [ ] Other

## Changes Made

- `headroom/transforms/code_compressor.py`: `CodeLanguage.PHP` +
`phtml`/`php5`/`php7`/`php8` aliases; PHP `LangConfig` built from the
actual tree-sitter-php grammar (node names verified by parsing samples):
`namespace_use_declaration` imports,
`function_definition`/`method_declaration` functions,
`class_declaration`/`interface_declaration`/`trait_declaration` classes,
`enum_declaration` types, `declaration_list` class bodies,
`compound_statement` function bodies. `namespace_definition` maps to
`package_node` so statement-scoped `namespace App;` hoists ahead of the
`use` imports (required PHP ordering); the rare block-scoped `namespace
A { }` form takes the same path and is preserved verbatim — valid
output, just no compression inside the block. PHP prefilter regexes
added; supported-languages error message updated; `<?php`-tag Perl
disambiguation in `detect_language`.
- `headroom/transforms/content_detector.py`: `php` entry in
`_CODE_PATTERNS` so raw PHP classifies as `SOURCE_CODE` and reaches the
code-aware route.
- `tests/test_transforms/test_code_compressor.py`: new `TestPhpSupport`
mirroring `TestCSharpSupport` — signatures preserved / bodies elided,
`<?php` → `namespace` → `use` → declarations ordering, auto-detection
despite the Perl sigil overlap, alias coercion, malformed passthrough.
- `tests/test_code_compressor_language_alias.py`: `php` in the canonical
list, `phtml` in the alias table.
- `docs/content/docs/code-compression.mdx`: PHP added to the Tier 2
supported-languages row.

No new dependency: `tree-sitter-language-pack` (the existing `[code]`
extra) already ships the PHP grammar. No Rust changes needed.

## Testing

- [x] New unit tests added and passing
- [x] Full affected test suites pass locally

**Test Output**

```
$ python -m pytest tests/test_transforms/test_code_compressor.py tests/test_code_compressor_language_alias.py -q
============================= 120 passed in 7.81s =============================

$ python -m pytest tests/test_transforms/ -q
3 failed, 443 passed   # the 3 failures (kompress ONNX thread caps, kompress size gate,
                       # text_crusher unicode parity) reproduce identically on a clean
                       # upstream/main checkout in this environment — pre-existing local
                       # ONNX runtime quirks, unrelated to this change

$ ruff check . (0.15.17, CI-pinned) → All checks passed!  |  ruff format --check → clean
$ mypy headroom/transforms/code_compressor.py headroom/transforms/content_detector.py --ignore-missing-imports
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13, tree-sitter +
tree-sitter-language-pack (<1.0) installed, branch
`feat/201-php-code-compression` off `upstream/main`.
- Exact command / steps: parsed PHP samples (namespaced class w/
methods, block-scoped namespace, mixed HTML+PHP) with
`tree_sitter_language_pack.get_parser('php')` to verify every node name
used in the config; then ran `CodeAwareCompressor().compress(php_code,
language="php")` and `compress(php_code)` (auto-detection) on a 48-line
realistic service class.
- Observed result: explicit and auto-detected paths both return
`language=CodeLanguage.PHP`, `compression_ratio=0.64`,
`syntax_valid=True`; method bodies elided to `// [N lines omitted]`
while `<?php`, `namespace`, `use` lines, class header, and all
signatures are preserved verbatim in the original order. Before the
detection fix, auto-detection returned `UNKNOWN` (Perl prefilter
dominance) — reproduced and then verified fixed.
- Not tested: exotic PHP shapes (heredoc-heavy code, attributes `#[...]`
on methods, interleaved multi-`<?php ?>` HTML templates beyond the basic
mixed case); these fall back to verbatim preservation via the
uncaptured-node pass or malformed-passthrough, both of which are covered
by tests for the simple cases.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-31 15:54:13 -07:00
Rod BoevandJerrettDavis dbbef4bd41 fix(code-compressor): recover valid Python rewrites after local syntax rejection (#2202)
## Description

A compile-invalid Python definition rewrite currently makes
`CodeAwareCompressor` discard every otherwise valid rewrite in the file
and return the original source at 0 percent reduction. The existing
whole-file safety guard stays in place, while a Python-only recovery
replay now preserves the rejected definition and keeps independent valid
compression.

The recovery reuses the current Python validation authority in
`ast.parse()` plus `compile()`, runs only after the first assembled
module already fails `_verify_syntax()`, and stays out of non-Python
paths.

Closes #1233

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added a Python-only recovery replay after the first assembled module
fails syntax validation.
- Preserved only the invalid function or class rewrite while allowing
independent valid definitions to remain compressed.
- Kept the existing whole-file syntax guard and original-source fallback
as the terminal safety check.
- Added focused invalid-node, valid-modern-syntax, and fail-safe
coverage.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_invalid_node_falls_back_locally
tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_does_not_block_valid_modern_syntax
tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_still_returns_original_when_all_candidates_invalid
tests/test_transforms/test_code_compressor.py::TestEdgeCases::test_syntax_errors_in_input
tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_ast_preserves_structure_for_python
-v`)
- [x] Linting passes (`uv run ruff check
headroom/transforms/code_compressor.py
tests/test_transforms/test_code_compressor.py`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_invalid_node_falls_back_locally tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_does_not_block_valid_modern_syntax tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_still_returns_original_when_all_candidates_invalid tests/test_transforms/test_code_compressor.py::TestEdgeCases::test_syntax_errors_in_input tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_ast_preserves_structure_for_python -v
5 passed in 0.34s
uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!
uv run ruff format headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py --check
2 files already formatted
```

## Real Behavior Proof

- Environment: Windows, synced `uv` environment with `dev` and `code`
extras installed
- Exact command / steps: run the focused invalid-node regression through
public `compress(..., language="python")`
- Observed result: `1 passed in 0.19s`; the invalid candidate stays
original, the neighboring valid candidate remains compressed, and
`headroom-PR-TARGET-1233-PROOF.md` records the base `ratio=1.0`
whole-file rollback against the fixed head behavior.
- Not tested: the stale future-import mismatch discussed in the old
issue comment, already covered on current main

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- The stale future-import comment on #1233 is not the live slice here;
current main already validates Python with `compile()` and already
covers that ordering case.
- This fix keeps the existing whole-file fail-safe and does not broaden
into cross-language recovery or new syntax models.
- `CHANGELOG.md` remains unchanged because Headroom generates release
notes from conventional commits.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-14 20:19:46 -07:00
Yevhen Koval ec55ddcfb3 feat(transforms): first-class C# support in CodeAwareCompressor (#1926)
Refs #1664

## Description

First-class C# support in `CodeAwareCompressor` via the tree-sitter
`csharp` grammar, at parity with Java/C++/Rust: `using` directives,
namespace headers, and type/member signatures preserved verbatim;
method/constructor/destructor/operator/local-function bodies compressed;
malformed input passes through unchanged. **No new dependencies** — the
grammar ships inside the already-pinned
`tree-sitter-language-pack==0.13.0` (resolves only as `"csharp"`;
`c_sharp`/`cs` raise `LookupError`). Spec and maintainer go-ahead in the
issue.

Closes #1664

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `CodeLanguage.CSHARP` + full `_LANG_CONFIGS` entry;
`_LANGUAGE_PREFILTER` and `content_detector` patterns chosen to be
C#-distinctive (so Java doesn't mis-tag).
- New data-driven `LangConfig` fields (pattern of #1334's
`class_body_node_types`): `container_node_types` — block-scoped
`namespace { }` routed through class compression so members compress
without the wrapper being re-emitted verbatim; `opaque_node_types` —
`#if`…`#endif` wrappers preserved verbatim without recursion (recursing
+ wrapper re-emit duplicated whole files, up to ~1.9x input on real
repos); `#if` blocks wrapping only usings are emitted with the imports
so they stay ahead of type declarations.
- Shared-path fixes surfaced by real C# repos, each guarded and covered
by a fail-before test: keep an Allman `{` on its own line in class
reconstruction (K&R path byte-for-byte unchanged; Allman Java now
compresses instead of falling back); line-based child extraction no
longer swallows the following line for nodes ending at column 0 (C#
`#region`/`#endregion` span their trailing newline — the over-slice
duplicated the next member's signature or the closing brace); uncaptured
top-level nodes preceding the first captured node (license banners,
`#region License`) are emitted first instead of relocated below the code
(tree-sitter-c-sharp rejects top-level `#region` after a type
declaration, so relocation forfeited compression for the whole file).
- `TestCSharpSupport` (8 tests) + a C# case in the parametrized
member-container test; CHANGELOG entry.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_transforms/test_code_compressor.py -q
2 failed, 78 passed, 1 warning, 4 errors        # the 2 failures / 4 errors reproduce
                                                # identically on main in the same env
                                                # (network-dependent tokenizer setup)

Fail-before: with both changed sources reverted to main, the new C#-scoped
selection reports "10 failed, 5 passed" (the 5 other languages keep passing);
on the branch: "15 passed".

$ ruff check headroom/transforms/code_compressor.py headroom/transforms/content_detector.py tests/test_transforms/test_code_compressor.py
All checks passed!
$ ruff format --check <same files>
3 files already formatted
```

## Real Behavior Proof

- Environment: Linux 6.8.0 aarch64, Python 3.10.12; `uv run --no-project
--with "tree-sitter-language-pack==0.13.0" --with
"tree-sitter>=0.25.2,<0.26" --with "pydantic>=2.0.0"`; real
`CodeAwareCompressor` (`CodeCompressorConfig(enable_ccr=False)`,
otherwise defaults), no mocks.
- Exact command / steps: cloned two real .NET repos at depth 1
(`github.com/JamesNK/Newtonsoft.Json @ 4f73e74`,
`github.com/App-vNext/Polly @ 7a1d10f`), ran `python proof_csharp.py
<repo>` over every `.cs` file (chars/4 token estimate; tiktoken BPE
download unavailable in my sandbox). Script in the collapsed section
below.
- Observed result: 16.1% tokens saved on Newtonsoft.Json (945/945
syntax-valid), 37.8% on Polly (797/797 syntax-valid), zero content
duplication; full output:

```text
repo: Newtonsoft.Json  (945 .cs files)
  tokens before: 1,777,691   after: 1,490,629   saved: 287,062 (16.1%)
  files compressed: 479   pass-through: 466   inflated(>before): 19
  syntax_valid: 945/945
  latency ms  P50: 0.7  P95: 18.7  P99: 44.1  max: 255.0  mean: 3.5

repo: Polly  (797 .cs files)
  tokens before: 1,100,523   after: 684,303   saved: 416,220 (37.8%)
  files compressed: 693   pass-through: 104   inflated(>before): 15
  syntax_valid: 797/797
  latency ms  P50: 0.8  P95: 11.6  P99: 28.9  max: 74.1  mean: 2.4
```

After rebasing onto current `main` (which touched the same transform
files via #1906/#1747/#1668) I re-ran the Polly proof on the rebased
tree: 37.8% saved, 797/797 syntax-valid, P99 28.5ms — unchanged.
Signatures/properties verbatim, bodies elided with call summaries,
`using` order and preproc balance intact; residual "inflated" files are
+2…+209 chars of assembly blank lines, not duplicated content.
Newtonsoft is the adversarial case (multi-targeting: heavy `#if`,
`#region`, Allman) — its conditional regions stay verbatim by design.
Latency at parity with Java (<50ms P99; max is the pre-existing
symbol-analysis cost on ~1800+-line files, shared with other languages).
- Not tested: proxy end-to-end path with C# through `ContentRouter`
(tested the `CodeAwareCompressor` API directly); CCR retrieval
round-trips (`enable_ccr=False` in proof runs); exact tiktoken counts
(chars/4 estimate — relative ratios are tokenizer-independent);
Windows/macOS; full native `uv run pytest` with the Rust extension (ran
the complete `test_code_compressor.py` in a lightweight venv; its 2
failures/4 errors reproduce identically on `main`); `mypy`.

<details>
<summary>proof_csharp.py (reproducible)</summary>

```python
"""Real behavior proof: run the real CodeAwareCompressor over a .NET repo."""

import pathlib
import statistics
import sys
import time

from headroom.transforms.code_compressor import (
    CodeAwareCompressor,
    CodeCompressorConfig,
)

try:
    import tiktoken

    ENC = tiktoken.get_encoding("cl100k_base")

    def toks(s: str) -> int:
        return len(ENC.encode(s, disallowed_special=()))
except Exception:
    def toks(s: str) -> int:
        return len(s) // 4

target = pathlib.Path(sys.argv[1])
comp = CodeAwareCompressor(CodeCompressorConfig(enable_ccr=False))

tot_before = tot_after = 0
n_files = n_compressed = n_valid = n_passthrough = n_inflated = 0
times_ms: list[float] = []

for f in sorted(target.rglob("*.cs")):
    try:
        code = f.read_text(encoding="utf-8-sig", errors="replace")
    except OSError:
        continue
    t0 = time.perf_counter()
    r = comp.compress(code, language="csharp")
    times_ms.append((time.perf_counter() - t0) * 1000)
    n_files += 1
    b, a = toks(code), toks(r.compressed)
    tot_before += b
    tot_after += a
    if r.compressed == code:
        n_passthrough += 1
    else:
        n_compressed += 1
    if r.syntax_valid:
        n_valid += 1
    if a > b:
        n_inflated += 1

times_ms.sort()
p = lambda q: times_ms[min(int(len(times_ms) * q), len(times_ms) - 1)]
print(f"repo: {target.name}  ({n_files} .cs files)")
print(f"  tokens before: {tot_before:,}   after: {tot_after:,}   saved: {tot_before - tot_after:,} ({(1 - tot_after / tot_before) * 100:.1f}%)")
print(f"  files compressed: {n_compressed}   pass-through: {n_passthrough}   inflated(>before): {n_inflated}")
print(f"  syntax_valid: {n_valid}/{n_files}")
print(f"  latency ms  P50: {p(0.50):.1f}  P95: {p(0.95):.1f}  P99: {p(0.99):.1f}  max: {times_ms[-1]:.1f}  mean: {statistics.mean(times_ms):.1f}")
```

</details>

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — terminal evidence above.

## Additional Notes

- Dependency justification: none added, none bumped; the `csharp`
grammar is inside the already-pinned `tree-sitter-language-pack==0.13.0`
wheel; `uv.lock` untouched.
- Architecture: malformed input passes through byte-identical; every
risky construct prefers the false negative (verbatim) over corruption;
invalid reassembly falls back to the original via the existing
validation gate (observed live); no new imports at module load; P99
<50ms on both proof repos.
- Known v1 limitations (deliberate false negatives, possible
follow-ups): expression-bodied members and property accessor bodies stay
verbatim; declarations inside `#if` regions stay verbatim.
- Related pre-existing finding, out of scope: C/C++ exhibit the same
`#if`-wrapper duplication on `main` (an `#if`-wrapped C++ class is
emitted twice, ratio 1.62). Happy to file separately.
- `mypy` unchecked above because I did not run it in my environment.
2026-07-12 13:54:38 -04:00
adf8fed9bd fix(code): stop TS export duplication + comment displacement (#1906)
## Description

`CodeAwareCompressor` (AST-based code compression,
`headroom/transforms/code_compressor.py`) had two bugs in its
structure-reassembly path, found while investigating a reported Go
brace-duplication issue (the Go bug itself — `statement_list` row-range
swallowing a block's closing brace — was already fixed on `main` in
#1668; this PR fixes what was *actually* still broken):

1. **TS/JS `export` keyword duplication.** `export function foo() {}` /
`export class Foo {}` compressed to `export export function foo() {}` —
invalid syntax, silently discarded by `_verify_syntax`'s fallback (the
caller never sees an error, compression just quietly no-ops). Root
cause: `_compress_function_ast` / `_compress_class_ast` slice a node's
source by **line**, not by byte offset, deliberately — to preserve
leading indentation for definitions nested inside classes. But when a
node shares its *first* line with a preceding sibling (the `export`
keyword is a sibling of the function inside tree-sitter's
`export_statement` node, not part of the function node itself), that
line-based slice pulled the sibling's text in too. The
`export_statement` handler then re-prepended the same `export` text on
top, producing the duplicate.
2. **Doc-comment displacement (all languages).** A `/** ... */` or `//`
doc comment directly above a top-level function/class/type got detached
from its declaration during AST extraction and re-emitted in one cluster
at the very end of the compressed output, instead of staying attached to
what it documents. Root cause: doc comments are top-level *siblings* of
the declaration they document, not children of it — the extractor didn't
attach them to anything, so they fell through to a "leftover top-level
code" bucket that gets flushed as a single block after all functions.

Also tightens `test_actual_go_compression`, which — per its own comment
— was written to *tolerate* the Go bug (`compression_ratio may be 1.0 if
compression produces invalid syntax`) rather than catch it. Since the
underlying Go bug is already fixed on `main`, this now asserts real
compression (`compression_ratio < 1.0`), matching its JS/Python
siblings.

Closes #1905

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

Two commits: the fix itself, then the tests that prove it — bisectable
independently, both pass the full suite on their own.

**Commit 1 — `fix(code):`**
- `headroom/transforms/code_compressor.py`: add `_get_node_lines()` —
line-based node slicing that still preserves indentation, but trims a
preceding sibling's text from the first line when that prefix isn't pure
whitespace (i.e. an `export` keyword sharing the line), so callers that
re-add the sibling text themselves don't get a duplicate; used by
`_compress_function_ast` and `_compress_class_ast`.
- `headroom/transforms/code_compressor.py`: add
`_get_leading_comment_text()` — walks a node's `prev_sibling` chain to
collect contiguous doc-comment nodes immediately above it (no blank line
in between) and returns them for the caller to prepend, also marking
their byte ranges as captured so they aren't independently swept into
the leftover top-level-code bucket; wired into every capture branch in
`_extract_structure` (package, import, export statement, decorator,
function, class, type).
- `CHANGELOG.md`: added an entry under `### Fixed`.

**Commit 2 — `test(code):`**
- `tests/test_transforms/test_code_compressor.py`:
`test_actual_go_compression` now asserts `compression_ratio < 1.0`
instead of tolerating a 1.0 fallback.
- `tests/test_code_aware_brace_comment_regressions.py` (new): 4
regression tests — TS `export` not duplicated + valid syntax, TS doc
comments stay attached, Go doc comments stay attached, and a
real-TS-compression parity test matching the existing JS/Python/Go
"actual compression" tests.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py tests/test_code_aware_brace_comment_regressions.py
All checks passed!

$ ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py tests/test_code_aware_brace_comment_regressions.py
3 files already formatted

$ pytest tests/test_transforms/test_code_compressor.py tests/test_code_aware_regressions.py tests/test_code_aware_brace_comment_regressions.py -q
83 passed in 6.23s

$ pytest -q   # full suite
7912 passed, 5 failed, 442 skipped in 417.43s (0:06:57)
# The 5 failures are pre-existing and unrelated: confirmed to fail identically
# with this PR's changes stashed out (clean upstream/main checkout).
#   - test_wrap_marker_is_stale_when_pid_reused (PID-reuse detection, env-specific)
#   - test_read_cached_oauth_token_falls_back_to_gh_cli (leaks real local `gh` credentials)
#   - test_rtk_reader_returns_none_on_nonzero_exit / test_lean_ctx_reader_returns_none_on_failure_and_logs
#     (pass in isolation; fail only in full-suite order — pre-existing test-pollution, unrelated to code_compressor.py)
#   - test_parser_usable_in_thread_pool (test itself passes a str to parser.parse(),
#     which tree-sitter's binding has always required as bytes — a pre-existing test
#     bug unrelated to this change; separate fix in progress on another branch)

$ mypy headroom
Success: no issues found in 408 source files
```

## Real Behavior Proof

- Environment: macOS (Darwin 24.6.0), Python 3.14.5, headroom-ai dev
checkout built via `uv sync --extra dev` + `maturin develop -m
crates/headroom-py/Cargo.toml` (real `headroom._core` build, not
mocked), `tree-sitter==0.25.2` / `tree-sitter-language-pack` per the
pinned `[code]` extra.
- Exact command / steps: ran
`CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=10,
enable_ccr=False)).compress(open("sdk/typescript/src/client.ts").read(),
language="typescript")` identically against `git stash`-ed (pre-fix) and
current (post-fix) trees; full snippet and additional samples below.
- Observed result: `client.ts` (real 20KB SDK file in this repo) went
from `compression_ratio=1.0` with a silent fallback (`export export
class HeadroomClient` in the raw AST attempt, invalid syntax) to
`compression_ratio=0.942`, `syntax_valid=True` — real compression, no
duplication; full before/after table below.
- Not tested: real-world repos beyond this repo's own SDK sample and the
bundled benchmark fixture — broader corpus testing may follow as a
comment on this PR.

**Exact command, full snippet:**

```python
from headroom.transforms.code_compressor import CodeAwareCompressor, CodeCompressorConfig
compressor = CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=10, enable_ccr=False))
with open("sdk/typescript/src/client.ts") as f:
    code = f.read()
result = compressor.compress(code, language="typescript")
```

**Observed result, before vs. after, real code:**

| Sample | Before (main) | After (this fix) |
|---|---|---|
| `sdk/typescript/src/client.ts` (real 20KB SDK file, this repo) |
`compression_ratio=1.0`, silent fallback — `export export class
HeadroomClient` in the raw AST attempt, invalid syntax |
`compression_ratio=0.942`, `syntax_valid=True` — real compression, no
duplication |
| TS fixture exercising both bugs (exported fn/class + doc comments) |
`compression_ratio=1.0`, silent fallback | `compression_ratio=0.993`,
`syntax_valid=True` |
| `middleware/ratelimit.go` (bundled benchmark sample) |
`compression_ratio=0.862`, `syntax_valid=True` — unaffected (Go bug
already fixed on `main` by #1668) | `compression_ratio=0.862`,
`syntax_valid=True` — unchanged, confirms no regression |
| `generate_go_code(3)` (existing test fixture) |
`compression_ratio=0.498` | `compression_ratio=0.498` — unchanged,
confirms no regression |

On code shaped to actually exercise elision (function bodies long enough
to exceed `max_body_lines=5`), TypeScript compresses in line with other
languages once the correctness bug stops blocking it entirely:

| Language | Compression savings (synthetic fixture, ~10-line function
bodies) |
|---|---|
| Python | 64.4% |
| Go | 52.3% |
| TypeScript | 49.0% |
| JavaScript | 42.8% |

(`client.ts`'s real-world 5.8% savings is lower than the synthetic
TypeScript number above because most of its methods are ≤5 lines — under
the elision threshold regardless of language — not because of a
language-specific limitation.)

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The Go brace-duplication bug that motivated this investigation was
already fixed on `main` (#1668, merged before this branch was based) —
confirmed via the minimal repro and `ratelimit.go`, both compress
cleanly with no duplicated braces. This PR fixes what was still actually
broken: the TS/JS `export`-duplication bug and the doc-comment
displacement bug (both present across languages), found empirically
while verifying the original bug report against the current `main`.

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-07-09 12:51:32 -05:00
Vinay Gupta 82384022bd fix(code): slice tree-sitter byte offsets as UTF-8 (#1332)
## Description

CodeAwareCompressor was slicing Python strings with tree-sitter
`start_byte` / `end_byte` offsets directly. That works for ASCII-only
files, but it corrupts slices after non-ASCII source text such as CJK
characters or emoji because tree-sitter offsets are UTF-8 byte offsets
while Python string indexes are character offsets.

This caused code-aware compression to produce invalid intermediate
Python and then safely fall back to the original file, resulting in 0%
compression on affected files.

Closes #1319

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `_slice_code_bytes()` in
`headroom/transforms/code_compressor.py` to slice source text using
UTF-8 byte offsets.
- Updated `_get_node_text()` to use byte-safe slicing.
- Routed the other direct tree-sitter byte-offset slices through the
same helper.
- Added regression tests in
`tests/test_transforms/test_code_compressor.py`:
  - `test_get_node_text_uses_utf8_byte_offsets`
  - `test_ast_compresses_python_after_non_ascii_source`

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py
68 passed, 1 warning

$ .venv/bin/python -m ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!

$ .venv/bin/python -m ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted

$ git diff --check
# no output

$ /tmp/headroom-1319-venv/bin/python -m mypy headroom
headroom/proxy/server.py:1186: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
headroom/proxy/server.py:1257: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
headroom/proxy/server.py:1261: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
pyproject.toml: note: unused section(s): module = ['mlx.*']
Success: no issues found in 394 source files
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.14.5, tree-sitter 0.25.2,
tree-sitter-language-pack 0.13.0
- Exact command / steps: On `main`, ran a local reproducer with a Python
source string containing a CJK docstring before a second function;
called `_get_node_text()` on the second tree-sitter function node; ran a
full `CodeAwareCompressor.compress(...)` repro with non-ASCII module
text before an import and a compressible function; re-ran both repros on
this branch.
- Observed result: Before fix, `_get_node_text()` returned the wrong
slice (`'nd():\n return 2\n'` instead of `'def second():\n return 2'`)
and full compression fell back to the original file with
`compression_ratio: 1.0`; after fix, `_get_node_text()` returns the full
expected function slice and full compression succeeds with
`compression_ratio < 1.0`, `syntax_valid: True`, and does not return the
original.
- Not tested: Full repository test suite; live proxy/provider
integrations; Windows/Linux platform-specific behavior.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation was not updated because this is an internal bug fix with
no user-facing API or behavior change beyond restoring intended
compression.
- `CHANGELOG.md` was not updated because the fix is narrow and
issue-scoped; maintainers can advise if they want a changelog entry.
- The fix is intentionally small and targeted: it only changes how
tree-sitter byte offsets are converted back into Python source text,
without changing compression heuristics or language behavior.
2026-06-23 15:03:44 -05:00
Vinay Gupta c35af858ea fix(code): compress class member containers (#1334)
## Description

CodeAwareCompressor used the same `body_node_types` config to find both
executable function bodies and class/impl member containers. That works
when those AST nodes happen to match, but it misses member containers
such as Java `class_body`, C++ `field_declaration_list`, and Rust
`declaration_list`, so class methods were returned essentially
uncompressed.

This adds an optional `class_body_node_types` override for class/impl
member containers and uses it only in class compression. It also skips
anonymous punctuation tokens while reconstructing class bodies and keeps
same-line C++ class semicolons attached to the compressed class
declaration.

Closes #1318

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `LangConfig.class_body_node_types` for languages whose
class/impl member container differs from executable method-body nodes.
- Configured class member containers for JavaScript, TypeScript, Java,
C++, and Rust.
- Updated `_compress_class_ast` to use class-member containers, skip
anonymous punctuation children, and preserve C++ `};` output without
creating stray top-level semicolons.
- Added regression coverage proving class/impl methods compress for
JavaScript, TypeScript, Java, C++, and Rust.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ PYTHONPATH=. /tmp/headroom-1319-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q
collected 71 items
tests/test_transforms/test_code_compressor.py .......................... [ 36%]
.............................................                            [100%]
71 passed, 1 warning in 0.36s

$ /tmp/headroom-1319-venv/bin/python -m ruff check .
All checks passed!

$ /tmp/headroom-1319-venv/bin/python -m ruff format --check .
965 files already formatted

$ PYTHONPATH=. /tmp/headroom-1319-venv/bin/python -m mypy headroom
headroom/proxy/server.py:1186: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
headroom/proxy/server.py:1257: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
headroom/proxy/server.py:1261: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
pyproject.toml: note: unused section(s): module = ['mlx.*']
Success: no issues found in 394 source files
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.14.5, branch
`fix-code-compressor-class-members`, tree-sitter grammar pack installed
in `/tmp/headroom-1319-venv`, repo imported with `PYTHONPATH=.`.
- Exact command / steps: Reproduced class-method compression with
`CodeAwareCompressor(CodeCompressorConfig(enable_ccr=False,
min_tokens_for_compression=1, max_body_lines=1))` for Java/C++/Rust
before the fix, then reran the pytest/ruff/mypy commands listed above
after the patch.
- Observed result: Java/C++/Rust class methods now compress below 1.0
while `syntax_valid` remains true; C++ output preserves `};`; regression
coverage also verifies JavaScript/TypeScript class member containers.
- Not tested: Full repository pytest suite; local `uv run` editable
builds are blocked on this machine by native C++ header failures in
optional/native dependencies (`hnswlib` / Rust `esaxx-rs`), so
validation used a lightweight venv with `PYTHONPATH=.`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

Documentation and CHANGELOG updates are not applicable for this narrow
bug fix. The pytest warning shown above is from running without
`pytest-asyncio` in the lightweight verification venv (`asyncio_mode`
config is unknown there); it is unrelated to this change.
2026-06-23 14:41:36 -05:00
Parafee41andJD Davis cbd361de2a fix(code): validate Python compressed syntax (#1302)
## Description

Fix a Python code-compression validity gap from #1233 where tree-sitter
parsing could mark compressed output as syntactically valid even when
Python compile-time syntax rules reject it.

This keeps `from __future__ import ...` statements in the
import-preservation bucket so they stay before executable definitions,
and adds Python `compile(..., "exec")` verification after `ast.parse`.
It also keeps the earlier conservative class-method decorator
indentation hardening from this branch.

Refs #1233.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Treat Python `future_import_statement` nodes as preserved imports.
- Verify Python compressed output with both `ast.parse` and
`compile(..., "exec")`.
- Preserve original source-line indentation for decorators attached to
class methods.
- Add a regression fixture covering `from __future__ import
annotations`, class decorators, property decorators, async methods, and
`match` statements.
- Add a direct regression assertion that future imports stay before
executable definitions.
- Document the user-visible fix in `CHANGELOG.md`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ /tmp/headroom-issue-1233-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py::TestTreeSitterIntegration::test_python_future_import_stays_at_module_start -q
1 passed, 1 warning

$ /tmp/headroom-issue-1233-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q
61 passed, 1 warning

$ uv run --extra dev ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!

$ uv run --extra dev ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted

$ git diff --check
# no output
```

## Real Behavior Proof

- Environment: macOS, Python 3.11.14, local checkout with `[code]`
dependencies installed in `/tmp/headroom-issue-1233-venv`.
- Exact command / steps: added
`test_python_future_import_stays_at_module_start`, ran it before the fix
to confirm the compressed output failure, then reran the focused test
and full `tests/test_transforms/test_code_compressor.py` after the
patch.
- Observed result: before this patch, the regression fixture produced
compressed Python with `from __future__ import annotations` after
class/function definitions. `result.syntax_valid` was `True`, but
`compile(result.compressed, "<test>", "exec")` failed with `SyntaxError:
from __future__ imports must occur at the beginning of the file`. After
this patch, the focused regression and full code-compressor test file
pass locally, and the regression now directly asserts that the future
import appears before executable definitions.
- Not tested: full repository pytest, `mypy headroom`, and a broad
corpus run over third-party source files.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

This PR is now scoped to the stable compile-time failure path in #1233.
The broader syntax-failure rate from the issue may still need
corpus-level follow-up.

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-06-23 14:41:14 -05:00
Rocker Zhang 5e0bb69725 fix(code): verify a real parse in tree-sitter availability check (#1231) (#1299)
## Description

`is_tree_sitter_available()` / `_check_tree_sitter_available()` in
`headroom/transforms/code_compressor.py` return `True` based on
importing `tree_sitter_language_pack` alone, without ever constructing a
parser or attempting a parse. When the installed pack/parser combination
is ABI-incompatible, `get_parser`/`parse` raises at runtime; the caller
catches it and silently falls back to the lossy text compressor, while
the availability flag and startup banner still report code-aware as on.
This is the defensive half that the `<1.0` pin in #1234 does not cover:
if that cap is ever lifted, the availability signal silently lies again.
Follow-up to #1231.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Make `_check_tree_sitter_available()` construct a parser and parse a
tiny snippet, returning `True` only if it yields a real `module` AST
instead of trusting an import.
- Add `_tree_sitter_importable()` for the cheap import-only probe, and
use it to guard parser construction so the real-parse check cannot
recurse.
- Add tests asserting the check is `False` when parsing raises and
`True` on a real parse, plus that AST compression runs for python/rust
without falling back.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
# pytest tests/test_transforms/test_code_compressor.py  -> passed locally (tree-sitter-language-pack 0.13.0)
# ruff check . and ruff format --check . pass locally on the rebased branch.
# Full pytest suite / mypy not run locally; left to CI.
```

## Real Behavior Proof

- Environment: local repo on tree-sitter-language-pack 0.13.0,
tree-sitter 0.25.2, Python 3.12, Linux
- Exact command / steps: call `is_tree_sitter_available()`, then run
`pytest tests/test_transforms/test_code_compressor.py`
- Observed result: with a working pack the probe parses and returns
`True` (code-aware runs, strategy `CODE_AWARE` rather than the kompress
fallback); the new
`test_check_tree_sitter_available_false_when_parse_broken` confirms that
when parsing raises the check now returns `False` instead of the old
import-only `True`, so the lossy fallback is no longer entered silently.
- Not tested: reproducing the specific ABI-incompatible 1.x pack combo
against a live install (covered instead by a mocked broken parse in the
test)

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
2026-06-22 23:00:57 -05:00
chopratejas 72eebd4fe6 Fix CI test, bump to 0.5.7
- test_nested_functions: guard syntax_valid assert behind
  is_tree_sitter_available() (CI doesn't have tree-sitter)
- Bump version to 0.5.7
2026-03-26 12:04:30 -07:00
chopratejas 3290a3d582 Remove LLMLingua: Kompress is the sole text compressor
LLMLingua was the original ML text compressor (BERT-based). Kompress
(ModernBERT, trained on 330K structured tool outputs) replaced it with
better compression quality and simpler architecture.

Removed across 35 files:
- Deleted headroom/transforms/llmlingua_compressor.py
- Deleted tests/test_transforms/test_llmlingua_compressor.py
- Deleted tests/test_proxy_llmlingua.py
- Removed all enable_llmlingua config, _get_llmlingua methods,
  LLMLingua fallback paths, LLMLINGUA strategy enum values
- Removed CLI flags, model configs, compression handler references
- Simplified ContentRouter: Kompress is primary and only text compressor
2026-03-26 11:11:00 -07:00
Tejas ChopraandClaude Opus 4.6 4a655fbc6e feat: provider-aware prefix cache tracking and combined savings dashboard
Add per-provider prefix cache metrics (Anthropic/OpenAI/Google/Bedrock)
with correct economics (read discounts, write premiums, bust detection).
Model-aware bust detection excludes cold starts when switching models.
Dashboard hero metric shows combined savings (compression + cache) with
per-provider breakdown table, efficiency bar, and hit rate tracking.

- Add _CACHE_ECONOMICS dict and _build_prefix_cache_stats() helper
- Track cache_by_provider with per-model cold start awareness
- Add _merge_cost_stats() to combine compression + cache savings
- Dashboard: "Prefix Cache Impact" section with provider breakdown
- Dashboard: hero "Total Savings" shows compress + cache breakdown
- Fix ruff (unused var, quoted annotations) and mypy type errors
- Refactor code_compressor to data-driven language config

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 14:14:15 -07:00
chopratejas 9c31b22bff feat(code): add semantic symbol importance to CodeAwareCompressor
CodeAwareCompressor now analyzes intra-file symbol relationships before
compression, using tree-sitter AST walks to count references, map call
graphs, and detect public/private visibility. This replaces uniform
"keep first N body lines" compression with budget-based allocation driven
by the existing target_compression_rate config.

Key design decisions:
- Distribution-based scoring (min-max normalized within each file) so it
  adapts to any file structure: utility libs, test files, orchestrators
- Budget allocation: target_compression_rate determines total body line
  budget, distributed proportionally to importance × body size
- max_body_lines respected as a hard cap over budget allocation
- Context-aware: the existing `context` parameter now boosts symbols
  matching the user's task (word-boundary matching, not substring)
- Qualified names (ClassName.method) internally to avoid collisions
  between identically-named methods in different classes
- Omitted comments include call graph info from AST analysis
- Zero new dependencies — uses tree-sitter already in headroom[code]
- semantic_analysis=True by default, fully backward-compatible when False
2026-03-03 14:18:41 -08:00
chopratejas 2ce26438a0 Integrate DynamicContentDetector into CacheAligner (Phase 1)
- Add DynamicContentDetector integration for comprehensive dynamic content
  detection (20+ patterns vs previous 4 date patterns)
- New detection: UUIDs, API keys, JWT tokens, Unix timestamps, request/trace
  IDs, hex hashes (MD5/SHA1/SHA256), version numbers, high-entropy strings
- Add CacheAlignerConfig options: use_dynamic_detector, detection_tiers,
  extra_dynamic_labels, entropy_threshold
- Maintain backward compatibility with legacy date-only mode
- Add 25 new comprehensive tests for Phase 1 functionality
- Fix code compressor fallback test to properly mock LLMLingua availability

Expected cache hit improvement: 30-50% by extracting more dynamic content
2026-01-19 22:56:20 -08:00
chopratejas 905c229251 Add AST-based code compression and custom model configuration
CodeAwareCompressor:
- Tree-sitter based AST parsing for Python, JS, TS, Go, Rust, Java, C, C++
- Preserves imports, signatures, type annotations, error handlers
- Guarantees syntactically valid output
- Uses tree-sitter-language-pack for broad language support

ContentRouter:
- Intelligent compression orchestrator
- Auto-routes content to optimal compressor based on type detection
- Source hint support for high-confidence routing

Custom Model Configuration:
- HEADROOM_MODEL_LIMITS env var and ~/.headroom/models.json support
- Pattern-based inference for unknown models (opus/sonnet/haiku tiers)
- Support for Claude 4.5, Claude 4, o3, o3-mini
- Graceful fallback - never crashes on unknown models
2026-01-14 13:46:55 -08:00