mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
The AST pass validates its own output and returns the original on
failure ("never serve broken code"). That guarantee is right, but it
means a grammar that keeps mangling real code costs the full compression
latency on every request and then throws the work away. Field
measurement from one install's retained logs: 356 invalid-syntax
discards (TypeScript 270, Python 47, JS 27, PHP 8, C++ 4), each preceded
by ~1.2s in `compressor:mixed` - the slowest stage in that install's
pipeline spending most of its time on work that was discarded, while the
user files the added latency under "the proxy is slow".
This PR adds a dynamic per-language circuit breaker beside the existing
static `_UNSAFE_TREE_SITTER_LANGUAGES` quarantine (which is for grammars
unsafe to load; this is for grammars that load fine but fail validation
on real traffic). Once 10 of the last 20 validations for a language
fail, attempts for that language pause for 30 minutes and requests pass
through at original size immediately. The trip logs one warning with the
counts; the window clears on trip so a fresh run of failures must
re-earn the next pause. Other languages are isolated, and compression
output is unchanged in every case - the breaker only decides whether to
attempt, never what to serve.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/transforms/code_compressor.py`: module-level breaker state
(`_SYNTAX_BREAKER_WINDOW=20`, `_SYNTAX_BREAKER_MIN_FAILURES=10`,
`_SYNTAX_BREAKER_COOLDOWN_S=1800`), `_syntax_breaker_open()` checked
after language detection and before the parse,
`_record_syntax_outcome()` fed by the existing syntax-validation result
(including the Python recovery retry's final verdict). Lock-guarded;
compression already runs in a thread pool.
- Kill switch: `HEADROOM_CODE_SYNTAX_BREAKER=0|false|off` disables both
recording and skipping.
- New `tests/test_code_compressor_syntax_breaker.py`: trip threshold,
success dilution, cooldown expiry and re-earn, env kill switch, and a
short-circuit test proving `_compress_with_ast` is not invoked while
open.
## Testing
- [x] Unit tests pass (`pytest`)
### Test Output
```text
uv run --frozen --extra dev pytest tests/test_code_compressor_syntax_breaker.py tests/test_code_compressor_language_alias.py tests/test_code_compressor_thread_safety.py -q
33 passed, 8 skipped
```
## Real Behavior Proof
- Environment: macOS 15 (arm64), Python 3.12 via uv, this branch's
worktree.
- Exact command / steps: pytest run above; the short-circuit test
pre-opens the breaker for python and monkeypatches `_compress_with_ast`
to raise if reached, then calls `CodeAwareCompressor.compress()` on a
20x-repeated function body.
- Observed result: with the breaker open the compressor returns the
original body byte-identical (ratio 1.0, syntax_valid) without entering
the AST pass; with it closed behavior is unchanged.
- Not tested: a live proxy soak against the field TypeScript corpus that
produced the 270 discards (not in the repo); the 10/20/30min constants
are judgment values, tunable in one place at the top of the module.
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: Yes, narrowly: a language failing
validation on at least 10 of its last 20 attempts passes through
uncompressed for 30 minutes instead of being attempted-and-discarded
every request. Served bytes are identical in both cases (the discard
path already returned the original).
- Kill switch / disable path: HEADROOM_CODE_SYNTAX_BREAKER=0 restores
today's always-attempt behavior.
- Unsafe override required: No.
- Qualification impact: Installs with a failing grammar lose the latency
of doomed attempts and a small amount of would-be compression on the
failing language only; the warning names the language and counts when it
trips.
- Rollback path: Revert the commit; no state or schema outlives the
process (breaker state is in-memory only).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
108 lines
4.3 KiB
Python
108 lines
4.3 KiB
Python
"""Per-language circuit breaker for AST compression that keeps failing.
|
|
|
|
The compressor validates its own output and discards invalid results, so a
|
|
misbehaving grammar burns the full compression latency on every request and
|
|
throws the work away (field measurement: 270 discarded TypeScript compressions
|
|
in one install's retained logs at ~1.2s each). After enough failures in the
|
|
recent window the breaker pauses that language for a cooldown; other languages
|
|
and the original content are unaffected.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from headroom.transforms import code_compressor as cc
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def reset_breaker_state():
|
|
cc._syntax_breaker_outcomes.clear()
|
|
cc._syntax_breaker_open_until.clear()
|
|
cc._syntax_breaker_trips.clear()
|
|
yield
|
|
cc._syntax_breaker_outcomes.clear()
|
|
cc._syntax_breaker_open_until.clear()
|
|
cc._syntax_breaker_trips.clear()
|
|
|
|
|
|
def test_breaker_trips_after_min_failures_in_window():
|
|
for _ in range(cc._SYNTAX_BREAKER_MIN_FAILURES - 1):
|
|
cc._record_syntax_outcome("typescript", False)
|
|
assert not cc._syntax_breaker_open("typescript")
|
|
cc._record_syntax_outcome("typescript", False)
|
|
assert cc._syntax_breaker_open("typescript")
|
|
# Per-language isolation: python keeps compressing.
|
|
assert not cc._syntax_breaker_open("python")
|
|
|
|
|
|
def test_successes_keep_the_breaker_closed():
|
|
for _ in range(cc._SYNTAX_BREAKER_WINDOW * 2):
|
|
cc._record_syntax_outcome("typescript", True)
|
|
for _ in range(cc._SYNTAX_BREAKER_MIN_FAILURES - 1):
|
|
cc._record_syntax_outcome("typescript", False)
|
|
assert not cc._syntax_breaker_open("typescript")
|
|
|
|
|
|
def test_breaker_reopens_after_cooldown(monkeypatch):
|
|
now = [1000.0]
|
|
# Patch the module-local indirection, not the real time module: anything
|
|
# else running concurrently (xdist worker, background thread) reads that one.
|
|
monkeypatch.setattr(cc, "_now", lambda: now[0])
|
|
for _ in range(cc._SYNTAX_BREAKER_MIN_FAILURES):
|
|
cc._record_syntax_outcome("typescript", False)
|
|
assert cc._syntax_breaker_open("typescript")
|
|
now[0] += cc._SYNTAX_BREAKER_COOLDOWN_S + 1
|
|
assert not cc._syntax_breaker_open("typescript")
|
|
# The cleared window means one more failure does not instantly re-trip.
|
|
cc._record_syntax_outcome("typescript", False)
|
|
assert not cc._syntax_breaker_open("typescript")
|
|
|
|
|
|
def test_env_kill_switch(monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_CODE_SYNTAX_BREAKER", "0")
|
|
for _ in range(cc._SYNTAX_BREAKER_WINDOW):
|
|
cc._record_syntax_outcome("typescript", False)
|
|
assert not cc._syntax_breaker_open("typescript")
|
|
|
|
|
|
def test_compress_short_circuits_while_open(monkeypatch):
|
|
monkeypatch.setattr(cc, "_check_tree_sitter_available", lambda: True)
|
|
|
|
def _boom(self, *a, **kw): # pragma: no cover - must not run
|
|
raise AssertionError("AST compression attempted while breaker open")
|
|
|
|
monkeypatch.setattr(cc.CodeAwareCompressor, "_compress_with_ast", _boom)
|
|
cc._syntax_breaker_open_until["python"] = cc._now() + 60.0
|
|
|
|
compressor = cc.CodeAwareCompressor(cc.CodeCompressorConfig(min_tokens_for_compression=1))
|
|
code = "def f():\n return 1\n" * 20
|
|
result = compressor.compress(code, language="python")
|
|
assert result.compressed == code
|
|
assert result.compression_ratio == 1.0
|
|
assert result.syntax_valid is True
|
|
|
|
|
|
def test_status_reports_open_languages_for_stats(monkeypatch):
|
|
"""While the breaker is open the language compresses at ratio 1.0, so /stats
|
|
needs to say why rather than leaving an unexplained savings drop."""
|
|
now = [1000.0]
|
|
monkeypatch.setattr(cc, "_now", lambda: now[0])
|
|
|
|
assert cc.syntax_breaker_status() == {}
|
|
|
|
for _ in range(cc._SYNTAX_BREAKER_MIN_FAILURES):
|
|
cc._record_syntax_outcome("typescript", False)
|
|
status = cc.syntax_breaker_status()["typescript"]
|
|
assert status["open"] is True
|
|
assert status["trips"] == 1
|
|
assert status["reopens_in_seconds"] == pytest.approx(cc._SYNTAX_BREAKER_COOLDOWN_S)
|
|
|
|
now[0] += cc._SYNTAX_BREAKER_COOLDOWN_S + 1
|
|
reopened = cc.syntax_breaker_status()["typescript"]
|
|
assert reopened["open"] is False
|
|
# The trip count survives the cooldown: a language that keeps tripping is
|
|
# the field diagnosis, and it would be invisible if this reset.
|
|
assert reopened["trips"] == 1
|
|
assert reopened["reopens_in_seconds"] == 0.0
|