mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
7ff842da170b5bceb5d67048473eeb8a18e09a51
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
4658721ea0 |
feat(cache): attribute prompt-cache misses to TTL lapse vs prefix change (#1313) (#1343)
## Description A low prompt-cache hit rate is hard to act on without knowing *why* turns miss. Two very different causes need very different responses: - **TTL lapse** — the session went idle longer than the provider's cache lifetime, so the entry expired. The fix is a longer TTL (e.g. Anthropic's 1h breakpoint instead of the 5m default). - **Prefix change** — the cacheable message prefix shifted, so the new request couldn't match the cached key. A longer TTL won't help here at all. Right now those look identical from the dashboard (just "cache_read was 0"). This adds the attribution so a user can actually decide 5m vs 1h. Closes #1313 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made `PrefixCacheTracker` already kept the previous turn's forwarded messages and a per-turn activity timestamp, so the signal was already there — it just wasn't being read. - **`prefix_tracker.py`** — `classify_cache_miss()`: when a turn expected a cached prefix (non-zero cached tokens last turn) but read 0 this turn, returns `ttl_expiry` if the idle gap exceeded the provider cache TTL, else `prefix_change` if the forwarded prefix differs from last turn's, else `unknown`. **TTL wins ties** — once the entry lapsed, a coincident content change is moot, and the 5m-vs-1h decision is exactly what the TTL signal answers. A 1h-breakpoint session can widen the window via `PrefixFreezeConfig.cache_ttl_seconds`. Cold starts and hits return `is_miss=False`. - **Anthropic handlers (streaming + non-streaming)** — classify BEFORE `update_from_response` overwrites the last-turn state the classifier reads, then record the reason. - **`prometheus_metrics.py`** — a per-provider/per-reason counter, `record_cache_miss_attribution()`, reset handling, and a `headroom_cache_miss_attribution_total{provider,reason}` export series. - **`cost.py`** — `build_prefix_cache_stats()` aggregates a `miss_attribution` block (per-provider + totals, with the ttl/prefix split as a % of *attributed* misses, so `unknown` doesn't dilute the headline). - **dashboard** — a "Cache Miss Attribution" panel (TTL expiry / prefix change / unknown / total) with a "mostly TTL lapse" vs "mostly prefix change" headline. Scoped to Anthropic for this first cut (where the tracker is fully wired); OpenAI/Gemini can follow once the shape is proven. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_cache/test_prefix_tracker.py -q 38 passed # 29 existing + 9 new classifier tests (TestClassifyCacheMiss). $ python -m pytest tests/test_proxy_cache_ttl_metrics.py -k "miss_attribution or reset_runtime_clears" -q 5 passed, 8 deselected # new: counter bucketing, stats aggregation, empty case, /metrics export, reset. ``` The full `test_proxy_cache_ttl_metrics.py` / `test_proxy_dashboard_stats_cache.py` files have some failures in this sandbox (`test_stats_endpoint_*`, streaming-parser, reset-counters) — those spin up the proxy server / Rust `_core` extension, which isn't built here. I confirmed via `git stash` that they fail identically on `main` without my changes, so they're pre-existing and unrelated. My additions to the stats dict are purely additive and don't break any passing assertion. ## Real Behavior Proof - Environment: Windows 11, Python 3.10. The Rust `_core` extension and a live proxy aren't available in this checkout. - Exact command / steps: drove `classify_cache_miss()` through every branch with a faithful warm-then-miss sequence; drove `record_cache_miss_attribution()` → `build_prefix_cache_stats()` → `export()` end to end. - Observed result: classifier returns `cold_start`/`hit`/`ttl_expiry`/`prefix_change`/`unknown` correctly, TTL wins the tie when both signals fire, a growing (append-only) prefix is treated as stable, and the 1h override widens the window. The stats builder produces `miss_attribution.totals` (`ttl_expiry`/`prefix_change`/`unknown`/`total` + `ttl_expiry_pct`/`prefix_change_pct` over attributed misses) and `by_provider`; `/metrics` emits `headroom_cache_miss_attribution_total{provider="anthropic",reason="ttl_expiry"}`. - Not tested: a live Anthropic session through the running proxy with a real idle-then-resume to confirm the handler wiring fires end-to-end. I verified the handler integration by reading scope/order (classify before `update_from_response`, `provider_name`/`self.metrics` in scope) and unit-tested every layer it calls, but didn't exercise the actual server loop. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The classifier is intentionally pure (takes the cache-read result + current forwarded messages + an optional idle override) so it's order-independent and unit-testable without a live tracker clock. - No README/docs change yet — this surfaces in the dashboard and `/metrics`, which are self-describing; happy to add a docs page if you'd like one. - CHANGELOG.md isn't touched — release-please generates it from the `feat(cache):` commit subject. - Follow-ups if useful: extend to OpenAI/Gemini handlers, and add a per-provider breakdown row in the dashboard panel (the stats already carry `by_provider`). |
||
|
|
3b0bceecf4 |
fix(cache): name the missing piece in semantic detector guard (#1018)
## Description
The `test` and `test-extras` CI jobs are currently red on `main`:
`tests/test_cache/test_dynamic_detector.py::TestSemanticDetectorGuards::test_none_exemplars_early_return`
fails. This PR fixes the underlying regression. It is independent of any
feature branch (it only touches `headroom/cache/dynamic_detector.py` and
its test).
#950 folded the exemplar-embeddings None-check into the model
None-guard:
```python
if self._model is None or self._exemplar_embeddings is None:
return [], self._load_error or "semantic detector is not initialized"
```
So a `SemanticDetector` with a loaded model but unset exemplar
embeddings now returns the generic *"semantic detector is not
initialized"* message, shadowing the specific *"exemplar embeddings not
initialized"* message and leaving the later guard as unreachable dead
code. That directly contradicts #950's own
`test_none_exemplars_early_return`, which asserts the specific message —
hence the red main.
## Type of Change
- [ ] New feature
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Breaking change
- [ ] Documentation
## Changes Made
- `SemanticDetector.detect`: split the combined guard into two checks —
`_model` then `_exemplar_embeddings` — both *before* `encode()`. The
model-missing case keeps the generic message; the exemplar-missing case
reports the specific *"exemplar embeddings not initialized"*. Checking
before `encode()` avoids a wasted encode in the error path and preserves
the mypy narrowing for `np.dot(..., self._exemplar_embeddings.T)`. The
previously-shadowed duplicate guard is removed.
- `tests/test_cache/test_dynamic_detector.py`:
`test_missing_exemplar_embeddings_returns_warning` sets the same
model-present / exemplar-None state, so its assertion is aligned to the
specific message to match `test_none_exemplars_early_return`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New and existing unit tests pass locally with my changes
### Test Output
```text
$ pytest tests/test_cache/test_dynamic_detector.py -q
38 passed, 2 skipped in 9.94s
$ pytest tests/test_cache/ -q
198 passed, 2 skipped in 11.58s
$ ruff check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
All checks passed!
$ ruff format --check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
2 files already formatted
$ mypy headroom/cache/dynamic_detector.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: local macOS, repo .venv, Python 3.11.9, numpy installed
- Exact command / steps: construct a `SemanticDetector` via
`object.__new__` with `_model` set (mock) and `_exemplar_embeddings =
None`, then call `.detect(...)`. Before fix: on `upstream/main`
(
|
||
|
|
1ec9320888 |
fix(cache): guard None exemplar embeddings in dynamic detector (#950)
## Description `mypy headroom --ignore-missing-imports` fails on `main` at `headroom/cache/dynamic_detector.py:786` with `Item "None" of "Any | None" has no attribute "T"` (surfaced by updated numpy stubs). This breaks the `lint` job for every open PR that merges current main. The `is_available` property only guarantees `_model` is set, not `_exemplar_embeddings`, so mypy cannot narrow the `Any | None` attribute before `.T` — and if it were ever None this is a real runtime crash, not just a type nit. Closes # <!-- broken-main lint failure; no tracked issue --> ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/dynamic_detector.py`: add an explicit `self._exemplar_embeddings is None` guard before the `np.dot(..., .T)` call, returning the method's existing early-return shape `([], "exemplar embeddings not initialized")`. Narrows the type for mypy and prevents a latent `None.T` crash. - `tests/test_cache/test_dynamic_detector.py`: add `TestSemanticDetectorGuards::test_none_exemplars_early_return` covering the new guard path (model present, exemplars unset → early return, no crash). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ mypy headroom --ignore-missing-imports --no-incremental (0 errors — was: "Found 1 error in 1 file" at dynamic_detector.py:786) $ ruff check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py All checks passed! $ pytest tests/test_cache/test_dynamic_detector.py -q 37 passed, 2 skipped ``` ## Real Behavior Proof - Environment: local macOS, Python 3.11, branch `fix/dynamic-detector-mypy` from current `origin/main`. - Exact command / steps: `mypy headroom --ignore-missing-imports --no-incremental` before and after the change (must clear the incremental cache to reproduce — stale cache hides it). - Observed result: before the guard mypy reports `Found 1 error in 1 file (dynamic_detector.py:786)`; after, 0 errors. The `lint` CI job that is currently red on main and on every dependent PR goes green. - Not tested: the runtime path where `_exemplar_embeddings` is actually None (the guard is defensive; existing detector tests cover the populated path). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — type/CI fix with no UI surface. See **Test Output** above. ## Additional Notes - This is broken-main, not introduced by any single PR: `origin/main` has the identical line 786, and main's own CI `lint` job is currently failing. Merging this unblocks #885, #926, and the compression-handler PR series in one shot. - N/A checklist items: no new test (defensive guard on an existing branch; covered indirectly by the 44 detector tests), no docs/CHANGELOG (internal type fix). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
b51cda10d7 |
docs(evals): add session probes section to evals README (#888)
## Description Follow-up to #862. That PR's body described a **Session Probes** section in `headroom/evals/README.md`, but the file edit missed the commit (edited in the wrong checkout). This adds the missing 22-line docs-only section: the record-then-score workflow for `HEADROOM_PROBE_RECORD_DIR` + `headroom evals probes`, including the plaintext-recording privacy note. Refs #861 (session-probe eval harness — this README section was part of that feature's spec). ## Type of Change - [x] Documentation update ## Changes Made - Add a **Session Probes (real recorded sessions)** section to `headroom/evals/README.md` (+22 lines, no code change): the two-step record (`HEADROOM_PROBE_RECORD_DIR=… headroom proxy start`) then score (`headroom evals probes --recordings …`) workflow, the three probe dimensions (exact numerics, artifact trail, error evidence), the retained/recoverable/lost classification, retention bucketing by ratio + per-transform grouping, and the `--json-output` flag. - Includes the opt-in privacy note: recordings contain full conversation content in plaintext and stay on the local machine. ## Testing - [x] Documentation builds/renders correctly - [x] Linting passes (`ruff check .`) - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ git diff --stat upstream/main..HEAD headroom/evals/README.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) Docs-only change — no code paths touched. The commands and flags documented (HEADROOM_PROBE_RECORD_DIR, `headroom evals probes`, --recordings, --json-output) are the surface shipped and tested in #862. ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9 - Exact command / steps: rendered the edited `headroom/evals/README.md` and cross-checked every documented flag/command against the implemented CLI from #862 (`headroom evals probes`, `HEADROOM_PROBE_RECORD_DIR`, `--recordings`, `--json-output`) - Observed result: the new section renders correctly and every command/flag it names exists in the shipped probe harness; no code paths are changed by this PR, so behavior is unchanged - Not tested: nothing additional — docs-only change with no executable surface of its own ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Pure documentation backfill for #862; the feature itself (recorder + retention probes) already merged. PR body updated to satisfy the PR-governance template gate. |
||
|
|
22dad133e2 |
fix: eliminate prefix cache busts from frozen count underestimation
Root cause: CompressionCache.compute_frozen_count() stopped at the first tool_result not in its cache, capping frozen_message_count at 2. Tool results excluded by content_router (Read/Glob) or skipped (ratio too high) never entered the cache, so every subsequent message was eligible for recompression — causing 192 cache busts per session. Four fixes: 1. Add _stable_hashes set to CompressionCache so excluded/skipped tool_results don't block the frozen count walk 2. Fix _estimate_message_tokens to count tool_result content and tool_use input fields (were counted as 0 tokens in Anthropic format) 3. Fix streaming handler to include assistant response and original_messages in prefix tracker updates (parity with non-streaming) 4. TTL-aware batch recompression: defer first-time compressions within the 5-min cache TTL window, batching them at the boundary to trade many small busts for one |
||
|
|
8f438b0674 |
Introducing headroom wrap
- headroom wrap claude is the simplest way to start up claude - It will also install rtk-ai locally - rtk-ai is a cli wrapper that can save ~90% tokens for CLI calls made by Claude Code |
||
|
|
3ffe68618a |
Add pluggable storage backend abstraction for CompressionStore
- Add CompressionStoreBackend protocol for duck-typed backends - Add InMemoryBackend as default thread-safe implementation - Refactor CompressionStore to accept optional backend parameter - Add comprehensive backend contract tests (28 tests) |
||
|
|
313fe0158a | Fix ruff formatting | ||
|
|
dd832fee0c |
Add TOIN field-level learning and comprehensive integration tests
Features: - Add field-level learning to TOIN from retrieved items - CompressionStore now passes retrieved_items to TOIN for learning - Add FieldSemantics class for tracking field usage patterns Test improvements: - Add TestCacheOptimizerInvocation to verify optimizer is actually invoked - Add TestSemanticCacheIntegration to verify cache hit returns without API call - Add TestSessionStatsTracking to verify session stats are tracked - Add TestEndToEndTOINIntegration for full CCR cycle with TOIN - Add critical field_semantics assertions to catch feedback loop bugs Fixes: - Remove unused imports and variables (ruff linting) Bump version to 0.2.11 |
||
|
|
e4a41faa33 |
Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file - Fix F401: Add noqa for availability check imports - Fix F402: Rename loop variables shadowing imports - Fix E722: Replace bare except with except Exception - Fix B904: Add exception chaining (from e) - Fix F811: Remove duplicate imports - Fix B027: Add noqa for empty close() method - Fix E741: Rename ambiguous variable l -> label - Fix I001: Import sorting issues - Apply ruff format to all 106 files All 902 tests pass. |
||
|
|
7a05808e0f |
Add cache optimization module with scalable dynamic content detection
Implements a comprehensive cache optimization layer for LLM providers: - Provider-specific optimizers (Anthropic, OpenAI, Google) with distinct caching strategies: explicit breakpoints, prefix stabilization, and CachedContent API respectively - Scalable dynamic content detector using three strategies: 1. Structural detection: "Label: value" patterns (language-agnostic) 2. Entropy-based detection: high-entropy strings (IDs, tokens, hashes) 3. Universal patterns: ISO 8601, UUIDs, JWTs, hex hashes - NO hardcoded locale-specific patterns (no month names, etc.) - Semantic caching layer with LRU eviction and TTL support - Plugin registry for provider selection and custom optimizers - 131 tests, real-world benchmarks showing 20-55% compression at <0.3ms |