Commit Graph
10 Commits
Author SHA1 Message Date
603d873992 fix(config): use the longest matching model_context_limits prefix (#4288)
## Description

`HeadroomConfig.get_context_limit()` resolves a user's
`model_context_limits` override by prefix, but it returned the **first**
key that prefixes the model name, in dict insertion order. A more
specific override listed after a general one was never used:

```python
HeadroomClient(..., model_context_limits={"gpt-4": 8192, "gpt-4o": 128000})
client._get_context_limit("gpt-4o-2024-08-06")   # 8192 on main, 128000 expected
```

`HeadroomClient._get_context_limit()` uses this value as the context
budget, so a dated `gpt-4o` snapshot was compressed against an 8K window
instead of 128K. `ModelRegistry.get()` already prefers the longest
prefix for the same reason; this applies the same rule to the
user-override path.

Closes: N/A (no matching issue found)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/config.py`: `get_context_limit()` collects every key that
prefixes the model and returns the longest one's limit. Exact matches
and the "no match → `None`" path are unchanged, and the existing loose
prefix behaviour (`"gpt-4"` still matches `"gpt-4o"` when it is the only
key) is kept, so `test_get_context_limit_prefix_match` passes unchanged.
- `tests/test_config.py`:
`test_get_context_limit_prefers_longest_prefix` puts the general key
first (`gpt-4` before `gpt-4o`, `claude-3` before `claude-3-5-sonnet`)
and checks that the specific one wins, and that the general one still
applies when it is the only match.

## Testing

- [x] Unit tests pass (`pytest`): `tests/test_config.py`, not the full
suite
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): not run
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New test on `main` (fix reverted, test kept):

```text
$ python -m pytest tests/test_config.py -q -k context_limit
    assert config.get_context_limit("gpt-4o-2024-08-06") == 128000
E   AssertionError: assert 8192 == 128000
================== 1 failed, 4 passed, 32 deselected in 4.71s ==================
```

With the fix:

```text
$ python -m pytest tests/test_config.py -q
============================== 37 passed in 2.20s ==============================

$ ruff check headroom/config.py tests/test_config.py && ruff format --check headroom/config.py tests/test_config.py
All checks passed!
2 files already formatted
```

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.12.11, source tree on this
branch (native `headroom._core` from the 0.41.0 wheel), real
`openai.OpenAI` client (no request sent) and the real `OpenAIProvider`.
- Exact command / steps: built a real
`HeadroomClient(original_client=openai.OpenAI(api_key=...),
provider=OpenAIProvider(), store_url="sqlite:///…",
model_context_limits={"gpt-4": 8192, "gpt-4o": 128000})` and printed
`client._get_context_limit(model)` for three models, on `main` and on
this branch.
- Observed result: on `main` the dated `gpt-4o-2024-08-06` snapshot gets
the 8192 `gpt-4` limit; on this branch it gets the 128000 `gpt-4o`
limit, and the other two models are unchanged.
  ```text
  == main
  gpt-4-0613           -> 8192
  gpt-4o               -> 128000
  gpt-4o-2024-08-06    -> 8192      # wrong: gpt-4o override ignored
  == this branch
  gpt-4-0613           -> 8192
  gpt-4o               -> 128000
  gpt-4o-2024-08-06    -> 128000
  ```
- Not tested: a live `chat.completions.create(headroom_mode="optimize")`
call against a real API; the full test suite.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: only for users whose
`model_context_limits` has two keys where one prefixes the other; the
more specific key now wins
- Kill switch / disable path: N/A (remove the overlapping key from
`model_context_limits`)
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

No docs change needed: the documented behaviour ("prefix matching for
versioned model names") is what this makes true.

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4288"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-11-04 22:42:37 -06:00
Satya BharadwajandJD Davis 47fd8cb3ba fix(client): store metrics at HeadroomConfig.store_url (#4270)
## Description

`HeadroomClient` ignored `HeadroomConfig.store_url`. The constructor
copied `config.store_url` into `self._store_url`, but then opened
storage with its own `store_url` argument:

```python
if config.store_url:
    self._store_url = config.store_url   # saved here
...
self._storage = create_storage(store_url)  # but storage uses the argument
```

So `HeadroomClient(...,
config=HeadroomConfig(store_url="sqlite:///mine.db"))` wrote metrics to
`<system temp>/headroom.db`, and `mine.db` was never created.
`validate_setup()` reads `self._store_url`, so it reported `mine.db` as
the storage URL. The client said it stored metrics in one place and
actually stored them in another. `get_metrics()` / `get_summary()` also
read from the temp database, so anything reading `mine.db` directly (for
example `generate_report(store_url="sqlite:///mine.db")`) found nothing.

The constructor docstring says the config "takes precedence over
individual settings like store_url".

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/client.py`: storage is opened with `self._store_url`, the
URL the constructor already resolves and reports.
- `headroom/client.py`: the client no longer writes its resolved URL
back into a caller-supplied config (the old `else:
self._config.store_url = store_url`).
- **Why:** a caller can reuse one `HeadroomConfig` object for several
clients. Client A would fill `shared.store_url` with its own URL, and
client B would then read that as an explicit config value and store into
A's database despite passing its own `store_url=`. (Reported by Devin
review; the write-back was harmless on `main` only because the old
truthy default meant it never ran and storage ignored the config
anyway.)
- The client keeps its resolved URL in `self._store_url`, which is all
it uses. A client built without a config still records the URL on the
`HeadroomConfig` it creates for itself.
- `headroom/config.py`: `HeadroomConfig.store_url` now defaults to
`None` instead of `"sqlite:///headroom.db"`. `None` means "let the
client decide": its `store_url` argument, else `headroom.db` in the
system temp directory.
- **Why the default had to change:** the old default was always truthy,
so with only the one-line fix every
`HeadroomClient(config=HeadroomConfig())` would have started writing
`headroom.db` into the current working directory. v0.2.0 (90d3aea4,
"Fixed SQLite default path (now uses temp directory)") deliberately
moved the client default from `./headroom.db` to the temp directory, but
didn't update this field. It was never read for storage, so it never
mattered until now.
- Nothing else reads `HeadroomConfig.store_url` (`grep -rn "\.store_url"
headroom`: only `client.py` and `testing/harness.py`, which sets it
explicitly).
- `tests/test_config.py`: the default-values test expects `store_url is
None`.
- `tests/test_client_store_url.py`, 5 tests (each one isolates the
system temp directory):
- `config.store_url` is where storage opens and the file is created, and
`validate_setup()` reports the same URL.
- `config.store_url` wins over the `store_url` argument (per the
docstring).
- A config with no `store_url` uses the `store_url` argument, and the
config is left unchanged.
- One default config shared by two clients with different `store_url=`
arguments: each client stores in its own database, and the shared config
is left unchanged.
- A default `HeadroomConfig()` with no argument stores in the temp
directory and creates nothing in the working directory.

**Resulting precedence:** `config.store_url` > `store_url=` argument >
`<system temp>/headroom.db`. Callers who pass only `store_url=` (no
config), or no storage settings at all, see no change.

## Testing

- [x] Unit tests pass (`pytest`): the new file plus every test file that
references `HeadroomClient`, `store_url` or `HeadroomConfig(` (17 files,
run one by one), not the full suite. The Strands and Agno test files
skip entirely in my env because those extras aren't installed
- [x] Linting passes (`ruff check .`): on the changed files
- [ ] Type checking passes (`mypy headroom`): ran `mypy
headroom/client.py headroom/config.py` only
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

New tests without the fix (`client.py` and `config.py` reverted to
`main`):

```text
$ uv run pytest tests/test_client_store_url.py -q
FAILED tests/test_client_store_url.py::test_config_store_url_is_where_metrics_are_stored
FAILED tests/test_client_store_url.py::test_config_store_url_takes_precedence_over_the_argument
FAILED tests/test_client_store_url.py::test_argument_is_used_when_config_has_no_store_url
FAILED tests/test_client_store_url.py::test_shared_config_does_not_leak_one_clients_url_into_another
========================= 4 failed, 1 passed in 1.72s ==========================
```

The first two fail on the storage path (`.../system-temp/headroom.db`
and `.../from-argument.db` instead of the configured database). The last
two store in the right place on `main` but fail on the config-default
assertion, since `main`'s default is `"sqlite:///headroom.db"`. The one
pass is `test_default_config_stores_in_temp_dir_not_cwd`, which is
expected: it guards that default callers keep the temp-directory
location, so it holds before and after.

The shared-config test against this PR's first commit (f96bc8e2, before
the write-back was removed) fails on the leak itself:

```text
E   AssertionError: assert PosixPath('/private/var/.../a.db') == PosixPath('/private/var/.../b.db')
FAILED tests/test_client_store_url.py::test_shared_config_does_not_leak_one_clients_url_into_another
```

With the fix:

```text
$ uv run pytest tests/test_client_store_url.py tests/test_config.py -q
============================== 41 passed in 2.20s ==============================

$ for f in <17 test files referencing HeadroomClient / store_url / HeadroomConfig(>; do uv run pytest "$f" -q; done
220 passed, 0 failed (summed across files; 9 skipped in test_chat_model.py)

$ uv run ruff check headroom/client.py headroom/config.py tests/test_client_store_url.py tests/test_config.py
All checks passed!
$ uv run ruff format --check (same files)
4 files already formatted
$ uv run mypy headroom/client.py headroom/config.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS 27.0.1 (arm64), Python 3.13.15, `uv sync --extra
dev --extra proxy`, openai 2.41.0, `HeadroomClient` against a local stub
OpenAI upstream.
- Exact command / steps: wrapped the real OpenAI SDK client in
`HeadroomClient(original_client=OpenAI(base_url=<stub>),
provider=OpenAIProvider(),
config=HeadroomConfig(store_url="sqlite:///<work>/mine.db"))`, sent one
`chat.completions.create` request, then counted rows in the `requests`
table of each candidate database, on `main` and on this branch.
- Stub upstream: a small `ThreadingHTTPServer` returning a canned chat
completion.
- The script pointed `tempfile.tempdir` at `<work>/system-temp` and
`chdir`-ed into `<work>/cwd`, so all three possible locations are
isolated and checkable.
- Observed result: on `main` the request's metrics row landed in the
temp-directory database while `validate_setup()` reported `mine.db`; on
this branch it lands in `mine.db`. Nothing is written to the working
directory in either case. Separately, two clients built from one shared
default config with `store_url=` `a.db` and `b.db` store in `a.db` and
`b.db` respectively, and the shared config's `store_url` stays `None`.

  | | `main` | this branch |
  |---|---|---|
| `validate_setup()["storage"]["url"]` | `sqlite:///<work>/mine.db` |
`sqlite:///<work>/mine.db` |
  | rows in `mine.db` | file never created | 1 |
  | rows in `<temp>/headroom.db` | 1 | file never created |
  | `headroom.db` in working directory | no | no |

- Not tested: a live upstream provider; `jsonl://` and plugin storage
URLs (they go through the same `create_storage` call, only the URL
passed in changed).

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: no for callers who don't set
`HeadroomConfig.store_url`: the default stays `<system
temp>/headroom.db`, and a `store_url=` argument is honored as before.
Callers who set `config.store_url` now get metrics stored there, which
is what they configured.
- Kill switch / disable path: N/A
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A: the
docs already describe the temp-directory default and `store_url` as the
way to change it)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

`HeadroomConfig().store_url` now reads `None` instead of
`"sqlite:///headroom.db"`, and a client no longer fills it in on a
caller-supplied config. That old value never controlled where anything
was stored, so code that read it was already getting a path nothing
wrote to.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-10-28 17:57:20 -05:00
Radhakrishnan Pachyappan fef99cc5b4 feat(compress): add per-message compression diagnostics (#3058)
## Description

Adds `CompressConfig.diagnostics` (bool, default `False`) and a
companion `HEADROOM_DIAGNOSTICS=1` environment variable that enable
per-message decision logging on every call to `compress()`.

When active, `CompressResult.diagnostics` is a list of `MessageDecision`
objects — one per input message — each describing:

| Field | Description |
|---|---|
| `message_index` | Zero-based position in the original messages list |
| `role` | `"user"` / `"assistant"` / `"tool"` / `"system"` / … |
| `tokens_before` | Token count before the ContentRouter acted on this
message |
| `tokens_after` | Token count after (equals `tokens_before` for
protected/passthrough) |
| `action` | What was done: `"compressed:kompress:0.72"`,
`"protected:user_message"`, `"passthrough:frozen"`,
`"passthrough:small"`, etc. |

Closes #2855.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`headroom/config.py`** — add `MessageDecision` dataclass; add
`message_decisions: list[MessageDecision]` field to `TransformResult`
- **`headroom/compress.py`** — add `CompressConfig.diagnostics`; add
`CompressResult.diagnostics`; derive `collect_diagnostics` from config +
env var; pass to pipeline; collect from `TransformResult`
- **`headroom/transforms/pipeline.py`** — accumulate `message_decisions`
from each transform result
- **`headroom/transforms/content_router.py`** — instrument every
decision branch in `apply()` with `_diag[i] = "action:..."` strings when
`collect_diagnostics=True`; build `MessageDecision` list after Pass 3
(zero overhead when flag is off)
- **`headroom/__init__.py`** — export `MessageDecision` in `__all__` and
lazy-import map
- **`tests/test_compress_diagnostics.py`** — 14 tests covering the full
`compress() <-> pipeline` contract via a mock pipeline

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_compress_diagnostics.py -v
========================= test session starts ==========================
collected 14 items

tests/test_compress_diagnostics.py::test_message_decision_dataclass PASSED
tests/test_compress_diagnostics.py::test_message_decision_importable_from_headroom PASSED
tests/test_compress_diagnostics.py::test_transform_result_has_message_decisions_field PASSED
tests/test_compress_diagnostics.py::test_transform_result_message_decisions_populated PASSED
tests/test_compress_diagnostics.py::test_compress_config_diagnostics_defaults_false PASSED
tests/test_compress_diagnostics.py::test_compress_config_diagnostics_can_be_set PASSED
tests/test_compress_diagnostics.py::test_compress_result_diagnostics_none_by_default PASSED
tests/test_compress_diagnostics.py::test_compress_passes_collect_diagnostics_when_config_true PASSED
tests/test_compress_diagnostics.py::test_compress_result_diagnostics_list_when_config_true PASSED
tests/test_compress_diagnostics.py::test_compress_result_diagnostics_contains_decisions PASSED
tests/test_compress_diagnostics.py::test_env_var_enables_diagnostics PASSED
tests/test_compress_diagnostics.py::test_env_var_absent_keeps_diagnostics_none PASSED
tests/test_compress_diagnostics.py::test_env_var_overrides_config_false PASSED
tests/test_compress_diagnostics.py::test_no_diagnostics_when_optimize_false PASSED
========================== 14 passed in 0.42s ==========================

$ ruff check .
All checks passed!
```

## Real Behavior Proof

- Environment: Python 3.12, headroom dev install
- Exact command / steps: `python -m pytest
tests/test_compress_diagnostics.py -v`
- Observed result: 14 tests pass; diagnostics list is populated when
`CompressConfig(diagnostics=True)` or `HEADROOM_DIAGNOSTICS=1`
- Not tested: ContentRouter integration tests (require compiled
`headroom._core` extension — run in CI)

## Runtime Rollout Safety

- Rollout-managed feature(s): None — diagnostics are opt-in via
`CompressConfig.diagnostics=True` or `HEADROOM_DIAGNOSTICS=1`
- Minimum rollout channel: N/A (additive, opt-in only)
- Stable/default behavior changed: No — `CompressConfig.diagnostics`
defaults to `False`; `CompressResult.diagnostics` is `None` unless
explicitly requested
- Kill switch / disable path: Simply omit `diagnostics=True` / unset
`HEADROOM_DIAGNOSTICS`
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert this PR; no schema or data migration involved

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — no UI changes.

## Additional Notes

The `collect_diagnostics` flag incurs zero overhead in the hot path when
not set: the `_diag` dict in `ContentRouter.apply()` is never allocated
and no `MessageDecision` objects are constructed.

Signed-off-by: Radha Krishnan P <gingeekrishna@gmail.com>

---------

Signed-off-by: Radhakrishnan Pachyappan <radhakrishnan.p@op.tech>
Signed-off-by: Radhakrishnan P <gingeekrishna@gmail.com>
2026-10-01 00:04:47 -05:00
Focused Instability f9285766dd feat: attribute reread waste to over-compression via marker check (#901)
## Description

Fixes #899. The `reread` signal (#853/#854) counts re-served tool
results but cannot answer the question that motivated it: **did Headroom
cause the re-read?** A re-read after an intact first serve is agent
behavior; a re-read after Headroom markerized the first serve is
over-compression cost. This PR splits the signal so the actionable part
is visible.

Request-local, no store lookups: the client resends full history each
turn and the pipeline recompresses it deterministically, so the current
request already holds the evidence. `TransformPipeline.apply` passes
`current_messages` into `parse_messages(compressed_messages=...)`. For
each counted reread group, if the transformed copy of the **first
serve** carries a CCR retrieval marker and its original text is gone,
the group's counted repeats go into `reread_compressed_tokens`. Lossless
reshaping (no marker) is deliberately not attributed.

Closes #899.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)

## Changes Made

- `parser.py`: `parse_messages` gains an optional `compressed_messages`
param; the content-hash reread loop accumulates per-group
`counted_tokens` and attributes them to `reread_compressed_tokens` when
the first serve's transformed copy carries a CCR marker
(`CCR_RETRIEVAL_MARKER_RE`, kept local to avoid a transforms import
cycle).
- `transforms/pipeline.py`: pass `current_messages` (post-transform
copy) into the existing waste-detection `parse_messages` call.
- `config.py`: new `reread_compressed_tokens` WasteSignals field;
`dashboard.html` + `reporting/generator.py` surface it.
- Tests: `tests/test_reread_attribution.py` + WasteSignals contract
update.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New tests added for new functionality
- [x] New and existing unit tests pass locally with my changes

### Test Output

```text
$ pytest tests/test_reread_attribution.py tests/test_parser.py tests/test_gemini_function_response_waste.py tests/test_codex_responses_waste_signals.py -q
122 passed in 1.50s

$ pytest tests/ -k "waste or pipeline or reporting or config or reread" -q
348 passed, 33 skipped, 6010 deselected
# (1 unrelated env-dependent failure: test_proxy_gemini_native_integration::test_generation_config — 404, reproduces on main without these changes; needs a Gemini key locally)

$ ruff check headroom/parser.py headroom/transforms/pipeline.py
All checks passed!
```

## Real Behavior Proof

- Environment: local macOS, repo .venv, Python 3.11.9
- Exact command / steps: rebased onto current main to resolve conflicts
with #909 (merged), then ran the reread + parser + waste suites above
- Observed result: a reread whose first serve is markerized attributes
to `reread_compressed_tokens`; an intact first serve and a lossless
(no-marker) reshape do not. #909's re-issued-call detection (same call,
different bytes) continues to count and dedup correctly alongside it —
all 122 targeted tests pass.
- Not tested: live proxy traffic; the one gemini-native route test above
(environmental 404, not introduced here).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Additional Notes

**Rebased onto current main after #909 merged.** #909 added a
re-issued-call reread pass *after* the original content-hash loop this
PR modifies — the conflict was textual/adjacent, not a re-architecture.
Resolution preserves #909's `counted_results` dedup contract and leaves
its new pass unchanged; #901's attribution stays scoped to the
content-hash groups it was reviewed against (attributing #909's call-key
pass too would be a separate follow-up). The diff differs from the prior
approval only by this reshape — worth a quick re-glance.
2026-06-13 10:43:35 -05:00
Focused InstabilityandAsh Rhodes 5f1d88ad27 feat: detect re-served tool results as over-compression waste signal (#854)
Closes #853

## What

Adds a `reread` waste signal: identical `tool_result` content appearing
at more than one message position means the agent re-fetched something
already in context — the dominant failure signature of over-compression
(Manus context-engineering; JetBrains "Complexity Trap",
arXiv:2508.21433). Per-request savings can't see this cost; this signal
makes it visible.

- `WasteSignals.reread_tokens` — new field, in `total()`, exported as
`"reread"` in `to_dict()`.
- `parse_messages()` groups `tool_result` blocks by their **existing**
`content_hash` and counts every repeat beyond the first serve. No new
hashing or tokenization; one O(blocks) dict pass.
- `REREAD_MIN_TOKENS = 50` guard: short outputs ("ok", empty diffs)
legitimately repeat and are skipped. Duplicates within a single message
(same `source_index`) are not counted.
- Works across all formats the parser already normalizes to
`tool_result` blocks: OpenAI `role=tool`, Anthropic `tool_result`,
Strands/Bedrock `toolResult` (#813/#815).
- Flows through existing generic plumbing with zero handler changes:
pipeline → `RequestOutcome.waste_signals` → Prometheus
`headroom_waste_signal_tokens_total{signal="reread"}` → dashboard "Waste
Detected" panel. Dashboard gains label/color entries for the new key.

## Tests

7 new tests in `tests/test_parser.py::TestRereadDetection` (red before,
green after): OpenAI + Anthropic format detection, repeat-counting
semantics (first serve free), single-occurrence, short-duplicate guard,
same-message guard, `total()`/`to_dict()` participation. Updated 2
exact-shape assertions in `tests/test_config.py`.

Local runs: `tests/test_parser.py` (72 passed), `tests/test_config.py` +
outcome/reporting/observability/storage/proxy-hooks suites (190 passed),
`tests/test_canonical_pipeline.py` +
`tests/test_proxy_pipeline_lifecycle.py` (11 passed). `ruff check` +
`ruff format --check` clean.

## Real behavior proof

**Setup:** macOS (Darwin 25.5), Python 3.11.9, this branch, real proxy
server (`python -m headroom.proxy.server --port 18970
--anthropic-api-url http://127.0.0.1:18971`) with a local mock Anthropic
upstream returning a canned `/v1/messages` response (no real key
needed).

**Steps:** POSTed an Anthropic-format conversation to the live proxy:
agent fetches a 14 KB JSON log array via `get_logs` tool, then fetches
the identical content again under a different `tool_use_id` (the
re-read).

**Observed result** — `curl http://127.0.0.1:18970/metrics` after the
request:

```
# HELP headroom_waste_signal_tokens_total Tokens attributed to detected waste signals
# TYPE headroom_waste_signal_tokens_total counter
headroom_waste_signal_tokens_total{signal="json_bloat"} 9858
headroom_waste_signal_tokens_total{signal="reread"} 4935
```

`reread` = 4935 tokens, exactly the second serve of the ~4.9k-token tool
result (json_bloat counts both occurrences ≈ 2×). `/stats` shows the
same: `"waste_signals": {"json_bloat": 9858, "reread": 4935, ...}` —
which is what the dashboard panel renders.

Also verified the negative path live: a conversation whose tool results
contain non-compressible plain code text produced no waste-signal
entries (the pipeline only attributes waste when compression actually
engaged, unchanged behavior).

**Not tested:** Gemini `functionResponse` path (parser doesn't produce
`tool_result` blocks for it — pre-existing gap tracked in #819);
dashboard rendering only verified via the `/stats` payload the panel
binds to, not a browser screenshot.

## Out of scope (per #853)

Tool-call argument matching, compression-marker attribution,
tokens-per-task metric, cache hit-rate panel.

---------

Co-authored-by: Ash Rhodes <ashley.rhodes@king.com>
2026-06-11 13:07:04 -05:00
chopratejas 967b0db439 fix: B1 — retire ICM, RollingWindow, scoring, relevance + dependents
Phase B step 1 of the live-zone-only realignment. Removes ~10K LOC of
"drop messages from history" machinery that became unreachable after
PR-A1 made `/v1/messages` a passthrough on the proxy. Live-zone-only
compression (PR-B2..B7) operates on content blocks within messages;
message-list mutation no longer happens in the pipeline.

Python deletes:
- headroom/transforms/intelligent_context.py (1077 LOC)
- headroom/transforms/rolling_window.py (395 LOC)
- headroom/transforms/progressive_summarizer.py (508 LOC)
- headroom/transforms/scoring.py (459 LOC)
- headroom/transforms/tool_crusher.py (338 LOC)
- 5 corresponding tests/test_transforms/* and tests/test_proxy_intelligent_context.py

Rust deletes:
- crates/headroom-core/src/context/* (manager, config, workspace,
  candidate, ccr_drop, strategy/, mod) + safety.rs replaced
- crates/headroom-core/src/scoring/* (mod, score, scorer, traits, weights)
- MessageScorerComparator from crates/headroom-parity (PR #338/#343
  becomes deletable; sunk cost stays sunk)
- 13 message_scorer fixtures + record_message_scorer.py

Rust adds (move + rewrite):
- crates/headroom-core/src/transforms/safety.rs — `tool_pair_indices`
  preserves the OpenAI/Anthropic tool_use ↔ tool_result pairing rule
  the live-zone dispatcher (PR-B2) needs. No IcmConfig dependency.

Surface refactors:
- HeadroomConfig: drop `tool_crusher`, `rolling_window`,
  `intelligent_context` fields; hoist `output_buffer_tokens` to top
  level (used by client.py).
- ProxyConfig: drop `intelligent_context*` fields.
- `headroom wrap` proxy server: retire IntelligentContextManager
  and RollingWindow imports + branch; pipeline is CacheAligner →
  ContentRouter (smart_routing) or CacheAligner → SmartCrusher
  (legacy).
- CLI: drop `--no-intelligent-context`, `--no-intelligent-scoring`,
  `--no-compress-first` flags.
- LangChain memory integration: rename `_apply_rolling_window` →
  `_apply_compression`, drop RollingWindowConfig dep. Threshold is
  now advisory — B6 will rework the contract.
- TransformPipeline.create_pipeline now takes only cache_aligner_config.
- headroom/__init__.py + headroom/transforms/__init__.py: strip
  exports of deleted symbols.

Bug fixes uncovered by full pytest sweep:
- providers/copilot/wrap.py: `environ or os.environ` collapsed
  empty-dict to falsy → callers passing `environ={}` accidentally
  pulled from os.environ. Use `environ if environ is not None else
  os.environ`.

Test correctness fixes:
- _DummyAnthropicHandler._retry_request gains **_kwargs to match
  the real handler signature post-A8.
- test_ws_http_fallback extracts JSON from `content=` (post-A3
  byte-faithful) rather than the obsolete `json=` kwarg.
- test_ccr_response_handler_extra fixture joins SSE events with
  `\n\n` per spec (post-A8 byte-buffer parser requirement).
- test_proxy_responses_phase_preservation: capture via direct
  handler attached to the named logger, so the assertion is
  order-independent (proxy `_setup_file_logging` flips
  `headroom.propagate=False` once any earlier test triggers it).
- conftest.py autouse fixture resets `headroom.propagate=True`
  before each test as a defensive measure for the same pollution.
- test_wrap_copilot_translated_backend_still_requires_byok:
  monkeypatch.delenv every provider key so the BYOK error
  actually fires.
- test_native_installers: skip when system bash < 4.3 (macOS ships 3.2).
- TestGeminiEmbedContent / TestGeminiBatchEmbedContents:
  pytest.mark.skip — proxy currently has no :embedContent route;
  feature gap, not regression.

Acceptance:
- cargo build --workspace + cargo clippy + cargo fmt --check: green.
- cargo test --workspace --exclude headroom-py: 777 passed.
- pytest: 4892 passed, 240 skipped, 0 failed.
- git grep returns only intentional comments referencing the deletion.

Per-PR-B1 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 12:23:17 -07:00
Tejas Chopra 08d81f2e2c fix: dashboard metrics, TTFB tracking, eager LLMLingua loading, and multi-provider consistency
Dashboard was showing wildly incorrect metrics (99.5% savings, 3ms overhead)
due to using Anthropic API's non-cached input_tokens instead of optimized_tokens,
and dividing overhead by total request count instead of optimized-only count.

Key fixes:
- Use optimized_tokens (what we sent) for dashboard aggregation, not API's
  input_tokens which excludes cached portion
- Track overhead_count separately from latency_count for correct averages
- Add TTFB (time to first byte) measurement, replace full stream latency in UI
- Eager-load LLMLingua model at proxy startup (eliminates 5.9s first-request delay)
- Simplify CostTracker to token-based accounting with counterfactual cost display
- Add two-tier compression cache to ContentRouter (skip set + result cache)
- Fix compression pinning to detect both CCR and ReadLifecycle markers
- Clamp tokens_saved to max(0, ...) across all provider paths
- Add per-transform timing instrumentation to pipeline
- Guard against over-aggressive code compression (<5% ratio)
- Fix ReadLifecycle partial read supersede logic (_read_covers range check)
- Disable CacheAligner and compress_superseded by default
- Fix all pre-existing mypy errors (CompressionCache return types)
- Fix test mocks to accept **kwargs for cache token parameters
2026-03-07 23:33:45 -08:00
chopratejas e4a41faa33 Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file
- Fix F401: Add noqa for availability check imports
- Fix F402: Rename loop variables shadowing imports
- Fix E722: Replace bare except with except Exception
- Fix B904: Add exception chaining (from e)
- Fix F811: Remove duplicate imports
- Fix B027: Add noqa for empty close() method
- Fix E741: Rename ambiguous variable l -> label
- Fix I001: Import sorting issues
- Apply ruff format to all 106 files

All 902 tests pass.
2026-01-10 15:33:44 -08:00
chopratejas c1feb60595 feat: Add CCR architecture, TOIN telemetry, and DevEx improvements
## Core Features

### Compress-Cache-Retrieve (CCR) Architecture
- Implement reversible compression with automatic retrieval support
- Add CompressionStore for caching original content with TTL-based eviction
- Add CompressionFeedback for learning from retrieval patterns
- Implement tool injection for LLM retrieval capability
- Add MCP server support for CCR operations
- Track retrieval rates to dynamically adjust compression aggressiveness

### Tool Output Intelligence Network (TOIN)
- Implement cross-session pattern learning for tool compression
- Add ToolSignature for structural hashing of tool outputs
- Track compression success rates per strategy (top_n, sample, truncate, etc.)
- Implement privacy-preserving telemetry with SHA256 hashing
- Add persistent storage with JSON file backend
- Support network-effect learning across tool types

### SmartCrusher Enhancements
- Add crushability analysis with variance/uniqueness detection
- Implement statistical anomaly detection for outlier preservation
- Add relevance-based item prioritization using BM25 scoring
- Support multiple compression strategies with quality retention
- Add change point detection for time-series data
- Implement constant factoring for homogeneous datasets

## Developer Experience Improvements

### Exception Hierarchy
- Add HeadroomError base class for all custom exceptions
- Add specific exceptions: ConfigurationError, ProviderError,
  StorageError, CompressionError, TokenizationError, CacheError,
  ValidationError, TransformError

### Client Enhancements
- Add validate_setup() for configuration verification
- Add get_stats() for in-memory session metrics without DB query
- Track session statistics (requests, tokens saved, cache hits)

### Logging Infrastructure
- Add structured logging to TransformPipeline with token savings
- Add logging to RollingWindow for dropped message tracking
- Add logging to ToolCrusher for compression events
- Add logging to CacheAligner for cache hit/miss detection
- Add logging to SmartCrusher for strategy selection

## Bug Fixes (from deep analysis)

### Critical Fixes
- Fix eviction heap memory leak with stale entry tracking
- Fix hash collision detection in compression store
- Fix strategy truncation desync in TOIN
- Fix non-deterministic set truncation with sorted iteration
- Fix race conditions in lazy initialization with proper locking
- Fix user count double-counting in TOIN metrics

### High Priority Fixes
- Fix unbounded strategy_success_rates growth with LRU eviction
- Fix mutable pattern references with defensive copying
- Fix lock held during file I/O with copy-then-write pattern
- Fix state divergence on eviction with success event recording
- Fix TOIN skip check order for CPU efficiency
- Fix preserve_fields type mismatch (set vs list)
- Fix prioritize_indices exceeding max_items limit
- Fix instance ID collision risk (32-bit to 64-bit hash)

## Testing

- Add comprehensive test suites for CCR, TOIN, and telemetry
- Add crushability detection tests
- Add quality retention tests for compression
- Add integration tests for cross-component data flow
- All 902 tests passing
2026-01-10 10:12:13 -08:00
chopratejas 9c7d4512d6 Initial commit: Headroom SDK - LLM context optimization toolkit
A comprehensive SDK for optimizing LLM context windows, reducing token
usage while preserving critical information for AI agents.

Core Features:
- SmartCrusher: Statistical compression of tool outputs (70-85% reduction)
- CacheAligner: Prefix optimization for prompt cache hits
- RollingWindow: Intelligent context window management
- BM25/Hybrid relevance scoring for smart item selection

Integrations:
- OpenAI and Anthropic provider support
- LangChain integration (ChatModel, Callbacks, Runnable)
- MCP (Model Context Protocol) integration for tool compression

Test Coverage:
- 372 tests passing across all modules
- 35 performance benchmarks
- Real-world agent evaluations with 88% token savings

Key Components:
- headroom/transforms/: Core compression transforms
- headroom/providers/: OpenAI and Anthropic support
- headroom/integrations/: LangChain and MCP integrations
- headroom/relevance/: BM25 and hybrid scoring
- headroom/pricing/: Model pricing registry
- benchmarks/: Performance benchmark suite
- examples/: Usage examples and demos
2026-01-06 23:16:58 -08:00