mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
feat/https-compose-caddy
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1d112cd36c |
fix(proxy): show turn-hook savings in /stats extension_savings, incl. Codex WS (#4233)
## Description
### The test we ran
A Daytona sandbox running Headroom Enterprise 0.4.3 (core 0.39.1), with
Codex CLI 0.162.1 and Claude Code connected to real GitHub, Context7 and
Fetch MCP servers, working through the workshop tasks.
### What we found
- The tool-search plugin's turn hook rewrote the Codex `/v1/responses`
input and saved about **8K tokens per request**. `run_request_hooks`
booked that saving through `ctx.record_savings(hook.savings_source,
tokens=...)`.
- After many such requests, `GET /stats` still showed
**`extension_savings: {}`**.
There were two causes:
1. **`extension_savings` has no writer in core.** The only thing that
fills it is `PrometheusMetrics.record_extension_savings`, and core never
calls it. Attribution rows from `ctx.record_savings` /
`record_scope_savings` go to `savings_by_source`, which `/stats` shows
as `savings.by_source`.
2. **The Codex WebSocket path drops attribution entirely.** Both WS
compression calls (first frame and later frames) called
`_compress_openai_responses_payload_in_executor` without `savings_tags`.
So `TurnContext.tags` was a throwaway `{}`, and the hook's row (along
with the built-in tool-search deferral row) never reached
`savings.by_source` either. Codex uses the WS transport by default. The
HTTP `/v1/responses` path already passed `savings_tags`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- **`/stats.extension_savings`** now reports measured savings per
source: the **realized** attribution rows, plus
`record_extension_savings` totals. If a source also calls
`record_extension_savings` (`headroom-hdr` reports through both APIs on
every request), it shows the larger of its two totals rather than their
sum, so hdr is not double-counted. Nothing ties an attribution row to a
`record_extension_savings` call, so a plugin that used the two APIs on
different turns would be under-reported, never over-reported.
Unrealized/estimated rows (tool-search deferral, `realized=False`) stay
out and remain visible in `savings.by_source`.
- **Codex WS:** each frame compresses into its own `savings_tags` dict.
Its rows are merged into the session's pending list on the event loop,
and only when that frame was actually rewritten and sent. Each per-turn
`RequestOutcome` carries the pending rows and then clears them. A turn
whose `response.completed` has no input usage leaves its rows pending,
the same gate as its headline `saved_delta`
(`_deferrable_savings_delta`). Rows no turn claimed go out with the
session-end residual outcome, and only if the residual has input usage.
A fresh dict per frame matters because a timed-out compression thread
keeps running (`_run_compression_in_executor` can't cancel it), and it
must not book rows for a frame that went out uncompressed.
**Realized vs estimated:** `savings.by_source` already keeps one row per
`(source, realized)`, so realized and unrealized savings are shown
separately. `extension_savings` now carries only the realized ones.
**Review fix (@JerrettDavis, bounded WS pending list):** the
session-wide pending list grew one row per rewritten frame, while
`savings_attribution.from_tags` read only the first `MAX_SOURCES` (32)
rows. After 33 frames that each booked a 100-token `tool_search` row,
the handoff cleared all 33 rows but booked only 3,200 of the 3,300
tokens. `savings_attribution.accumulate(rows, items)` replaces the
`extend` at both WS frame paths.
**Bounded rows, nothing dropped (follow-up to @JerrettDavis's two
reviews):** the 32-row cap that silently dropped rows is gone from
`record_savings`, `from_tags` and `accumulate`. Storage is bounded by
coalescing plus one overflow row:
- every writer goes through the one `accumulate` helper: the per-request
ledger (`record_savings`, so every handler) and the WS pending list.
Rows with the same `(source, realized, estimated)` key are summed into
one row (`tokens`, `usd`, `events`). `details` are kept only while every
summed row has the same `details`.
- **cardinality bound:** a ledger keeps at most `MAX_NAMED_ROWS` (256)
named rows. A new source past that bound is summed into an `other` row
with the same `(realized, estimated)`, and its `details` are dropped.
Tokens, dollars and events are conserved: nothing is dropped, only the
source name. `tool_search` never folds, because
`tool_schema_savings_policy` finds the deferral's row by name to lower
it when hooks un-defer tools.
- **finite sums:** a row's `usd` saturates at ±`MAX_ROW_USD` (1e15) and
its `tokens` stay within 0..`MAX_ROW_TOKENS` (10^15). A row that was
clipped, or summed from one that was, carries `saturated: true`. Every
total stays finite, so the Prometheus exposition and
`json.dumps(allow_nan=False)` never see `inf`/`nan`.
- each row carries `events`: how many recordings it stands for (1 on a
fresh row, summed on merge). `PrometheusMetrics.record_request`
(`headroom_savings_attribution_events_total`), the OTel
`headroom.savings.attribution.events` counter and the perf analyzer's
"Savings by Source" add that count instead of 1 per row, so 33 coalesced
frames are 33 events, not 1 (Devin's finding). A row without `events`
(older PERF logs, rows built by hand) counts as 1.
- a row that a reader edited in place (`reconcile_deferred_tokens`, the
tool-search plugin) is summed defensively, so a junk value counts as 0
instead of raising inside the handler. The core deferral and the native
deferral hook both book `tool_search` as estimated, so they now share
one row, and the reconcile debits that row; totals are unchanged.
- `MAX_SOURCES` is no longer used. No importer was found outside this
repo's own test, but `savings_attribution` is imported by plugins, so
the name is kept behind a module `__getattr__` that raises a
`DeprecationWarning`.
- **Scope of the bound:** it is per ledger (one request, or one WS
session's pending list). The process-wide
`PrometheusMetrics.savings_by_source` table has no source cap yet; that
is a separate PR.
### Before / after
| | origin/main | this PR |
|---|---|---|
| A turn hook books 8,000 tokens on 2 requests →
`/stats.extension_savings` | `{}` | `{"tool_search": 16000}` |
| An hdr-style plugin calls both APIs (300 tokens × 2) | `{"hdr": 600}`
| `{"hdr": 600}` (not 1200) |
| A Codex WS turn with a folding hook → attribution rows on the turn
outcome | `[]` | `[("ws_fold", realized=True)]` |
| 33 WS frames, each booking `tool_search` 100 tokens, then one billed
turn → booked `tool_search` tokens (helper-level, the reviewer's
scenario) | 3,200 of 3,300, 33 rows stored | 3,300, 1 row stored |
| Real WS handler, 33 frames each with a folding hook (`ws_fold` row)
plus a 100-token hook row → non-`ws_fold` tokens booked | 1,600 of 3,300
(66 rows stored) | 3,300 |
| Same, repeated source → rows stored / hook events in
`headroom_savings_attribution_events_total` |
|
||
|
|
02eb90f243 |
feat(metrics): record per-extension token savings (#2371)
## What Adds `PrometheusMetrics.record_extension_savings(key, saved)` so proxy extensions can report the tokens they save, and surfaces the per-extension totals in the `/stats` payload. ## Why Proxy extensions that perform their own token reduction currently have no supported way to report their savings to the metrics object — there is no method for it, so that telemetry is silently dropped. This adds the recording method and exposes the aggregate alongside the existing per-strategy compression breakdown. ## How - New `extension_savings: dict[str, int]` counter on `PrometheusMetrics`, populated lazily per extension-supplied `key` (no hardcoded list of extensions). - `record_extension_savings(key, saved)` accumulates positive savings per key, mirroring how `record_compression` aggregates `tokens_saved_by_strategy` (lock-free `defaultdict(int)`, atomic under the GIL for these key types); non-positive values are ignored. - Cleared in `reset_runtime()` with the other in-memory counters. - Surfaced in `/stats` as `extension_savings`, next to `compressions_by_strategy` / `tokens_saved_by_strategy`. No new Prometheus series. ## Behavior change None to existing metrics. ## Testing - Two focused tests in `tests/test_compression_observability.py` (per-key accumulation incl. zero/negative ignored; surfaced in `/stats` via `create_app`) → 2 passed (13 in file). - `ruff check` / `ruff format` → clean; `mypy` → clean on changed source. |
||
|
|
53be64ca12 |
chore(telemetry): remove Supabase anonymous beacon; fix contact domain to headroomlabs.ai (#1526)
## Description Removes the anonymous-telemetry **beacon** — the only external, third-party data flow Headroom ever initiated. When telemetry was opted in, it POSTed aggregate `/stats` to a hardcoded **Supabase** REST endpoint (with an embedded anon API key in the source). For enterprise/on-prem deployments this is exactly the kind of vendor-controlled data egress a security review flags, so it's gone entirely — **zero "Supabase" references remain in the codebase.** What stays (by design): the **local** telemetry collector + the `HEADROOM_TELEMETRY` opt-in (it only feeds `/stats` and `/v1/telemetry` — nothing leaves the process), **OpenTelemetry export** (`HEADROOM_OTEL_METRICS_*`, so operators send operational metrics to *their own* collector), and the license usage reporter (your own domain, license-key-gated). Also fixes the contact domain: `headroom.dev` → `headroomlabs.ai` everywhere. Closes # (no tracking issue) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) > Non-breaking: `HEADROOM_TELEMETRY` is still accepted (now gates local collection only). The only behavior change is that no telemetry is ever sent externally. ## Changes Made - **Deleted the Supabase beacon**: `TelemetryBeacon` class, `_SUPABASE_URL`/`_SUPABASE_KEY`/`_TABLE`/`_ENDPOINT`, the JSONB projection helper, the proxy-lifespan beacon wiring, the `SUPABASE_` install env passthrough, and `tests/test_strategy_stats_supabase.py`. - **Kept** the local opt-in predicate (`is_telemetry_enabled` etc.) in `beacon.py` — still used by the local collector + CLI — reworded to "local only". - **Retained** the single-worker-owner file lock (the cc-switch reconciler depends on it); updated its comments to drop the beacon framing. - `/stats` `anon_telemetry_shipping` is now always `False` (nothing ships externally); startup log reworded to "Local telemetry". - Reworded remaining "Supabase" comments in `collector.py`, `context.py`, `prometheus_metrics.py`, and two test docstrings. - Contact domain: `security@headroom.dev` → `security@headroomlabs.ai`, `conduct@headroom.dev` → `conduct@headroomlabs.ai`, FUNDING.yml sponsor URL. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ grep -rniI "supabase" --include=*.py --include=*.md --include=*.mdx . # (excl .venv/sbom) >>> ZERO Supabase references $ grep -rniI "headroom.dev" . >>> ZERO headroom.dev references $ ruff check <changed files> -> All checks passed! $ ruff format --check <changed files> -> 10 files already formatted $ mypy <changed telemetry files> -> Success: no issues found $ pytest tests/test_telemetry.py tests/test_telemetry_warning.py \ tests/test_proxy_telemetry_env.py tests/test_compression_observability.py \ tests/test_paths.py tests/test_paths_backward_compat.py -q ============================= 173 passed in 6.67s ============================== ``` ## Real Behavior Proof - **Environment:** macOS, Python 3.12 (`.venv`). - **Exact command / steps:** repo-wide grep for `supabase`/`headroom.dev`; `create_app(...)` driven through a full `TestClient` lifespan (startup + shutdown) in `test_proxy_telemetry_env.py`; `/stats` exercised in `test_telemetry_warning.py`. - **Observed result:** zero `supabase`/`headroom.dev` strings remain; the proxy starts and shuts down cleanly with the beacon removed (the worker-owner lock + reconciler still elect a single owner); `/stats.anon_telemetry_shipping` is `False` even with `HEADROOM_TELEMETRY=on`; local collector + OTEL paths unchanged. - **Not tested:** no live network call was ever made (the point — the external POST is gone). OTEL export and the license reporter were not exercised (unchanged by this PR). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The license usage reporter (`reporter.py` → `app.headroomlabs.ai`) is intentionally **kept** — it's license-key-gated (dormant for unlicensed/OSS deployments) and goes to your own domain, not a third party. - Docs/CHANGELOG left unchecked: a couple of docs mention the telemetry beacon and may want a follow-up note that it now collects locally only; happy to add. |
||
|
|
e9cae0131b |
fix: expose compression latency bottlenecks
Add Codex WS unit-level timing and bounded parallel compression, clarify context-tool session savings, and avoid costly diff/log fallbacks to Kompress. |
||
|
|
967b0db439 |
fix: B1 — retire ICM, RollingWindow, scoring, relevance + dependents
Phase B step 1 of the live-zone-only realignment. Removes ~10K LOC of
"drop messages from history" machinery that became unreachable after
PR-A1 made `/v1/messages` a passthrough on the proxy. Live-zone-only
compression (PR-B2..B7) operates on content blocks within messages;
message-list mutation no longer happens in the pipeline.
Python deletes:
- headroom/transforms/intelligent_context.py (1077 LOC)
- headroom/transforms/rolling_window.py (395 LOC)
- headroom/transforms/progressive_summarizer.py (508 LOC)
- headroom/transforms/scoring.py (459 LOC)
- headroom/transforms/tool_crusher.py (338 LOC)
- 5 corresponding tests/test_transforms/* and tests/test_proxy_intelligent_context.py
Rust deletes:
- crates/headroom-core/src/context/* (manager, config, workspace,
candidate, ccr_drop, strategy/, mod) + safety.rs replaced
- crates/headroom-core/src/scoring/* (mod, score, scorer, traits, weights)
- MessageScorerComparator from crates/headroom-parity (PR #338/#343
becomes deletable; sunk cost stays sunk)
- 13 message_scorer fixtures + record_message_scorer.py
Rust adds (move + rewrite):
- crates/headroom-core/src/transforms/safety.rs — `tool_pair_indices`
preserves the OpenAI/Anthropic tool_use ↔ tool_result pairing rule
the live-zone dispatcher (PR-B2) needs. No IcmConfig dependency.
Surface refactors:
- HeadroomConfig: drop `tool_crusher`, `rolling_window`,
`intelligent_context` fields; hoist `output_buffer_tokens` to top
level (used by client.py).
- ProxyConfig: drop `intelligent_context*` fields.
- `headroom wrap` proxy server: retire IntelligentContextManager
and RollingWindow imports + branch; pipeline is CacheAligner →
ContentRouter (smart_routing) or CacheAligner → SmartCrusher
(legacy).
- CLI: drop `--no-intelligent-context`, `--no-intelligent-scoring`,
`--no-compress-first` flags.
- LangChain memory integration: rename `_apply_rolling_window` →
`_apply_compression`, drop RollingWindowConfig dep. Threshold is
now advisory — B6 will rework the contract.
- TransformPipeline.create_pipeline now takes only cache_aligner_config.
- headroom/__init__.py + headroom/transforms/__init__.py: strip
exports of deleted symbols.
Bug fixes uncovered by full pytest sweep:
- providers/copilot/wrap.py: `environ or os.environ` collapsed
empty-dict to falsy → callers passing `environ={}` accidentally
pulled from os.environ. Use `environ if environ is not None else
os.environ`.
Test correctness fixes:
- _DummyAnthropicHandler._retry_request gains **_kwargs to match
the real handler signature post-A8.
- test_ws_http_fallback extracts JSON from `content=` (post-A3
byte-faithful) rather than the obsolete `json=` kwarg.
- test_ccr_response_handler_extra fixture joins SSE events with
`\n\n` per spec (post-A8 byte-buffer parser requirement).
- test_proxy_responses_phase_preservation: capture via direct
handler attached to the named logger, so the assertion is
order-independent (proxy `_setup_file_logging` flips
`headroom.propagate=False` once any earlier test triggers it).
- conftest.py autouse fixture resets `headroom.propagate=True`
before each test as a defensive measure for the same pollution.
- test_wrap_copilot_translated_backend_still_requires_byok:
monkeypatch.delenv every provider key so the BYOK error
actually fires.
- test_native_installers: skip when system bash < 4.3 (macOS ships 3.2).
- TestGeminiEmbedContent / TestGeminiBatchEmbedContents:
pytest.mark.skip — proxy currently has no :embedContent route;
feature gap, not regression.
Acceptance:
- cargo build --workspace + cargo clippy + cargo fmt --check: green.
- cargo test --workspace --exclude headroom-py: 777 passed.
- pytest: 4892 passed, 240 skipped, 0 failed.
- git grep returns only intentional comments referencing the deletion.
Per-PR-B1 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
|
44944fb3fe |
fix(proxy): restore Anthropic compression on token mode (issue #327)
Three bugs combined to drive end-to-end compression on the Anthropic backend to ~0% in token mode (the default). User report #327 saw a ~9× drop in dashboard savings from one day to the next on Claude Code traffic; the dashboard headline was technically correct but the underlying compression genuinely was not running. After this change the same Claude Code-shape multi-turn conversation goes from 14987 → 14371 tokens at the request boundary on turn 1 and only recompresses the freshest tool_result on subsequent turns, with the prior turns frozen byte-identical to preserve the upstream prefix cache. Bug 1 — IntelligentContextManager inner ContentRouter has no observer PR #302 (commit |
||
|
|
2a6ab38177 |
test(observability): isolate SmartCrusher.apply tests from global TOIN file
The new SmartCrusher.apply() tests added in this PR feed the global TOIN learning store (default path `~/.headroom/toin.json`). On Python 3.11, CI runs the suite twice (regular + coverage); a pattern written in the first pass changes which rows the lossy sampler keeps in the second pass and breaks `tests/test_integrations/langchain/test_evals.py::TestRelevancePreservation::test_first_last_items_always_preserved`. Fix: an `isolated_toin` fixture that points `HEADROOM_TOIN_PATH` at a tempdir for the test and resets the singleton on enter and exit. Same isolation pattern PR #300 already uses for `test_smart_crusher_toin_attachment.py`. |
||
|
|
cf97995834 |
chore(proxy): per-strategy compression observability
Adds a `CompressionObserver` Protocol (`headroom.transforms.observability`) and wires `ContentRouter` and `SmartCrusher` to call it once per real compression event. `PrometheusMetrics` implements the protocol and accumulates per-strategy counters (`compressions_by_strategy`, `tokens_saved_by_strategy`). Why: the TOIN→SmartCrusher silent disconnect was invisible for three weeks because no signal distinguished by strategy. With per-strategy counters in place, the next regression of that shape fails the test suite the day it lands instead of waiting on a manual audit. Counters live as in-process state only — deliberately NOT exported via the Prometheus scrape or OTel surface. The metric→Supabase pipeline treats each metric name as a column, and we cannot add new columns. CI-level observability via `tests/test_compression_observability.py` is sufficient to catch silent regressions; production export waits on a non-column-adding pipeline. |