Files
headroom/tests/test_backend_streaming_cache_metrics.py
f78e66f7cc fix(proxy): stop headroom logger from suppressing propagation to stdout/stderr (#3096)
## Description

`_setup_file_logging()` in `headroom/proxy/helpers.py` unconditionally
set `headroom_logger.propagate = False`, on the theory that `wrap.py`
redirects stderr into the same `proxy.log` file the rotating handler
writes to, so propagation would double-write records. That premise is
wrong: `wrap.py` redirects the proxy subprocess's stdout/stderr to a
separate sibling file, `proxy-stdio.log` (see
`_get_proxy_stdio_log_path()`), never `proxy.log`. Severing propagation
bought no deduplication — it only stopped every `headroom.*` log record
from reaching root's `logging.basicConfig()` handler
(`proxy/server.py`), which is what writes to stderr/stdout. In
containerized deployments (Docker/Kubernetes), stdout/stderr is the only
channel the platform collects, so this silently dropped all application
logs.

Closes #3087

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Removed `headroom_logger.propagate = False` from
`_setup_file_logging()` in `headroom/proxy/helpers.py`.
- Updated the surrounding comment to explain why propagation is safe to
leave on: `proxy.log` (file handler) and stderr/stdout (root's
`basicConfig` handler) are two distinct destinations, so nothing is
duplicated in either file.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check .
All checks passed!

$ pytest tests/test_corrupt_golden_bytes_recovery.py -q
collected 9 items
tests/test_corrupt_golden_bytes_recovery.py .........                    [100%]
9 passed in 0.47s
```

`mypy headroom` was not run to a clean result: it fails before reaching
this file with a pre-existing, unrelated error in the installed `numpy`
stub (`Type statement is only supported in Python 3.12 and greater`) in
this sandbox's dependency set. Not caused by this change (a 4-line
pure-Python logging tweak).

The broader `pytest tests/` collection also hits ~170 pre-existing,
unrelated errors in this sandbox: `headroom._core`, the compiled Rust
extension, isn't built here (no `cargo`/`maturin` available), so every
module importing it fails at collection. This change doesn't touch Rust
code or `_core` and isn't implicated.

No new tests added: this restores an existing, already-tested code path
(see `tests/conftest.py`'s autouse fixture and
`tests/test_corrupt_golden_bytes_recovery.py`, both of which already
assert/require `propagate=True` for `caplog` to work) to its intended
default rather than adding new behavior.

## Real Behavior Proof

- Environment: local sandbox, Python 3.12.3, unable to build/import
`headroom.proxy.server` directly (missing compiled `headroom._core`).
Verified with an isolated script that copies `_setup_file_logging()`'s
exact logic byte-for-byte plus the same `logging.basicConfig()` call
`proxy/server.py` makes at import time.
- Exact command / steps: ran the fixed logic, then emitted
`logging.getLogger("headroom.proxy").info("compression timeout: request
abc123 exceeded 30s budget")`.
- Observed result: the `headroom.proxy` record was written to stderr AND
to `proxy.log` simultaneously (full output below); re-running the same
script with the old `headroom_logger.propagate = False` line restored
reproduces the bug — the record disappears from stderr entirely and only
lands in `proxy.log`.
  ```text
  propagate flag on 'headroom' logger: True
--- emitting a headroom.proxy log record, watch for it below on stderr
---
2026-08-17 21:17:39,545 - headroom.proxy - INFO - compression timeout:
request abc123 exceeded 30s budget
  --- also present in proxy.log (file handler): OK ---
  ```
- Not tested: end-to-end through the actual compiled proxy binary/server
process (blocked by missing `headroom._core` in this sandbox) and real
container log collection (Docker/Kubernetes) — the isolated repro
exercises the exact same logging configuration code the real process
runs, so this is considered sufficient proxy proof for a pure
logging-config change.

## Runtime Rollout Safety

- Rollout-managed feature(s): None — this is unconditional startup
logging configuration, not behind a rollout flag.
- Minimum rollout channel: N/A
- Stable/default behavior changed: Yes — `headroom.*` log records now
also reach stderr/stdout (previously file-only). No change to
request/response handling, compression, or any user-facing proxy
behavior.
- Kill switch / disable path: None needed; this restores default Python
logging propagation. If a deployment truly wants file-only logging,
`HEADROOM_LOG_TO_STDOUT`-style suppression could be added later, but no
such request exists today.
- Unsafe override required: No
- Qualification impact: None
- Rollback path: Revert this commit to restore `propagate = False`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — logging configuration change, no UI impact.

## Additional Notes

- "New and existing unit tests pass locally" is left unchecked because
the full `pytest tests/` collection can't complete in this sandbox
(missing compiled `headroom._core` Rust extension, unrelated to this
change) — see Test Output above for what was actually run and passed.
- "I have added tests" is left unchecked: existing tests
(`tests/conftest.py` autouse fixture,
`tests/test_corrupt_golden_bytes_recovery.py`) already assert the code
path this restores; happy to add a focused regression test asserting
`logging.getLogger("headroom").propagate is True` after
`_setup_file_logging()` if maintainers want one.

---------

Co-authored-by: = <=>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-01 00:03:53 -05:00

349 lines
12 KiB
Python

"""Cache-metric coverage for backend-routed streaming.
Two regressions in main as of 2026-05-14 (issue #327):
* ``StreamingMixin._stream_openai_via_backend`` (Azure/LiteLLM/AnyLLM OpenAI
streaming) never inspects ``usage.prompt_tokens_details.cached_tokens`` from
the upstream SSE chunks. Cache reads/writes are absent from
``cost_tracker.record_tokens``, ``SavingsTracker.record_request``, the
``RequestLog``, *and* the ``PERF`` log line — the latter is missing entirely
for this path, so ``headroom perf`` shows 0 cache writes for every
Azure-GPT/Codex backend-routed request.
* ``StreamingMixin._stream_response_bedrock`` (Bedrock-native streaming) hard-
codes ``cache_read=0 cache_write=0 cache_hit_pct=0`` in its PERF log line
regardless of what ``message_start.usage`` reported.
Both surface to the user as "Cache write: 0 tokens" in ``headroom perf``.
"""
from __future__ import annotations
import json
import logging
import re
from collections.abc import AsyncIterator
from typing import Any
from unittest.mock import MagicMock, patch
import pytest
fastapi = pytest.importorskip("fastapi")
httpx = pytest.importorskip("httpx")
from fastapi.testclient import TestClient # noqa: E402
from headroom.backends.base import StreamEvent # noqa: E402
from headroom.proxy.server import ProxyConfig, create_app # noqa: E402
PERF_RE = re.compile(
r"\bcache_read=(?P<cr>\d+)\s+cache_write=(?P<cw>\d+)\s+cache_hit_pct=(?P<chp>\d+)"
)
def _find_perf_record(records: list[logging.LogRecord]) -> tuple[int, int, int]:
"""Find the structured PERF log line and return (cache_read, cache_write, hit_pct)."""
for record in records:
msg = record.getMessage()
if " PERF " not in msg:
continue
m = PERF_RE.search(msg)
if m:
return int(m["cr"]), int(m["cw"]), int(m["chp"])
raise AssertionError(
"No PERF log line with cache_read/cache_write/cache_hit_pct found. "
f"Captured {len(records)} records.\n" + "\n".join(r.getMessage() for r in records[-15:])
)
class _ListHandler(logging.Handler):
"""Tiny direct handler immune to unrelated propagation mutations."""
def __init__(self) -> None:
super().__init__(level=logging.INFO)
self.records: list[logging.LogRecord] = []
def emit(self, record: logging.LogRecord) -> None: # noqa: D401
self.records.append(record)
def _attach_proxy_log_capture():
handler = _ListHandler()
target = logging.getLogger("headroom.proxy")
target.addHandler(handler)
prior_level = target.level
target.setLevel(logging.INFO)
return handler, target, prior_level
def _detach_proxy_log_capture(handler, target, prior_level) -> None:
target.removeHandler(handler)
target.setLevel(prior_level)
def _make_openai_backend(chunks: list[str]) -> MagicMock:
"""Build a mock backend that yields OpenAI-format SSE chunks."""
async def fake_stream(body: dict, headers: dict) -> AsyncIterator[str]:
for chunk in chunks:
yield chunk
mock = MagicMock()
mock.name = "anyllm-openai"
mock.stream_openai_message = fake_stream
return mock
def _make_bedrock_backend(events: list[StreamEvent]) -> MagicMock:
"""Build a mock backend that yields Anthropic StreamEvent objects."""
async def fake_stream(body: dict, headers: dict) -> AsyncIterator[StreamEvent]:
for evt in events:
yield evt
mock = MagicMock()
mock.name = "bedrock"
mock.stream_message = fake_stream
mock.map_model_id = MagicMock(return_value="claude-3-5-sonnet-20241022")
mock.supports_model = MagicMock(return_value=True)
return mock
# =============================================================================
# Bug A — _stream_openai_via_backend (Azure/LiteLLM/AnyLLM OpenAI streaming)
# =============================================================================
def test_openai_backend_streaming_emits_perf_with_cache_read_and_inferred_write() -> None:
"""OpenAI backend streaming must surface cache reads + inferred writes.
Real upstream (OpenAI Chat Completions w/ ``stream_options.include_usage=true``,
or Azure GPT-5.5 through LiteLLM) emits a final chunk carrying::
usage: {
prompt_tokens: 1000,
completion_tokens: 50,
prompt_tokens_details: { cached_tokens: 700 }
}
OpenAI never reports a separate write counter, so we infer it as
``max(prompt_tokens - cached_tokens, 0)`` (see
``_infer_openai_cache_write_tokens``). The PERF log line consumed by
``headroom perf`` must report both.
"""
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
backend="anyllm",
anyllm_provider="openai",
)
chunks = [
'data: {"id":"c1","object":"chat.completion.chunk",'
'"choices":[{"index":0,"delta":{"role":"assistant","content":"hi"}}]}\n\n',
'data: {"id":"c1","object":"chat.completion.chunk",'
'"choices":[{"index":0,"delta":{"content":" there"},"finish_reason":"stop"}]}\n\n',
'data: {"id":"c1","object":"chat.completion.chunk","choices":[],'
'"usage":{"prompt_tokens":1000,"completion_tokens":50,"total_tokens":1050,'
'"prompt_tokens_details":{"cached_tokens":700}}}\n\n',
"data: [DONE]\n\n",
]
backend = _make_openai_backend(chunks)
log_handle = _attach_proxy_log_capture()
try:
with patch("headroom.proxy.server.AnyLLMBackend", return_value=backend):
app = create_app(config)
with TestClient(app) as client:
resp = client.post(
"/v1/chat/completions",
json={
"model": "gpt-5.5",
"messages": [{"role": "user", "content": "hi"}],
"stream": True,
"stream_options": {"include_usage": True},
},
headers={"Authorization": "Bearer test-key"},
)
assert resp.status_code == 200, resp.text[:200]
body = resp.text
assert "[DONE]" in body, body[:300]
finally:
_detach_proxy_log_capture(*log_handle)
handler = log_handle[0]
cr, cw, chp = _find_perf_record(handler.records)
assert cr == 700, f"expected cache_read=700, got {cr}"
assert cw == 300, f"expected inferred cache_write=300 (=1000-700), got {cw}"
assert chp == 70, f"expected cache_hit_pct=70, got {chp}"
def test_openai_backend_streaming_perf_zeros_when_upstream_omits_usage() -> None:
"""When the upstream omits a usage chunk, cache values must be zero — not absent.
Without ``stream_options.include_usage=true`` (or when upstream drops the
final usage chunk) the PERF line still has to emit so ``headroom perf``
counts the request.
"""
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
backend="anyllm",
anyllm_provider="openai",
)
chunks = [
'data: {"id":"c1","object":"chat.completion.chunk",'
'"choices":[{"index":0,"delta":{"role":"assistant","content":"hi"}}]}\n\n',
"data: [DONE]\n\n",
]
backend = _make_openai_backend(chunks)
log_handle = _attach_proxy_log_capture()
try:
with patch("headroom.proxy.server.AnyLLMBackend", return_value=backend):
app = create_app(config)
with TestClient(app) as client:
resp = client.post(
"/v1/chat/completions",
json={
"model": "gpt-5.5",
"messages": [{"role": "user", "content": "hi"}],
"stream": True,
},
headers={"Authorization": "Bearer test-key"},
)
assert resp.status_code == 200
assert "[DONE]" in resp.text
finally:
_detach_proxy_log_capture(*log_handle)
handler = log_handle[0]
cr, cw, chp = _find_perf_record(handler.records)
assert (cr, cw, chp) == (0, 0, 0)
# =============================================================================
# Bug B — _stream_response_bedrock (Bedrock-native Anthropic streaming)
# =============================================================================
def _sse_data(event_type: str, data: dict[str, Any]) -> str:
return f"event: {event_type}\ndata: {json.dumps(data)}\n\n"
def test_bedrock_streaming_emits_perf_with_message_start_cache_usage() -> None:
"""Bedrock streaming must surface cache_read + cache_write from message_start.
Anthropic streaming reports cache usage on ``message_start.message.usage``
(cache_read_input_tokens + cache_creation_input_tokens). The Bedrock streamer
currently captures only ``input_tokens`` and ``output_tokens`` from the same
event and hardcodes ``cache_read=0 cache_write=0`` into the PERF log.
"""
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
backend="anyllm",
anyllm_provider="anthropic",
)
message_start = {
"type": "message_start",
"message": {
"id": "msg_1",
"model": "claude-3-5-sonnet-20241022",
"role": "assistant",
"type": "message",
"content": [],
"usage": {
"input_tokens": 1000,
"cache_read_input_tokens": 500,
"cache_creation_input_tokens": 200,
},
},
}
block_start = {
"type": "content_block_start",
"index": 0,
"content_block": {"type": "text", "text": ""},
}
block_delta = {
"type": "content_block_delta",
"index": 0,
"delta": {"type": "text_delta", "text": "hi"},
}
block_stop = {"type": "content_block_stop", "index": 0}
message_delta = {
"type": "message_delta",
"delta": {"stop_reason": "end_turn"},
"usage": {"output_tokens": 50},
}
message_stop = {"type": "message_stop"}
events = [
StreamEvent(
event_type=e["type"],
data=e,
raw_sse=_sse_data(e["type"], e),
)
for e in [
message_start,
block_start,
block_delta,
block_stop,
message_delta,
message_stop,
]
]
backend = _make_bedrock_backend(events)
log_handle = _attach_proxy_log_capture()
try:
with patch("headroom.proxy.server.AnyLLMBackend", return_value=backend):
app = create_app(config)
with TestClient(app) as client:
resp = client.post(
"/v1/messages",
json={
"model": "claude-3-5-sonnet-20241022",
"messages": [{"role": "user", "content": "hi"}],
"max_tokens": 64,
"stream": True,
},
headers={
"x-api-key": "sk-ant-test",
"anthropic-version": "2023-06-01",
},
)
assert resp.status_code == 200, resp.text[:200]
assert "message_stop" in resp.text
finally:
_detach_proxy_log_capture(*log_handle)
handler = log_handle[0]
cr, cw, chp = _find_perf_record(handler.records)
assert cr == 500, f"expected cache_read=500, got {cr}"
assert cw == 200, f"expected cache_write=200, got {cw}"
# round(500 / (500 + 200) * 100) = round(71.43) = 71
assert chp == 71, f"expected cache_hit_pct=71, got {chp}"
# =============================================================================
# Regression guard
# =============================================================================
def test_streaming_perf_log_has_no_hardcoded_cache_zeros() -> None:
"""Catch any future re-introduction of ``cache_read=0 cache_write=0`` literal."""
from pathlib import Path
src = Path(__file__).resolve().parents[1] / "headroom" / "proxy" / "handlers" / "streaming.py"
text = src.read_text()
assert "cache_read=0 cache_write=0" not in text, (
"streaming.py contains a hardcoded `cache_read=0 cache_write=0` PERF log fragment. "
"Wire the real cache_read/cache_write values into the PERF line instead."
)