Files
headroom/tests/test_litellm_callback.py
Tejas ChopraandClaude Opus 5.5 1b7466d1d7 test(integrations): drop duplicated LangChain and LiteLLM tests (#4133)
## Description

This PR removes duplicated tests for the LangChain and LiteLLM
integrations. It changes no product code.

- **`tests/test_integrations/langchain/test_extended.py`** repeated
tests for:
  - `HeadroomChatMessageHistory`
  - `HeadroomDocumentCompressor`
  - `HeadroomToolWrapper`
  - the streaming metrics tracker

The per-module test files already cover all four. Line and branch
coverage of `headroom/integrations/langchain` is identical with and
without these tests.
- **The LiteLLM tests removed** only asserted `hasattr`, `callable` or
not-None on hooks that other tests in the same files call and await.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `test_extended.py`: delete these classes:
  - `TestHeadroomChatMessageHistory` (5 tests)
  - `TestHeadroomDocumentCompressor` (7)
  - `TestStreamingMetricsTracker` (8)
  - 5 of the 6 `TestHeadroomToolWrapper` tests
- Kept from `TestHeadroomToolWrapper`:
`test_call_compresses_large_json`. It is the only LangChain test that
runs `compress_tool_result` unmocked. It is also the only test in this
set that reaches the non-JSON fallback in
`headroom/integrations/mcp/server.py:343-344`;
`tests/test_integrations/mcp` does not cover those lines.
- Also kept: `TestProviderDetection`,
`TestHeadroomLangSmithCallbackHandler` and
`TestAutoDetectProviderInChatModel`. They are the only coverage for
`providers.py` and `langsmith.py`.
- Moved, not deleted: `test_output_tokens` and
`test_messages_passthrough_under_threshold`. Every other memory and
streaming test passes a mock provider or patches `OpenAIProvider`; these
two were the only ones that ran the real default provider
(`OpenAIProvider().get_token_counter(model)`) end to end. They now live
next to their siblings:
-
`test_streaming.py::TestStreamingMetricsTrackerProperties::test_output_tokens_with_default_provider`
-
`test_memory.py::TestHeadroomChatMessageHistoryMessages::test_messages_below_threshold_with_default_provider`
- The `langchain_core` availability probe now imports the package
instead of 4 names that are no longer used.
- `tests/test_litellm_callback.py`: delete
`test_instantiates_without_error`,
`test_has_async_post_call_success_hook`,
`test_async_post_call_success_hook_is_callable` and
`test_async_pre_call_hook_still_works`. Still covered by:
  - `test_async_post_call_success_hook_does_not_raise`;
  - `test_all_current_litellm_async_hooks_present`;
  - the route tests that await `async_pre_call_hook`.
- `tests/test_integrations/test_litellm_callback.py`: delete
`test_method_exists` and `test_method_is_coroutine`.
`test_returns_response_unchanged` awaits the same hook.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_integrations/langchain/test_extended.py tests/test_litellm_callback.py tests/test_integrations/test_litellm_callback.py
origin/main: 59 passed
this branch: 28 passed   (31 removed: 25 + 4 + 2)
$ pytest <the same 3 files> tests/test_integrations/langchain/test_memory.py tests/test_integrations/langchain/test_streaming.py
origin/main: 124 passed
this branch: 95 passed   (29 fewer: 31 removed, 2 moved into test_memory.py / test_streaming.py)

# coverage evidence, on origin/main (coverage run --branch over tests/test_integrations/langchain, 244 passed / 21 skipped)
headroom/integrations/langchain/*  with the 26 tests:    1150 stmts, 161 miss, 346 branches, 52 partial
headroom/integrations/langchain/*  without the 26 tests: 1150 stmts, 161 miss, 346 branches, 52 partial
# the only difference outside langchain/ in that run: mcp/server.py 343-344, sqlite.py, compression_store.py, toin.py
# per-test check: mcp/server.py 343-344 come from test_call_compresses_large_json, which this branch keeps.
# The sqlite/compression_store/toin lines are side effects of real compression; their own test suites own them.

$ uvx ruff@0.16.8 check / format <3 touched files>  -> All checks passed! / unchanged
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo venv (langchain-core and litellm
installed).
- Exact command / steps:
- `python -m coverage run --branch -m pytest -p no:cov
tests/test_integrations/langchain`, with and without `--deselect` of the
four classes, then `coverage report -m` and a diff.
- `pytest-cov` breaks the litellm import (`KeyError:
'pydantic.root_model'`), so coverage was run directly.
- Observed result: identical numbers for
`headroom/integrations/langchain`. The one test that adds unique
coverage elsewhere is kept.
- Smoke harness (real proxy started from this checkout against a local
fake upstream; temp HOME and workspace, offline):

```text
branch: cleanup/cli-integrations-dedupe-integration-tests @ fc66c0bc
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-63921.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: <tmp>)
```
- Direct runtime command on this PR's own path (temp HOME,
HEADROOM_WORKSPACE_DIR and CODEX_HOME):

```text
$ python stream_run.py  # real StreamingMetricsTracker and HeadroomChatMessageHistory, no provider passed
default-provider output_tokens: 9
default-provider history: ['Hello', 'Hi there!'] compressions: 0
```
- Not tested: n/a (tests only).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a, tests only.
- Stable/default behavior changed: no.
- Kill switch / disable path: n/a.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

- This keeps one more test than the audit proposed (25 of 26 removed
from `test_extended.py`), for the reason above, and moves two of the 25
into the focused files so the default-provider path stays tested.
- mypy is unchecked because the PR changes no product code. Docs need no
change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4133"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:01:58 -07:00

140 lines
4.7 KiB
Python

"""Tests for HeadroomCallback LiteLLM integration.
Regression for #1114: HeadroomCallback did not inherit CustomLogger, so any
hook LiteLLM added post-1.89.x (e.g. async_post_call_success_hook) raised
AttributeError and crashed the LiteLLM proxy.
"""
from __future__ import annotations
import asyncio
from tests._dotenv import importorskip_no_env_leak
importorskip_no_env_leak("litellm")
from headroom.integrations.litellm_callback import HeadroomCallback # noqa: E402
class TestHeadroomCallbackCustomLoggerInheritance:
def test_async_post_call_success_hook_does_not_raise(self) -> None:
"""Calling the hook (no-op from CustomLogger) must not raise."""
cb = HeadroomCallback()
async def _run() -> None:
await cb.async_post_call_success_hook(
data={"model": "gpt-4o", "messages": []},
user_api_key_dict={},
response=None,
)
asyncio.run(_run())
def test_all_current_litellm_async_hooks_present(self) -> None:
"""HeadroomCallback must expose every async hook CustomLogger defines."""
from litellm.integrations.custom_logger import CustomLogger
cb = HeadroomCallback()
missing = [
name
for name in dir(CustomLogger)
if name.startswith("async_") and not hasattr(cb, name)
]
assert not missing, f"Missing CustomLogger hooks: {missing}"
def test_total_tokens_saved_property(self) -> None:
cb = HeadroomCallback()
assert cb.total_tokens_saved == 0
class TestAnthropicMessagesRouteCompression:
"""LiteLLM's proxy maps /v1/messages to the "anthropic_messages" call
type (and its async twin), so compression must fire there too (#3503)."""
@staticmethod
def _anthropic_payload() -> dict:
import json
big = json.dumps(
[
{"id": i, "status": "active", "name": f"item_{i}", "value": i * 17}
for i in range(200)
]
)
return {
"model": "claude-sonnet-4-5",
"system": "You are helpful.",
"messages": [
{"role": "user", "content": "What are the top items?"},
{
"role": "assistant",
"content": [
{
"type": "tool_use",
"id": "toolu_1",
"name": "list_items",
"input": {},
}
],
},
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "toolu_1",
"content": big,
}
],
},
],
}
@staticmethod
def _run(call_type: str, data: dict):
import asyncio
cb = HeadroomCallback()
out = asyncio.run(
cb.async_pre_call_hook(user_api_key_dict={}, cache=None, data=data, call_type=call_type)
)
return cb, out
def test_anthropic_messages_route_compresses(self) -> None:
cb, out = self._run("anthropic_messages", self._anthropic_payload())
assert cb.total_tokens_saved > 0
assert out["messages"][2]["content"][0]["type"] == "tool_result"
def test_aanthropic_messages_route_compresses(self) -> None:
cb, out = self._run("aanthropic_messages", self._anthropic_payload())
assert cb.total_tokens_saved > 0
assert out["messages"][2]["content"][0]["type"] == "tool_result"
def test_completion_route_still_compresses(self) -> None:
big = self._anthropic_payload()["messages"][2]["content"][0]["content"]
payload = {
"model": "gpt-4o",
"messages": [
{"role": "user", "content": "What are the top items?"},
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": "call_1",
"type": "function",
"function": {"name": "list_items", "arguments": "{}"},
}
],
},
{"role": "tool", "content": big, "tool_call_id": "call_1"},
],
}
cb, _ = self._run("completion", payload)
assert cb.total_tokens_saved > 0
def test_non_chat_routes_still_skip(self) -> None:
data = self._anthropic_payload()
_, out = self._run("embeddings", data)
assert out["messages"] == data["messages"]