mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description
This PR removes duplicated tests for the LangChain and LiteLLM
integrations. It changes no product code.
- **`tests/test_integrations/langchain/test_extended.py`** repeated
tests for:
- `HeadroomChatMessageHistory`
- `HeadroomDocumentCompressor`
- `HeadroomToolWrapper`
- the streaming metrics tracker
The per-module test files already cover all four. Line and branch
coverage of `headroom/integrations/langchain` is identical with and
without these tests.
- **The LiteLLM tests removed** only asserted `hasattr`, `callable` or
not-None on hooks that other tests in the same files call and await.
Refs #4136
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `test_extended.py`: delete these classes:
- `TestHeadroomChatMessageHistory` (5 tests)
- `TestHeadroomDocumentCompressor` (7)
- `TestStreamingMetricsTracker` (8)
- 5 of the 6 `TestHeadroomToolWrapper` tests
- Kept from `TestHeadroomToolWrapper`:
`test_call_compresses_large_json`. It is the only LangChain test that
runs `compress_tool_result` unmocked. It is also the only test in this
set that reaches the non-JSON fallback in
`headroom/integrations/mcp/server.py:343-344`;
`tests/test_integrations/mcp` does not cover those lines.
- Also kept: `TestProviderDetection`,
`TestHeadroomLangSmithCallbackHandler` and
`TestAutoDetectProviderInChatModel`. They are the only coverage for
`providers.py` and `langsmith.py`.
- Moved, not deleted: `test_output_tokens` and
`test_messages_passthrough_under_threshold`. Every other memory and
streaming test passes a mock provider or patches `OpenAIProvider`; these
two were the only ones that ran the real default provider
(`OpenAIProvider().get_token_counter(model)`) end to end. They now live
next to their siblings:
-
`test_streaming.py::TestStreamingMetricsTrackerProperties::test_output_tokens_with_default_provider`
-
`test_memory.py::TestHeadroomChatMessageHistoryMessages::test_messages_below_threshold_with_default_provider`
- The `langchain_core` availability probe now imports the package
instead of 4 names that are no longer used.
- `tests/test_litellm_callback.py`: delete
`test_instantiates_without_error`,
`test_has_async_post_call_success_hook`,
`test_async_post_call_success_hook_is_callable` and
`test_async_pre_call_hook_still_works`. Still covered by:
- `test_async_post_call_success_hook_does_not_raise`;
- `test_all_current_litellm_async_hooks_present`;
- the route tests that await `async_pre_call_hook`.
- `tests/test_integrations/test_litellm_callback.py`: delete
`test_method_exists` and `test_method_is_coroutine`.
`test_returns_response_unchanged` awaits the same hook.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_integrations/langchain/test_extended.py tests/test_litellm_callback.py tests/test_integrations/test_litellm_callback.py
origin/main: 59 passed
this branch: 28 passed (31 removed: 25 + 4 + 2)
$ pytest <the same 3 files> tests/test_integrations/langchain/test_memory.py tests/test_integrations/langchain/test_streaming.py
origin/main: 124 passed
this branch: 95 passed (29 fewer: 31 removed, 2 moved into test_memory.py / test_streaming.py)
# coverage evidence, on origin/main (coverage run --branch over tests/test_integrations/langchain, 244 passed / 21 skipped)
headroom/integrations/langchain/* with the 26 tests: 1150 stmts, 161 miss, 346 branches, 52 partial
headroom/integrations/langchain/* without the 26 tests: 1150 stmts, 161 miss, 346 branches, 52 partial
# the only difference outside langchain/ in that run: mcp/server.py 343-344, sqlite.py, compression_store.py, toin.py
# per-test check: mcp/server.py 343-344 come from test_call_compresses_large_json, which this branch keeps.
# The sqlite/compression_store/toin lines are side effects of real compression; their own test suites own them.
$ uvx ruff@0.16.8 check / format <3 touched files> -> All checks passed! / unchanged
```
## Real Behavior Proof
- Environment: macOS, Python 3.12, repo venv (langchain-core and litellm
installed).
- Exact command / steps:
- `python -m coverage run --branch -m pytest -p no:cov
tests/test_integrations/langchain`, with and without `--deselect` of the
four classes, then `coverage report -m` and a diff.
- `pytest-cov` breaks the litellm import (`KeyError:
'pydantic.root_model'`), so coverage was run directly.
- Observed result: identical numbers for
`headroom/integrations/langchain`. The one test that adds unique
coverage elsewhere is kept.
- Smoke harness (real proxy started from this checkout against a local
fake upstream; temp HOME and workspace, offline):
```text
branch: cleanup/cli-integrations-dedupe-integration-tests @ fc66c0bc
health: 200
POST /v1/messages -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-63921.log):
2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: <tmp>)
```
- Direct runtime command on this PR's own path (temp HOME,
HEADROOM_WORKSPACE_DIR and CODEX_HOME):
```text
$ python stream_run.py # real StreamingMetricsTracker and HeadroomChatMessageHistory, no provider passed
default-provider output_tokens: 9
default-provider history: ['Hello', 'Hi there!'] compressions: 0
```
- Not tested: n/a (tests only).
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a, tests only.
- Stable/default behavior changed: no.
- Kill switch / disable path: n/a.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.
- This keeps one more test than the audit proposed (25 of 26 removed
from `test_extended.py`), for the reason above, and moves two of the 25
into the focused files so the default-provider path stays tested.
- mypy is unchecked because the PR changes no product code. Docs need no
change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4133"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
140 lines
4.7 KiB
Python
140 lines
4.7 KiB
Python
"""Tests for HeadroomCallback LiteLLM integration.
|
|
|
|
Regression for #1114: HeadroomCallback did not inherit CustomLogger, so any
|
|
hook LiteLLM added post-1.89.x (e.g. async_post_call_success_hook) raised
|
|
AttributeError and crashed the LiteLLM proxy.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
|
|
from tests._dotenv import importorskip_no_env_leak
|
|
|
|
importorskip_no_env_leak("litellm")
|
|
|
|
from headroom.integrations.litellm_callback import HeadroomCallback # noqa: E402
|
|
|
|
|
|
class TestHeadroomCallbackCustomLoggerInheritance:
|
|
def test_async_post_call_success_hook_does_not_raise(self) -> None:
|
|
"""Calling the hook (no-op from CustomLogger) must not raise."""
|
|
cb = HeadroomCallback()
|
|
|
|
async def _run() -> None:
|
|
await cb.async_post_call_success_hook(
|
|
data={"model": "gpt-4o", "messages": []},
|
|
user_api_key_dict={},
|
|
response=None,
|
|
)
|
|
|
|
asyncio.run(_run())
|
|
|
|
def test_all_current_litellm_async_hooks_present(self) -> None:
|
|
"""HeadroomCallback must expose every async hook CustomLogger defines."""
|
|
from litellm.integrations.custom_logger import CustomLogger
|
|
|
|
cb = HeadroomCallback()
|
|
missing = [
|
|
name
|
|
for name in dir(CustomLogger)
|
|
if name.startswith("async_") and not hasattr(cb, name)
|
|
]
|
|
assert not missing, f"Missing CustomLogger hooks: {missing}"
|
|
|
|
def test_total_tokens_saved_property(self) -> None:
|
|
cb = HeadroomCallback()
|
|
assert cb.total_tokens_saved == 0
|
|
|
|
|
|
class TestAnthropicMessagesRouteCompression:
|
|
"""LiteLLM's proxy maps /v1/messages to the "anthropic_messages" call
|
|
type (and its async twin), so compression must fire there too (#3503)."""
|
|
|
|
@staticmethod
|
|
def _anthropic_payload() -> dict:
|
|
import json
|
|
|
|
big = json.dumps(
|
|
[
|
|
{"id": i, "status": "active", "name": f"item_{i}", "value": i * 17}
|
|
for i in range(200)
|
|
]
|
|
)
|
|
return {
|
|
"model": "claude-sonnet-4-5",
|
|
"system": "You are helpful.",
|
|
"messages": [
|
|
{"role": "user", "content": "What are the top items?"},
|
|
{
|
|
"role": "assistant",
|
|
"content": [
|
|
{
|
|
"type": "tool_use",
|
|
"id": "toolu_1",
|
|
"name": "list_items",
|
|
"input": {},
|
|
}
|
|
],
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "tool_result",
|
|
"tool_use_id": "toolu_1",
|
|
"content": big,
|
|
}
|
|
],
|
|
},
|
|
],
|
|
}
|
|
|
|
@staticmethod
|
|
def _run(call_type: str, data: dict):
|
|
import asyncio
|
|
|
|
cb = HeadroomCallback()
|
|
out = asyncio.run(
|
|
cb.async_pre_call_hook(user_api_key_dict={}, cache=None, data=data, call_type=call_type)
|
|
)
|
|
return cb, out
|
|
|
|
def test_anthropic_messages_route_compresses(self) -> None:
|
|
cb, out = self._run("anthropic_messages", self._anthropic_payload())
|
|
assert cb.total_tokens_saved > 0
|
|
assert out["messages"][2]["content"][0]["type"] == "tool_result"
|
|
|
|
def test_aanthropic_messages_route_compresses(self) -> None:
|
|
cb, out = self._run("aanthropic_messages", self._anthropic_payload())
|
|
assert cb.total_tokens_saved > 0
|
|
assert out["messages"][2]["content"][0]["type"] == "tool_result"
|
|
|
|
def test_completion_route_still_compresses(self) -> None:
|
|
big = self._anthropic_payload()["messages"][2]["content"][0]["content"]
|
|
payload = {
|
|
"model": "gpt-4o",
|
|
"messages": [
|
|
{"role": "user", "content": "What are the top items?"},
|
|
{
|
|
"role": "assistant",
|
|
"content": None,
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_1",
|
|
"type": "function",
|
|
"function": {"name": "list_items", "arguments": "{}"},
|
|
}
|
|
],
|
|
},
|
|
{"role": "tool", "content": big, "tool_call_id": "call_1"},
|
|
],
|
|
}
|
|
cb, _ = self._run("completion", payload)
|
|
assert cb.total_tokens_saved > 0
|
|
|
|
def test_non_chat_routes_still_skip(self) -> None:
|
|
data = self._anthropic_payload()
|
|
_, out = self._run("embeddings", data)
|
|
assert out["messages"] == data["messages"]
|