Commit Graph
4 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 1b7466d1d7 test(integrations): drop duplicated LangChain and LiteLLM tests (#4133)
## Description

This PR removes duplicated tests for the LangChain and LiteLLM
integrations. It changes no product code.

- **`tests/test_integrations/langchain/test_extended.py`** repeated
tests for:
  - `HeadroomChatMessageHistory`
  - `HeadroomDocumentCompressor`
  - `HeadroomToolWrapper`
  - the streaming metrics tracker

The per-module test files already cover all four. Line and branch
coverage of `headroom/integrations/langchain` is identical with and
without these tests.
- **The LiteLLM tests removed** only asserted `hasattr`, `callable` or
not-None on hooks that other tests in the same files call and await.

Refs #4136

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `test_extended.py`: delete these classes:
  - `TestHeadroomChatMessageHistory` (5 tests)
  - `TestHeadroomDocumentCompressor` (7)
  - `TestStreamingMetricsTracker` (8)
  - 5 of the 6 `TestHeadroomToolWrapper` tests
- Kept from `TestHeadroomToolWrapper`:
`test_call_compresses_large_json`. It is the only LangChain test that
runs `compress_tool_result` unmocked. It is also the only test in this
set that reaches the non-JSON fallback in
`headroom/integrations/mcp/server.py:343-344`;
`tests/test_integrations/mcp` does not cover those lines.
- Also kept: `TestProviderDetection`,
`TestHeadroomLangSmithCallbackHandler` and
`TestAutoDetectProviderInChatModel`. They are the only coverage for
`providers.py` and `langsmith.py`.
- Moved, not deleted: `test_output_tokens` and
`test_messages_passthrough_under_threshold`. Every other memory and
streaming test passes a mock provider or patches `OpenAIProvider`; these
two were the only ones that ran the real default provider
(`OpenAIProvider().get_token_counter(model)`) end to end. They now live
next to their siblings:
-
`test_streaming.py::TestStreamingMetricsTrackerProperties::test_output_tokens_with_default_provider`
-
`test_memory.py::TestHeadroomChatMessageHistoryMessages::test_messages_below_threshold_with_default_provider`
- The `langchain_core` availability probe now imports the package
instead of 4 names that are no longer used.
- `tests/test_litellm_callback.py`: delete
`test_instantiates_without_error`,
`test_has_async_post_call_success_hook`,
`test_async_post_call_success_hook_is_callable` and
`test_async_pre_call_hook_still_works`. Still covered by:
  - `test_async_post_call_success_hook_does_not_raise`;
  - `test_all_current_litellm_async_hooks_present`;
  - the route tests that await `async_pre_call_hook`.
- `tests/test_integrations/test_litellm_callback.py`: delete
`test_method_exists` and `test_method_is_coroutine`.
`test_returns_response_unchanged` awaits the same hook.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_integrations/langchain/test_extended.py tests/test_litellm_callback.py tests/test_integrations/test_litellm_callback.py
origin/main: 59 passed
this branch: 28 passed   (31 removed: 25 + 4 + 2)
$ pytest <the same 3 files> tests/test_integrations/langchain/test_memory.py tests/test_integrations/langchain/test_streaming.py
origin/main: 124 passed
this branch: 95 passed   (29 fewer: 31 removed, 2 moved into test_memory.py / test_streaming.py)

# coverage evidence, on origin/main (coverage run --branch over tests/test_integrations/langchain, 244 passed / 21 skipped)
headroom/integrations/langchain/*  with the 26 tests:    1150 stmts, 161 miss, 346 branches, 52 partial
headroom/integrations/langchain/*  without the 26 tests: 1150 stmts, 161 miss, 346 branches, 52 partial
# the only difference outside langchain/ in that run: mcp/server.py 343-344, sqlite.py, compression_store.py, toin.py
# per-test check: mcp/server.py 343-344 come from test_call_compresses_large_json, which this branch keeps.
# The sqlite/compression_store/toin lines are side effects of real compression; their own test suites own them.

$ uvx ruff@0.16.8 check / format <3 touched files>  -> All checks passed! / unchanged
```

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo venv (langchain-core and litellm
installed).
- Exact command / steps:
- `python -m coverage run --branch -m pytest -p no:cov
tests/test_integrations/langchain`, with and without `--deselect` of the
four classes, then `coverage report -m` and a diff.
- `pytest-cov` breaks the litellm import (`KeyError:
'pydantic.root_model'`), so coverage was run directly.
- Observed result: identical numbers for
`headroom/integrations/langchain`. The one test that adds unique
coverage elsewhere is kept.
- Smoke harness (real proxy started from this checkout against a local
fake upstream; temp HOME and workspace, offline):

```text
branch: cleanup/cli-integrations-dedupe-integration-tests @ fc66c0bc
health: 200
POST /v1/messages        -> HTTP200 smoke-ok
POST /v1/chat/completions -> HTTP200 smoke-ok
stats: requests=2 failed=0 tokens_saved=13574 savings_percent=99.82
proxy.log WARNING/ERROR lines (<tmp>/ws/logs/proxy-63921.log):
   2 headroom.proxy - WARNING - event=proxy_offline_mode air-gap active — every connection Headroom initiates is refused (telemetry, update check, license reporter, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP/Langfuse export, Copilot auth, subscription polling, OpenAI embedders, Headroom Cloud compression). Still allowed on purpose: forwarding your requests to the upstream you configured, operator-configured local endpoints, loopback health probes.
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch found no usable file for chopratejas/kompress-v2-base
   4 headroom.transforms.kompress_compressor - WARNING - Kompress: artifact prefetch refused for chopratejas/kompress-v2-base (HEADROOM_OFFLINE is set): HEADROOM_OFFLINE is set: refusing outbound network access for HuggingFace download of chopratejas/kompress-v2-base/onnx/kompress-int8-wo.onnx to huggingface.co. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection.
(smoke dir: <tmp>)
```
- Direct runtime command on this PR's own path (temp HOME,
HEADROOM_WORKSPACE_DIR and CODEX_HOME):

```text
$ python stream_run.py  # real StreamingMetricsTracker and HeadroomChatMessageHistory, no provider passed
default-provider output_tokens: 9
default-provider history: ['Hello', 'Hi there!'] compressions: 0
```
- Not tested: n/a (tests only).

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a, tests only.
- Stable/default behavior changed: no.
- Kill switch / disable path: n/a.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

Part of maintainer-requested cleanup #4136 (dead code, duplication,
debuggability), split into small PRs for review.

- This keeps one more test than the audit proposed (25 of 26 removed
from `test_extended.py`), for the reason above, and moves two of the 25
into the focused files so the default-provider path stays tested.
- mypy is unchecked because the PR changes no product code. Docs need no
change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4133"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 09:01:58 -07:00
Abdellatif Anaflous 63c5df8a39 fix(litellm): compress anthropic messages route in HeadroomCallback (#3504)
## Description

Closes #3503

LiteLLM's proxy maps /v1/messages and /anthropic/v1/messages to the
`anthropic_messages` call type (`aanthropic_messages` for the async
path), but `HeadroomCallback.async_pre_call_hook` only accepted
`completion` and `acompletion`, so every request through the Anthropic
Messages route exited at the call-type gate and no compression ran at
all
the payload on that route still carries `messages` in Anthropic format
and `compress()` already handles that shape, so the gate now accepts the
two anthropic call types next to the existing two

### Steps to reproduce
1. build a `HeadroomCallback()` and call
`async_pre_call_hook(user_api_key_dict={}, cache=None, data=<anthropic
payload with a 200-row JSON tool_result block>,
call_type="anthropic_messages")`
2. Expected: the tool_result content gets compressed and
`total_tokens_saved` grows
3. Actual (raw output on untouched main e67b3c8a):

```text
=== control: OpenAI-format completion route ===
call_type='completion': tokens_saved=3387 messages_modified=True
=== lane: /v1/messages route (reporter's case) ===
call_type='anthropic_messages': tokens_saved=0 messages_modified=False
=== lane: async variant ===
call_type='aanthropic_messages': tokens_saved=0 messages_modified=False
```

after the fix the same probe saves tokens on both anthropic call types
and the completion control still fires

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/integrations/litellm_callback.py`: the pre-call gate now
also accepts `anthropic_messages` and `aanthropic_messages`, with a
comment naming where LiteLLM defines those call types
(`CallTypes.anthropic_messages` in litellm/types/utils.py, route table
for /v1/messages)
- `tests/test_litellm_callback.py`: regression tests for both call types
(tokens saved + tool_result block structure preserved), a
completion-route control, and a guard that non-chat routes still skip

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

```text
python -m pytest -q tests/test_litellm_callback.py tests/test_integrations/test_litellm_callback.py tests/test_compress_api.py
32 passed, 5 skipped in 2.23s

without the fix (git checkout main -- headroom/integrations/litellm_callback.py):
2 failed, 9 passed in 2.22s
(the two new anthropic_messages tests fail, the completion control and the skip guard pass)

ruff check .
All checks passed!

ruff format --check .
1567 files already formatted

mypy headroom
Success: no issues found in 530 source files
```

## Real Behavior Proof

- Environment: Linux, Python 3.12, headroom main at e67b3c8a in a venv
- Exact command / steps: the probe above, then the pytest triple before
and after the fix, with the fix file checked out from main to show the
tests fail without it
- Observed result: `anthropic_messages` and `aanthropic_messages` saved
0 tokens and left messages untouched before the fix, the same payload
compresses after it, the completion control saves tokens in both runs,
`embeddings` still skips
- Not tested: a live LiteLLM proxy with an Anthropic-format client end
to end, the unit tests stand in for how the proxy passes the call type

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: yes, the callback now compresses on
the Anthropic Messages routes, that is the fix
- Kill switch / disable path: none needed, revert the commit or drop the
callback from litellm.callbacks
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the one commit, nothing else to unwind

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review
2026-09-11 20:42:08 -07:00
JerrettDavis 8527b910dc test(litellm): remove unused pytest import 2026-07-09 20:59:12 -05:00
Peter Lodri 9be727de68 fix(litellm): inherit CustomLogger so future hooks don't crash proxy (#1114) (#1391)
## Summary

Fixes #1114 — LiteLLM 1.89.x added `async_post_call_success_hook` and
started calling it after every successful completion. `HeadroomCallback`
was a plain `object` subclass with no such method, causing:

```
AttributeError: type object 'HeadroomCallback' has no attribute 'async_post_call_success_hook'
```

This crashed the LiteLLM proxy on every successful API call.

### Root cause

```python
class HeadroomCallback:   # plain object — no-op hooks not inherited
    ...
```

### Fix

Inherit from `litellm.integrations.custom_logger.CustomLogger` which
provides no-op defaults for every hook it defines. Future additions to
`CustomLogger` will be covered automatically.

```python
try:
    from litellm.integrations.custom_logger import CustomLogger as _CustomLogger
except ImportError:
    _CustomLogger = object  # litellm not installed — graceful fallback

class HeadroomCallback(_CustomLogger):
    ...
    def __init__(self, ...):
        super().__init__()
        ...
```

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/integrations/litellm_callback.py` — inherit `CustomLogger`;
add `super().__init__()`
- `tests/test_litellm_callback.py` — 7 tests: instantiation,
`async_post_call_success_hook` present + callable + no-op, all current
hooks present, pre-call hook still works

## Real behavior proof

**Setup:** Python 3.13, litellm 1.89.1, headroom-ai 0.27.0-dev  
**Steps after patch:**  
```bash
python3 -c "
from headroom.integrations.litellm_callback import HeadroomCallback
import asyncio
cb = HeadroomCallback()
# Simulate what litellm proxy calls on success
asyncio.run(cb.async_post_call_success_hook(data={}, user_api_key_dict={}, response=None))
print('OK — no AttributeError')
"
```
**After-fix evidence:** Runs without exception. Before fix:
`AttributeError: type object 'HeadroomCallback' has no attribute
'async_post_call_success_hook'`.

**What I did not test:** Live LiteLLM proxy with YAML config (no LiteLLM
proxy running in test env); tested via unit tests and direct Python
instantiation.

## Test Results

```
tests/test_litellm_callback.py  7/7 passed
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-09 17:09:33 -05:00