mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(langchain): check optional integration availability (#4042)
## Description
Closes #4039.
The optional integration uses the existing LANGCHAIN_AVAILABLE guard;
installed LangChain binding precedence remains intact. Current head:
`ac9e49af43501d2b3bbb6d0a21cb07fe94030bc9`, normally updated onto main
`f1bc`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update
## Changes Made
- Install langchain-core in the existing CI lint job, with the
optional-extra version range, so the normal mypy invocation covers the
integration. Remove the pytest-launched mypy subprocess.
- Keep our nested unwrap-precedence regression (sync and async) and two
real loopback HTTP cases: bind_tools and nested caller-overrides. Remove
the vendor tool-response/usage-parsing matrix, as requested by the
maintainer.
- Replace the private LiteLLM cache invalidator with the public
register_model API; preserve the independent cost_per_token billing
oracle and fixture/catalog restoration.
- Correct the compatibility doc numbering and document the normal typing
path in CONTRIBUTING.
- Preserve the published startup SDK/provider module and parent-package
restoration fixes.
## Testing
- [ ] Unit tests pass (pytest) — no whole-project suite pass claimed.
- [ ] Linting passes (ruff check .) — changed-file Ruff passed; no new
whole-repository Ruff claim.
- [x] Type checking passes (mypy headroom)
- [x] Manual testing performed
### Test Output
Current published head, Python 3.13.14, installed langchain-core 1.6.10,
langchain-openai 1.7.1, LiteLLM 1.105.0 and a fresh native build:
`python -m pytest tests/test_integrations/langchain/test_binding_http.py
tests/test_integrations/langchain/test_langchain_1x_compat.py
tests/test_startup_log_noise.py tests/test_xai_cache_economics.py -q`
```text
28 passed, 1 skipped in 14.73s
```
The skip is optional trafilatura unavailable. A SQLite ResourceWarning
was emitted at interpreter teardown; it is not suppressed or claimed
fixed.
Normal mypy command in an isolated lint environment with mypy 1.20.2 and
installed langchain-core: success, no issues in 569 source files. This
local lint environment uses Python 3.14; the runtime consumer suite uses
Python 3.13.14. Existing untyped-body/unused-ignore notes remain.
Ordinary push gates: Rust workspace tests, fresh native extension build,
181 critical Python tests, and commitlint passed. Two prior normal
publication attempts failed seven native parity cases; neither was
bypassed or repinned. Publication passed with a separate clean Cargo
target. This establishes the successful isolated build, not a proven
root cause for the earlier build discrepancy.
## Real Behavior Proof
- Environment: installed SDKs, actual localhost HTTP server, no
provider-network or billing requests.
- Exact command / steps: an independent runtime driver invoked
HeadroomChatModel with bind_tools and nested RunnableBinding objects
against the real local ChatOpenAI transport; the retained HTTP
regressions exercised both cases again on the published head.
- Observed result: two actual requests; bind_tools sent lookup; nested
caller temperature 0.3 won, and inner top_p 0.8 survived. The sync/async
unwrap regression retains the real RunnableBinding precedence check. The
pricing oracle and startup teardown regressions passed.
- Not tested: live provider invoices, paid requests, remote production
deployment, or current-head hosted CI completion. Historical CI/approval
is not attributed to this new head.
## Runtime Rollout Safety
- Rollout-managed feature(s): none; optional integration guard and test
isolation.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: no binding precedence, runtime
defaults, authentication or serialization changes; CI lint now
explicitly installs the existing optional typing dependency.
- Kill switch / disable path: existing optional extra remains optional;
no new runtime switch.
- Unsafe override required: no.
- Qualification impact: optional-extra typing and real binding consumers
are exercised; normal local native/critical gates passed.
- Rollback path: revert the bounded guard/CI/test edits; no
persistent-state migration.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I did not edit CHANGELOG.md — release-please owns it
## Additional Notes
AI-assisted without subagents. No force push, skipped hook, new approval
or merge. The reduced HTTP scope follows the maintainer's explicit
request:
https://github.com/headroomlabs-ai/headroom/pull/4042#issuecomment-6157529674
.
### Maintainer current-main requalification
Exact head `ff329f3e487940f85d56ba0b35bd9994afadb506`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.
Requalified against upstream main
c01cf07d94. The only merge conflict
combined main's Python 3.11 minimum with the contributor's
optional-integration typing guidance; both retained. No contributor
availability/binding policy rewritten. Current author/chopratejas/Devin
discussion reviewed, including the intentional removal of vendor-owned
response-parsing coverage. Nested binding precedence, real loopback HTTP
controls, public LiteLLM register_model isolation, independent billing
oracle and parent-module restoration remain covered.
Fresh focused tests: 29 passed with installed langchain-core 1.4.2 and
langchain-openai 1.2.2. Full mypy against the Linux CI target checked
580 source files: PR reports 22 errors; an actual detached current-main
run with the same environment reports those same 22 plus a
RunnableBinding truthy-function error that this PR removes. The 22
shared errors are in unrelated unchanged prefix-tracker, optional
memory/MCP and CrewAI modules; they are not contributor blockers and
this is not a clean full-mypy claim. Five changed Python files pass Ruff
lint/format and main-relative whitespace checks. Current-main DLL
supports local startup, without a branch-specific native-build or live
model-provider claim. No new source blocker; approval awaits exact-head
hosted qualification. Nothing merged.
Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
JerrettDavis
parent
a8905214d9
commit
cde584d7fe
@@ -119,7 +119,7 @@ jobs:
|
||||
- name: Verify Ruff version alignment
|
||||
id: ruff-version
|
||||
run: echo "version=$(python scripts/verify-ruff-version.py --print-version)" >> "$GITHUB_OUTPUT"
|
||||
- run: python -m pip install --upgrade pip "ruff==${{ steps.ruff-version.outputs.version }}" "mypy==1.20.2"
|
||||
- run: python -m pip install --upgrade pip "ruff==${{ steps.ruff-version.outputs.version }}" "mypy==1.20.2" "langchain-core>=1.3.3,<4.0"
|
||||
- name: ruff check
|
||||
run: ruff check .
|
||||
- name: ruff format --check
|
||||
@@ -393,7 +393,7 @@ jobs:
|
||||
python -m pip install --upgrade pip
|
||||
pip install torch --index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple
|
||||
WHEEL="$(ls dist/*.whl)"
|
||||
pip install "${WHEEL}[dev]" pytest-split
|
||||
pip install "${WHEEL}[dev,langchain]" pytest-split
|
||||
# cwd's ./headroom source tree shadows the installed wheel; copy the
|
||||
# compiled extension in so tests import it (no second cargo build).
|
||||
SITE="$(python -c 'import sysconfig; print(sysconfig.get_path("platlib"))')"
|
||||
|
||||
@@ -130,6 +130,11 @@ Enable or disable automatic Copilot review in **Settings → Rules → Rulesets
|
||||
- Type hints on public functions; Google-style docstrings.
|
||||
- Cover new behavior + edge cases; aim >80% coverage on new code.
|
||||
- Python 3.11+. Optional features go behind extras.
|
||||
- For an optional-integration change, run its owner tests and `mypy headroom`
|
||||
with the relevant extra installed. A hook environment without that dependency
|
||||
does not check the imported library's types.
|
||||
CI lint installs `langchain-core` for its normal mypy run; keep typechecking
|
||||
there rather than launching a second mypy subprocess from pytest.
|
||||
- **Headroom writes LF and normalizes on read, on every platform.** Pass
|
||||
`newline="\n"` to every `write_text`/`open` that writes a context, memory,
|
||||
or state file, and normalize `\r\n`/`\r` when you read one back. Without
|
||||
|
||||
@@ -377,7 +377,7 @@ class HeadroomChatModel(BaseChatModel):
|
||||
pass them explicitly.
|
||||
"""
|
||||
bound_kwargs: dict[str, Any] = {}
|
||||
if not RunnableBinding:
|
||||
if not LANGCHAIN_AVAILABLE:
|
||||
return model, bound_kwargs
|
||||
# isinstance, not hasattr: a Mock answers hasattr("bound") truthfully
|
||||
# and would be unwrapped into one of its own auto-created children.
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
"""Cover Headroom's binding-kwargs merge with real LangChain and loopback HTTP."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from threading import Thread
|
||||
|
||||
import pytest
|
||||
|
||||
MODEL = "headroom-binding-loopback"
|
||||
|
||||
|
||||
def _tool(name):
|
||||
return {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": name,
|
||||
"description": "Look up the weather in a city.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"city": {"type": "string"}},
|
||||
"required": ["city"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _choice(name):
|
||||
return {"type": "function", "function": {"name": name}}
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def offline_headroom(monkeypatch, tmp_path):
|
||||
"""Disable automatic egress and isolate normal product state/config."""
|
||||
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
|
||||
# Track these too: apply_offline_env() sets them lazily during product use.
|
||||
monkeypatch.setenv("HF_HUB_OFFLINE", "1")
|
||||
monkeypatch.setenv("TRANSFORMERS_OFFLINE", "1")
|
||||
monkeypatch.setenv("DO_NOT_TRACK", "1")
|
||||
monkeypatch.setenv("HEADROOM_CONFIG_DIR", str(tmp_path / "config"))
|
||||
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path / "workspace"))
|
||||
monkeypatch.setenv("HEADROOM_MODEL_LIMITS", json.dumps({"context_limits": {MODEL: 128000}}))
|
||||
for name in ("LANGCHAIN_TRACING", "LANGCHAIN_TRACING_V2", "LANGSMITH_TRACING"):
|
||||
monkeypatch.setenv(name, "false")
|
||||
monkeypatch.setenv("OTEL_SDK_DISABLED", "true")
|
||||
monkeypatch.setenv("OPENAI_PROXY", "")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def openai_loopback():
|
||||
"""Run only a local HTTP fixture, without hosted calls or real credentials."""
|
||||
pytest.importorskip("langchain_openai")
|
||||
requests = []
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_POST(self):
|
||||
body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
|
||||
requests.append((self.path, body))
|
||||
message = {"role": "assistant", "content": "Loopback reply"}
|
||||
finish_reason = "stop"
|
||||
response = json.dumps(
|
||||
{
|
||||
"id": "chatcmpl-loopback",
|
||||
"object": "chat.completion",
|
||||
"created": 1,
|
||||
"model": MODEL,
|
||||
"choices": [{"index": 0, "message": message, "finish_reason": finish_reason}],
|
||||
"usage": {"prompt_tokens": 41, "completion_tokens": 9, "total_tokens": 50},
|
||||
}
|
||||
).encode()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(response)))
|
||||
self.end_headers()
|
||||
self.wfile.write(response)
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
|
||||
thread = Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
yield f"http://127.0.0.1:{server.server_port}/v1", requests
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
thread.join(timeout=5)
|
||||
assert not thread.is_alive(), "Loopback HTTP server did not stop"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("binding", ["bind_tools", "nested"])
|
||||
def test_binding_reaches_openai_http(openai_loopback, binding):
|
||||
import httpx
|
||||
from langchain_core.runnables import RunnableBinding
|
||||
from langchain_openai import ChatOpenAI
|
||||
from langsmith import tracing_context
|
||||
|
||||
from headroom.integrations import HeadroomChatModel
|
||||
|
||||
base_url, requests = openai_loopback
|
||||
with tracing_context(enabled=False), httpx.Client(trust_env=False) as http_client:
|
||||
original = ChatOpenAI(
|
||||
model=MODEL,
|
||||
api_key="loopback-dummy-key",
|
||||
base_url=base_url,
|
||||
max_retries=0,
|
||||
timeout=5,
|
||||
use_responses_api=False,
|
||||
http_client=http_client,
|
||||
)
|
||||
kwargs = {}
|
||||
expected_name = "inner_weather"
|
||||
if binding == "bind_tools":
|
||||
wrapped = HeadroomChatModel(original, auto_detect_provider=False).bind_tools(
|
||||
[_tool(expected_name)], tool_choice=expected_name
|
||||
)
|
||||
else:
|
||||
inner = original.bind_tools(
|
||||
[_tool("inner_weather")],
|
||||
tool_choice="inner_weather",
|
||||
temperature=0.1,
|
||||
top_p=0.8,
|
||||
)
|
||||
outer = RunnableBinding(
|
||||
bound=inner,
|
||||
kwargs={
|
||||
"tools": [_tool("outer_weather")],
|
||||
"tool_choice": _choice("outer_weather"),
|
||||
"temperature": 0.2,
|
||||
},
|
||||
)
|
||||
wrapped = HeadroomChatModel(outer, auto_detect_provider=False)
|
||||
expected_name = "caller_weather"
|
||||
kwargs = {
|
||||
"tools": [_tool(expected_name)],
|
||||
"tool_choice": _choice(expected_name),
|
||||
"temperature": 0.3,
|
||||
}
|
||||
wrapped.cache = False
|
||||
wrapped.invoke("What is the weather in Paris?", **kwargs)
|
||||
|
||||
_, body = requests[0]
|
||||
assert body["tools"] == [_tool(expected_name)]
|
||||
assert body["tool_choice"] == _choice(expected_name)
|
||||
if binding == "nested":
|
||||
assert body["temperature"] == 0.3
|
||||
assert body["top_p"] == 0.8
|
||||
|
||||
|
||||
def test_unwrap_binding_matches_real_runnable_precedence():
|
||||
pytest.importorskip("langchain_core")
|
||||
from langchain_core.runnables import RunnableBinding, RunnableLambda
|
||||
from langsmith import tracing_context
|
||||
|
||||
from headroom.integrations import HeadroomChatModel
|
||||
|
||||
def affine(value, scale=1, offset=0):
|
||||
return value * scale + offset
|
||||
|
||||
target = RunnableLambda(affine)
|
||||
inner = RunnableBinding(bound=target, kwargs={"scale": 2, "offset": 3})
|
||||
outer = RunnableBinding(bound=inner, kwargs={"scale": 5})
|
||||
unwrapped, kwargs = HeadroomChatModel._unwrap_binding(outer)
|
||||
plain, plain_kwargs = HeadroomChatModel._unwrap_binding(target)
|
||||
assert unwrapped is target
|
||||
|
||||
with tracing_context(enabled=False):
|
||||
assert outer.invoke(4) == unwrapped.invoke(4, **kwargs) == 23
|
||||
caller_kwargs = {**kwargs, "scale": 7}
|
||||
assert outer.invoke(4, scale=7) == unwrapped.invoke(4, **caller_kwargs) == 31
|
||||
assert asyncio.run(outer.ainvoke(4)) == asyncio.run(unwrapped.ainvoke(4, **kwargs)) == 23
|
||||
assert asyncio.run(outer.ainvoke(4, scale=7)) == 31
|
||||
assert asyncio.run(unwrapped.ainvoke(4, **caller_kwargs)) == 31
|
||||
assert plain.invoke(4, **plain_kwargs) == 4
|
||||
# Caller overrides must not mutate either reusable binding.
|
||||
assert inner.invoke(4) == 11
|
||||
assert outer.invoke(4) == 23
|
||||
@@ -3,27 +3,21 @@
|
||||
Each test here corresponds to a defect that shipped in 0.37.0 and was found by
|
||||
running the documented examples against langchain-core 1.6:
|
||||
|
||||
1. ``HeadroomChatModel.bind_tools()`` returned a model that emitted no tool
|
||||
calls, because ``_generate`` called the private method on the
|
||||
``RunnableBinding`` and the bound kwargs were dropped.
|
||||
2. ``wrap_tools_with_headroom`` produced tools that raised ``TypeError`` on
|
||||
1. ``wrap_tools_with_headroom`` produced tools that raised ``TypeError`` on
|
||||
invoke, because ``StructuredTool`` calls its ``func`` with unpacked kwargs
|
||||
while ``BaseTool.invoke`` takes a single input.
|
||||
3. The wrapped tool lost the original ``args_schema``, so a model saw a tool
|
||||
2. The wrapped tool lost the original ``args_schema``, so a model saw a tool
|
||||
with no parameters.
|
||||
4. ``HeadroomDocumentCompressor`` silently subclassed a local stub rather than
|
||||
3. ``HeadroomDocumentCompressor`` silently subclassed a local stub rather than
|
||||
LangChain's ``BaseDocumentCompressor``, so retrievers rejected it.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
try:
|
||||
from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
|
||||
from langchain_core.messages import AIMessage
|
||||
from langchain_core.tools import tool
|
||||
|
||||
LANGCHAIN_AVAILABLE = True
|
||||
@@ -61,60 +55,8 @@ def query_database(query: str) -> str:
|
||||
return BIG_RESULT
|
||||
|
||||
|
||||
class _RecordingModel(GenericFakeChatModel):
|
||||
"""Fake model that records the kwargs its _generate actually received."""
|
||||
|
||||
last_kwargs: dict[str, Any] = {}
|
||||
|
||||
def _generate(self, messages, stop=None, run_manager=None, **kwargs):
|
||||
type(self).last_kwargs = dict(kwargs)
|
||||
kwargs.pop("tools", None)
|
||||
return super()._generate(messages, stop=stop, run_manager=run_manager, **kwargs)
|
||||
|
||||
def bind_tools(self, tools, **kwargs):
|
||||
# Mirrors what real providers do: return a RunnableBinding carrying the
|
||||
# tools in .kwargs rather than a new model instance.
|
||||
return self.bind(tools=list(tools), **kwargs)
|
||||
|
||||
|
||||
class TestBindToolsSurvivesWrapping:
|
||||
"""Defect 1: bound kwargs must reach the underlying model."""
|
||||
|
||||
def test_bound_tools_reach_generate(self):
|
||||
from headroom.integrations import HeadroomChatModel
|
||||
|
||||
_RecordingModel.last_kwargs = {}
|
||||
inner = _RecordingModel(messages=iter([AIMessage("ok")]))
|
||||
wrapped = HeadroomChatModel(inner).bind_tools([query_database])
|
||||
|
||||
wrapped.invoke("call the tool")
|
||||
|
||||
assert "tools" in _RecordingModel.last_kwargs, (
|
||||
"bind_tools kwargs were dropped before reaching the wrapped model; "
|
||||
"an agent built on this model would never call a tool"
|
||||
)
|
||||
|
||||
def test_unbound_model_passes_no_tools(self):
|
||||
from headroom.integrations import HeadroomChatModel
|
||||
|
||||
_RecordingModel.last_kwargs = {}
|
||||
inner = _RecordingModel(messages=iter([AIMessage("ok")]))
|
||||
HeadroomChatModel(inner).invoke("hello")
|
||||
|
||||
assert "tools" not in _RecordingModel.last_kwargs
|
||||
|
||||
def test_unwrap_binding_ignores_non_bindings(self):
|
||||
from headroom.integrations import HeadroomChatModel
|
||||
|
||||
inner = GenericFakeChatModel(messages=iter([AIMessage("ok")]))
|
||||
model, kwargs = HeadroomChatModel._unwrap_binding(inner)
|
||||
|
||||
assert model is inner
|
||||
assert kwargs == {}
|
||||
|
||||
|
||||
class TestWrappedToolIsUsable:
|
||||
"""Defects 2 and 3: the wrapped tool must invoke, and keep its schema."""
|
||||
"""Defects 1 and 2: the wrapped tool must invoke, and keep its schema."""
|
||||
|
||||
def test_invoke_with_keyword_arguments(self):
|
||||
from headroom.integrations import wrap_tools_with_headroom
|
||||
|
||||
@@ -118,6 +118,11 @@ class TestLiteLLMLogSuppression:
|
||||
"""The env flag must exist before litellm itself is imported."""
|
||||
import os
|
||||
|
||||
import headroom.providers as providers
|
||||
|
||||
# Import also replaces the parent package attribute. Track it even when
|
||||
# initially absent so teardown restores or removes the temporary module.
|
||||
monkeypatch.setattr(providers, "litellm", None, raising=False)
|
||||
monkeypatch.delenv("LITELLM_SUPPRESS_DEBUG_INFO", raising=False)
|
||||
# delitem, not pop: teardown puts the real modules back. A bare pop made
|
||||
# every later `import litellm` load a second copy, so a test pinning a
|
||||
|
||||
@@ -72,6 +72,10 @@ def test_xai_cache_savings_and_billing_use_their_respective_price_sources(
|
||||
}
|
||||
if catalog_read is not None:
|
||||
row["cache_read_input_token_cost"] = catalog_read
|
||||
original_row = litellm.model_cost[model]
|
||||
# Register through the public API; keep the isolated row free of inherited rates.
|
||||
monkeypatch.setitem(litellm.model_cost, model, dict(row))
|
||||
litellm.register_model({model: dict(row)})
|
||||
monkeypatch.setitem(litellm.model_cost, model, row)
|
||||
resolve_rates.cache_clear()
|
||||
try:
|
||||
@@ -83,21 +87,19 @@ def test_xai_cache_savings_and_billing_use_their_respective_price_sources(
|
||||
|
||||
# A provider-wide counterfactual estimate is not a billed cache rate.
|
||||
# Use the real calculator for the same reported usage, as billing does.
|
||||
if catalog_read is None:
|
||||
canonical_input, _ = litellm.cost_per_token(
|
||||
model=model,
|
||||
prompt_tokens=10_000,
|
||||
completion_tokens=0,
|
||||
cache_read_input_tokens=10_000,
|
||||
)
|
||||
else:
|
||||
canonical_input = 10_000 * catalog_read
|
||||
canonical_input, _ = litellm.cost_per_token(
|
||||
model=model,
|
||||
prompt_tokens=10_000,
|
||||
completion_tokens=0,
|
||||
cache_read_input_tokens=10_000,
|
||||
)
|
||||
tracker = CostTracker()
|
||||
tracker.record_tokens(model, tokens_saved=0, tokens_sent=10_000, cache_read_tokens=10_000)
|
||||
assert tracker.totals()[1] == pytest.approx(canonical_input)
|
||||
assert tracker.stats()["total_input_cost_usd"] == pytest.approx(canonical_input)
|
||||
finally:
|
||||
resolve_rates.cache_clear()
|
||||
litellm.register_model({model: original_row})
|
||||
|
||||
|
||||
@pytest.mark.parametrize("user_agent", ["grok/1.2.3", "grok-shell/0.2.112"])
|
||||
|
||||
Reference in New Issue
Block a user