mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(offline): keep litellm's pricing table local under HEADROOM_OFFLINE (#4242)
## Description
`HEADROOM_OFFLINE=1` still lets litellm reach GitHub. `import litellm`
downloads `model_prices_and_context_window.json` from
`raw.githubusercontent.com` during the import (three attempts) unless
`LITELLM_LOCAL_MODEL_COST_MAP` is `true`. Headroom imports litellm from
24 places, among them the pricing helpers, the LiteLLM provider and
backend, the proxy's cost and savings trackers, and both Headroom Cloud
integrations. With the air-gap switch on, the first of those imports
dialled out.
Nothing in the chokepoint suite caught it, because no Headroom-built
client is involved, so no `guard_egress` site can dominate it.
`TestHeadroomCloudCompressionOffline` only passed when an earlier test
had already imported litellm. Run on its own (`pytest
tests/test_offline_egress_chokepoint.py -k HeadroomCloud`), it fails on
`main` with `SocketOpened: ... ('raw.githubusercontent.com', 443)`.
The only lever is the env var, and litellm reads it once, during its own
import. So:
- `pin_litellm_cost_map_local()` in `headroom/offline.py` sets
`LITELLM_LOCAL_MODEL_COST_MAP=True` while `HEADROOM_OFFLINE` is on.
- Every litellm import in `headroom/` calls it first, in the same scope.
A pin only at package import would miss a switch turned on later:
`is_offline()` is read when checked, and
`create_app(ProxyConfig(offline=True))` sets it after `import headroom`.
- The pin is Headroom's, so it is undone: called with the switch off, it
restores the value it replaced if the variable still holds the pin. A
process that leaves offline mode before its first litellm import gets
the online pricing refresh back.
- It overwrites the variable rather than using `setdefault`. litellm
only accepts `true`, so an operator value of `1` (the form every
Headroom switch takes), `False` or an empty string would otherwise
reopen the fetch.
- A meta-test fails the build for any litellm import without a preceding
pin, the same pattern as the existing site scanner.
No tracking issue; found while chasing a test that only failed when run
on its own.
Review found a second import-time leak on the same path:
`headroom/integrations/__init__.py` imports the CrewAI integration
whenever CrewAI is installed, and `import crewai` requests a Scarf pixel
from `api.scarf.sh` on a background thread unless
`CREWAI_DISABLE_TELEMETRY` is `true`. So `import
headroom.integrations.litellm_callback` still dialled out under the
switch. `pin_crewai_telemetry_off()` handles it the same way, through a
shared `_OfflineEnvPin`, and the meta-test now covers `crewai` imports
too.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/offline.py`: new `pin_litellm_cost_map_local()` (a no-op
unless `HEADROOM_OFFLINE` is on), documented in the module's list of
refused egress.
- All 24 litellm import sites call it immediately before importing:
`pricing/`, `providers/` (litellm, anthropic, openai, cohere, google),
`backends/litellm.py`, `proxy/` (cost, savings_tracker, route_advice),
`integrations/litellm_callback.py`, `telemetry/session.py`,
`perf/analyzer.py`, `learn/analyzer.py`, `cli/` (evals, learn) and
`evals/`.
- `pin_crewai_telemetry_off()` forces `CREWAI_DISABLE_TELEMETRY=true`
while offline (restored the same way), called before the one `crewai`
import (`headroom/integrations/crewai/agents.py`). Both pins share
`_OfflineEnvPin`.
- `headroom/__init__.py` also calls it, which covers a host that imports
litellm itself after importing Headroom.
- `tests/test_offline_egress_chokepoint.py`, new
`TestLiteLLMCostMapOffline`:
- subprocess probes (litellm only fetches on its first import, which an
earlier test usually did already) for four litellm-backed modules, with
the switch set both before and after `import headroom`;
- operator values `1`, `False` and empty;
- leaving offline mode restores the replaced value (absent or `False`),
and an operator change made after the pin is kept;
- `test_every_litellm_import_is_pinned_first`, the AST meta-test;
- the `offline` fixture now restores `LITELLM_LOCAL_MODEL_COST_MAP` at
teardown.
- `TestCrewAITelemetryOffline`: a stand-in `crewai` package with crewai
1.6.1's import-time tracking (no extra or CI job installs CrewAI),
probed early and late for `headroom.integrations.crewai` and
`headroom.integrations.litellm_callback`, plus operator values, a
restore test, the same probes against a real CrewAI when installed, and
`test_every_crewai_import_is_pinned_first`. A sanity test checks that
the stand-in does dial when online.
- `docs/metrics-technical-guide.md` and `docs/content/docs/proxy.mdx`
add the refresh and CrewAI's tracking to the refused list. The guide
also states the limits: a host that imports litellm *before* Headroom
has already made the fetch and must set the variable itself, and offline
prices come from the table bundled with the installed litellm release.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py
216 passed, 3 skipped, 1 warning in 38.44s
$ pytest -q --continue-on-collection-errors tests/test_providers tests/test_proxy tests/test_pricing* tests/test_perf* \
tests/test_telemetry* tests/test_learn* tests/test_route_advice* tests/test_backend* tests/test_litellm* tests/test_cost* \
tests/test_savings* tests/test_counterfactual* tests/test_evals* tests/test_integrations*
1742 passed, 337 skipped, 6 warnings, 1 error in 65.33s
# the 1 error is test_langchain_1x_compat.py failing collection (NameError: tool) because langchain_core
# is not installed in my venv; it fails identically on main df11f274
$ ruff check headroom tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py # ruff 0.17.0
All checks passed!
$ ruff format --check headroom tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py
573 files already formatted
$ mypy <the 23 changed headroom/ files>
Success: no issues found in 23 source files
# The new tests fail without the fix: removing one pin call, and reverting to setdefault, gives
FAILED ...::test_operator_value_cannot_reopen_the_fetch[1] assert ['raw.githubusercontent.com', ...] == []
FAILED ...::test_operator_value_cannot_reopen_the_fetch[False]
FAILED ...::test_operator_value_cannot_reopen_the_fetch[]
FAILED ...::test_every_litellm_import_is_pinned_first names the unpinned import in headroom/backends/litellm.py
```
## Real Behavior Proof
- Environment: macOS arm64, Python 3.10.18 venv, litellm 1.101.0 (the
`uv.lock` version), `main` at df11f274 in a worktree vs this branch, the
same venv for both, `LITELLM_LOCAL_MODEL_COST_MAP` unset unless shown.
- Exact command / steps: a probe script replaces
`socket.create_connection` with a trap that records the address and
raises, runs `import headroom`, sets `HEADROOM_OFFLINE=1` before
(`early`) or after (`late`) that import, then imports the module and
prints the recorded attempts. Run with `HEADROOM_OFFLINE=1 python
probe2.py early headroom.integrations.asgi`, then the same for
`headroom.backends.litellm`, then `python probe2.py late
headroom.backends.litellm`, then `HEADROOM_OFFLINE=1
LITELLM_LOCAL_MODEL_COST_MAP=1 python probe2.py early
headroom.providers.litellm`.
- Observed result: main df11f274 gives `attempts=3
['raw.githubusercontent.com:443']` for all four runs; this branch gives
`attempts=0 []` for all four, with litellm loaded in every case.
- CrewAI: in a venv with crewai 1.6.1 + litellm 1.101.0 (the review
setup) and CrewAI telemetry at its default, `pytest
tests/test_offline_egress_chokepoint.py -k 'LiteLLMCostMap or CrewAI'`
gives 28 passed, including both real-CrewAI imports. Without the
`agents.py` pin, 10 CrewAI checks fail with `attempts=['api.scarf.sh']`.
- Not tested: a live proxy on an air-gapped host. The probe covers the
import paths; the proxy's lazy imports (`proxy/cost.py`,
`proxy/savings_tracker.py`, `proxy/route_advice.py`) are covered by the
meta-test, not by a running proxy.
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: only with HEADROOM_OFFLINE set;
litellm then uses its bundled pricing table, which is what it already
fell back to after the three failed fetches. Without the switch nothing
changes.
- Kill switch / disable path: unset HEADROOM_OFFLINE.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert this PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
- A host process that imports litellm before Headroom (a LiteLLM proxy
that loads the Headroom callback) has already fetched by the time any
Headroom code runs. That connection isn't Headroom's, so the docs state
the condition instead of claiming it.
- `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE` keep their `setdefault` in
`apply_offline_env()`: the Hub path is protected by `guard_egress` in
`onnx_runtime.py`, not by those variables.
- Possible follow-up, not in this PR: show which cost map litellm loaded
(litellm's `get_model_cost_map_source_info()`) in the offline startup
banner or `headroom doctor`.
---------
Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Kenneth Wong
JerrettDavis
parent
8c07cf4a0f
commit
3fef15d376
@@ -433,7 +433,7 @@ with a privileged job.
|
||||
| `HEADROOM_PROXY_TOKEN` | none | Require a token from non-loopback callers, as `X-Headroom-Proxy-Token: <token>` or `Authorization: Bearer <token>`. The token is removed from the request before it is forwarded upstream; clients that also send a provider credential should use `X-Headroom-Proxy-Token` and keep `Authorization` for the provider. Also gates the read-only operator routes (`/stats-history`, `/quota`, `/subscription-window`, `/metrics`), which otherwise answer loopback and trusted-CIDR callers only. |
|
||||
| `HEADROOM_ALLOW_UNAUTHENTICATED_BIND` | `false` | Acknowledge a non-loopback bind (`--host 0.0.0.0`, `::`, a LAN address) with **no** `HEADROOM_PROXY_TOKEN`. Without a token or this flag the proxy refuses to start, because `/v1/*` would relay upstream with the operator's credentials for every peer that can reach the port. Only set it when the runtime already restricts reachability (a container published on `127.0.0.1`). |
|
||||
| `HEADROOM_COMPRESS_ALLOW_REMOTE` | `false` | Allow non-loopback callers to reach [`POST /v1/compress`](#post-v1compress) and `POST /v1/usage`. Required to run Headroom as a remote gateway/sidecar; without it remote callers get `404`. |
|
||||
| `HEADROOM_OFFLINE` | `false` | Air-gap mode: refuse every connection Headroom itself initiates — telemetry, update checks, license reporting, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP and Langfuse export, Copilot auth, the subscription pollers, the OpenAI embedders, Headroom Cloud compression and the TLS diagnostics (`headroom doctor --network` reports them as skipped). Four things still work on purpose: forwarding your own requests to the upstream you configured, operator-configured local endpoints (the Ollama embedder), loopback health probes, and paths an `is_offline()` check already disables. Refusals surface as messages, not tracebacks. See the air-gap notes in `docs/metrics-technical-guide.md`. |
|
||||
| `HEADROOM_OFFLINE` | `false` | Air-gap mode: refuse every connection Headroom itself initiates — telemetry, update checks, license reporting, model/tokenizer/binary/dataset downloads, litellm's pricing-table refresh, CrewAI's install tracking, remote Kompress, OTLP and Langfuse export, Copilot auth, the subscription pollers, the OpenAI embedders, Headroom Cloud compression and the TLS diagnostics (`headroom doctor --network` reports them as skipped). Four things still work on purpose: forwarding your own requests to the upstream you configured, operator-configured local endpoints (the Ollama embedder), loopback health probes, and paths an `is_offline()` check already disables. Refusals surface as messages, not tracebacks. See the air-gap notes in `docs/metrics-technical-guide.md`. |
|
||||
| `HEADROOM_LICENSE` | unset | Licence token, read by core and every licensed extension. `HEADROOM_LICENSE_KEY` is a deprecated alias for one release and logs a warning. A licence on its own sends nothing to Headroom. |
|
||||
| `HEADROOM_USAGE_REPORTING` | `false` | Set `1` to let the proxy validate the licence and post aggregate usage counts (no content) to the Headroom cloud. Off unless set; `HEADROOM_OFFLINE` still overrides it. |
|
||||
| `--stateless` / `HEADROOM_STATELESS` | `false` | Keep all state in memory; no filesystem writes (disables logs, memory, TOIN). |
|
||||
|
||||
@@ -268,7 +268,7 @@ Verify with `curl -s localhost:8787/stats | jq .otel`.
|
||||
|
||||
**Multi-tenant labels:** `register_otel_metric_attribute_provider()` adds request-scoped attributes (tenant, team, cost centre) to every OTel datapoint. Max 16 attributes, 256 chars each.
|
||||
|
||||
**Air-gapped deployments:** `HEADROOM_OFFLINE=1` refuses every connection Headroom itself decides to make to a destination Headroom itself chose. That is the whole list, not a sample: the anonymous usage beacon (which is **on by default**), the update check, the license/usage reporter, **OTLP metric and Langfuse trace export**, HuggingFace / Kompress / fastembed model and tokenizer downloads (Python and Rust), the remote Kompress endpoint, the `headroom install` release-binary and codebase-memory-mcp downloads, the BFCL eval-dataset fetch and the provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI embedders, the Headroom Cloud compression modes in the ASGI and LiteLLM integrations, and the TLS diagnostics (`headroom doctor --network` endpoint checks and the certificate-chain re-probe after an upstream TLS failure).
|
||||
**Air-gapped deployments:** `HEADROOM_OFFLINE=1` refuses every connection Headroom itself decides to make to a destination Headroom itself chose. That is the whole list, not a sample: the anonymous usage beacon (which is **on by default**), the update check, the license/usage reporter, **OTLP metric and Langfuse trace export**, HuggingFace / Kompress / fastembed model and tokenizer downloads (Python and Rust), the remote Kompress endpoint, the `headroom install` release-binary and codebase-memory-mcp downloads, litellm's import-time pricing-table refresh, CrewAI's import-time install tracking, the BFCL eval-dataset fetch and the provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI embedders, the Headroom Cloud compression modes in the ASGI and LiteLLM integrations, and the TLS diagnostics (`headroom doctor --network` endpoint checks and the certificate-chain re-probe after an upstream TLS failure).
|
||||
|
||||
Four things are deliberately still allowed, and they are the complete set of exceptions:
|
||||
|
||||
@@ -279,6 +279,8 @@ Four things are deliberately still allowed, and they are the complete set of exc
|
||||
|
||||
Those four are enumerated with written reasons in `_EGRESS_ALLOWLIST` in `tests/test_offline_egress_chokepoint.py`, which fails the build if a new egress path appears that is neither guarded nor one of them. There is no category for "known violation": a path that can dial the internet with the flag set is a bug. A refusal reaches you as a message — a Click error on the CLI, a named "model unavailable" from a model loader, a logged line from a background poller — never as a traceback and never as a silent degradation. Enforcing the same policy at the network layer as well is still good practice; it is no longer the only thing standing between you and Headroom-initiated egress.
|
||||
|
||||
litellm's pricing-table refresh is refused by setting `LITELLM_LOCAL_MODEL_COST_MAP=True` before litellm is imported, so it holds only when Headroom is the first thing in the process to import litellm. A host that imports litellm before Headroom, such as a LiteLLM proxy loading the Headroom callback, has made that fetch already and must set the variable itself. Offline, cost and savings figures use the pricing table bundled with the installed litellm release, so a model released after it prices as unknown.
|
||||
|
||||
OTLP export is refused loudly: with `HEADROOM_OFFLINE=1` set, `HEADROOM_OTEL_METRICS_ENABLED=1` plus the default `otlp_http` exporter makes the proxy exit 78 at startup with an explanation, rather than quietly dropping metrics. There is no exemption for a collector that looks local — an in-cluster address is not reliably distinguishable from an internet one. For metrics under an air-gap, either scrape `localhost:8787/metrics` or set `HEADROOM_OTEL_METRICS_EXPORTER=console`, both of which stay on-box. `HEADROOM_KOMPRESS_ENDPOINT` and `HEADROOM_LANGFUSE_ENABLED` are refused the same way, for the same reason.
|
||||
|
||||
---
|
||||
|
||||
@@ -75,6 +75,7 @@ from typing import Any
|
||||
|
||||
from ._ort import ensure_ort_dylib_pinned
|
||||
from ._version import __version__ # noqa: F401
|
||||
from .offline import pin_litellm_cost_map_local
|
||||
|
||||
# Must run before anything can import `headroom._core`: on Windows the
|
||||
# Rust core resolves onnxruntime.dll at runtime (ort load-dynamic), and
|
||||
@@ -83,6 +84,12 @@ from ._version import __version__ # noqa: F401
|
||||
# ~microseconds. See `headroom/_ort.py` for the full story.
|
||||
ensure_ort_dylib_pinned()
|
||||
|
||||
# Under HEADROOM_OFFLINE, litellm must not fetch its pricing table from GitHub
|
||||
# at import time. Each litellm import in this package pins that itself; this
|
||||
# call also covers a host that imports litellm after Headroom. A no-op unless
|
||||
# the switch is set. See `headroom/offline.py`.
|
||||
pin_litellm_cost_map_local()
|
||||
|
||||
from .compress import ( # noqa: E402
|
||||
CompressConfig,
|
||||
CompressResult,
|
||||
|
||||
@@ -21,6 +21,7 @@ from collections.abc import AsyncIterator
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
from headroom.proxy.public_errors import client_message
|
||||
from headroom.utils import format_exception_message
|
||||
|
||||
@@ -67,6 +68,7 @@ try:
|
||||
import os as _os
|
||||
|
||||
_env_snapshot = set(_os.environ)
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
from litellm import acompletion
|
||||
from litellm.utils import supports_prompt_caching
|
||||
@@ -536,6 +538,7 @@ def _caller_key_travels_to(model: str, key: str) -> bool:
|
||||
if not key.startswith("sk-ant-"):
|
||||
return True
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
from litellm import get_llm_provider
|
||||
|
||||
provider = (get_llm_provider(model)[1] or "").lower()
|
||||
|
||||
@@ -7,6 +7,8 @@ from pathlib import Path
|
||||
|
||||
import click
|
||||
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
|
||||
from .main import main
|
||||
|
||||
|
||||
@@ -480,6 +482,7 @@ def _run_memory_eval(
|
||||
answer_fn = None
|
||||
if answer_model:
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
|
||||
def answer_fn(question: str, memories: list[str]) -> str:
|
||||
|
||||
@@ -7,6 +7,8 @@ from typing import TYPE_CHECKING, Any, NamedTuple
|
||||
|
||||
import click
|
||||
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ..learn.base import LearnPlugin
|
||||
|
||||
@@ -390,6 +392,7 @@ def _make_llm_judge(model: str) -> Any:
|
||||
try:
|
||||
import json
|
||||
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
except ImportError:
|
||||
return None
|
||||
|
||||
@@ -15,7 +15,7 @@ from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
from headroom.offline import guard_egress, pin_litellm_cost_map_local
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from headroom.transforms.html_extractor import HTMLExtractor
|
||||
@@ -316,6 +316,7 @@ class HTMLExtractionEvaluator:
|
||||
|
||||
else:
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
|
||||
def judge(question: str, ground_truth: str, prediction: str) -> tuple[float, str]:
|
||||
@@ -396,6 +397,7 @@ Answer concisely and factually based only on the content provided."""
|
||||
)
|
||||
|
||||
else:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
|
||||
litellm_response = litellm.completion(
|
||||
|
||||
@@ -17,7 +17,7 @@ from __future__ import annotations
|
||||
import logging
|
||||
from collections.abc import Callable
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
from headroom.offline import guard_egress, pin_litellm_cost_map_local
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -169,6 +169,7 @@ def create_litellm_judge(
|
||||
score, reasoning = judge_fn(question, ground_truth, prediction)
|
||||
"""
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
except ImportError as e:
|
||||
raise ImportError(
|
||||
|
||||
@@ -30,6 +30,7 @@ from headroom.evals.memory.locomo import (
|
||||
)
|
||||
from headroom.evals.metrics import compute_exact_match, compute_f1
|
||||
from headroom.memory import HierarchicalMemory, MemoryConfig
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -277,6 +278,7 @@ class LoCoMoEvaluator:
|
||||
List of extracted memory dicts with 'content' and 'category'.
|
||||
"""
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
except ImportError:
|
||||
logger.warning("litellm not available, falling back to raw dialogue storage")
|
||||
|
||||
@@ -24,8 +24,6 @@ from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import litellm
|
||||
|
||||
from headroom.evals.memory.locomo import (
|
||||
LOCOMO_CATEGORIES,
|
||||
LoCoMoCase,
|
||||
@@ -37,6 +35,12 @@ from headroom.memory.backends.local import LocalBackend, LocalBackendConfig
|
||||
from headroom.memory.models import Memory
|
||||
from headroom.memory.system import MemoryBackend
|
||||
from headroom.memory.tools import MEMORY_TOOLS
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
|
||||
# Before the import: litellm fetches its pricing table at import time unless
|
||||
# told not to, and HEADROOM_OFFLINE forbids that fetch.
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm # noqa: E402
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@@ -27,7 +27,10 @@ from __future__ import annotations
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import pin_crewai_telemetry_off
|
||||
|
||||
try:
|
||||
pin_crewai_telemetry_off()
|
||||
from crewai.tools.base_tool import BaseTool
|
||||
|
||||
CREWAI_AVAILABLE = True
|
||||
|
||||
@@ -24,12 +24,13 @@ from headroom.integrations._compress_backend import ( # noqa: F401
|
||||
_DEFAULT_CLOUD_URL,
|
||||
CompressBackend,
|
||||
)
|
||||
from headroom.offline import guard_egress # noqa: F401
|
||||
from headroom.offline import guard_egress, pin_litellm_cost_map_local # noqa: F401
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
from litellm.integrations.custom_logger import CustomLogger as _CustomLogger
|
||||
except ImportError: # litellm not installed — fall back to plain object
|
||||
_CustomLogger = object # type: ignore[assignment,misc]
|
||||
|
||||
@@ -25,6 +25,7 @@ import time
|
||||
import typing
|
||||
|
||||
from headroom._subprocess import Popen, run
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
|
||||
from .loops import LoopPattern, apply_loop_weighting, detect_loops, format_loops_for_digest
|
||||
from .models import (
|
||||
@@ -1095,6 +1096,7 @@ def _call_llm(
|
||||
if model in _CLI_MODEL_IDS:
|
||||
return _call_cli_llm(digest, model, on_progress=on_progress)
|
||||
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
|
||||
# Suppress LiteLLM's verbose logging
|
||||
|
||||
+84
-2
@@ -8,8 +8,9 @@ deployment exists to carry.
|
||||
Refused (this is not a sample — it is the rule, and the meta-test enforces it):
|
||||
the telemetry beacon, the update check, the license/usage reporter, OTLP and
|
||||
Langfuse export, HuggingFace/Kompress/fastembed model downloads, release-binary
|
||||
downloads (codebase-memory-mcp among them), eval dataset downloads and the provider SDK
|
||||
clients the eval harness drives, GitHub Copilot device-flow auth and token
|
||||
downloads (codebase-memory-mcp among them), litellm's import-time pricing-table
|
||||
refresh, CrewAI's import-time install tracking, eval dataset downloads and the
|
||||
provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token
|
||||
exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI
|
||||
embedders, both Headroom Cloud compression integrations, and the TLS
|
||||
diagnostics (``doctor --network`` endpoint checks and the certificate-chain
|
||||
@@ -101,6 +102,87 @@ def apply_offline_env() -> None:
|
||||
os.environ.setdefault("TRANSFORMERS_OFFLINE", "1")
|
||||
|
||||
|
||||
class _OfflineEnvPin:
|
||||
"""An environment variable Headroom forces while ``HEADROOM_OFFLINE`` is
|
||||
on, for a third-party library that dials out during its own import and
|
||||
only listens to that variable.
|
||||
|
||||
Overwritten rather than ``setdefault``: these libraries accept only
|
||||
``true``, so an operator ``1``, ``False`` or empty value would reopen a
|
||||
connection the air-gap switch exists to refuse. The pin is Headroom's, so
|
||||
it is also undone: called with the switch off, it puts back the value it
|
||||
replaced, provided the variable still holds the pin. An operator's own
|
||||
value is never touched outside offline mode.
|
||||
"""
|
||||
|
||||
def __init__(self, name: str, value: str) -> None:
|
||||
self.name = name
|
||||
self.value = value
|
||||
# Unset while no pin is in place; otherwise a 1-tuple of the value the
|
||||
# pin replaced (None when the variable was absent).
|
||||
self.before: tuple[str | None] | None = None
|
||||
|
||||
def __call__(self) -> None:
|
||||
if is_offline():
|
||||
if self.before is None:
|
||||
self.before = (os.environ.get(self.name),)
|
||||
os.environ[self.name] = self.value
|
||||
elif self.before is not None:
|
||||
(before,) = self.before
|
||||
self.before = None
|
||||
if os.environ.get(self.name) == self.value:
|
||||
if before is None:
|
||||
os.environ.pop(self.name, None)
|
||||
else:
|
||||
os.environ[self.name] = before
|
||||
|
||||
|
||||
LITELLM_LOCAL_COST_MAP_ENV = "LITELLM_LOCAL_MODEL_COST_MAP"
|
||||
_litellm_cost_map_pin = _OfflineEnvPin(LITELLM_LOCAL_COST_MAP_ENV, "True")
|
||||
|
||||
CREWAI_TELEMETRY_ENV = "CREWAI_DISABLE_TELEMETRY"
|
||||
_crewai_telemetry_pin = _OfflineEnvPin(CREWAI_TELEMETRY_ENV, "true")
|
||||
|
||||
|
||||
def pin_litellm_cost_map_local() -> None:
|
||||
"""Keep litellm on its bundled pricing table under ``HEADROOM_OFFLINE``.
|
||||
|
||||
``import litellm`` fetches ``model_prices_and_context_window.json`` from
|
||||
raw.githubusercontent.com at import time (three attempts) unless
|
||||
``LITELLM_LOCAL_MODEL_COST_MAP`` is ``true``. That is not a client Headroom
|
||||
builds, so no ``guard_egress`` site can dominate it; the only lever is the
|
||||
env var, read once during that import. So every ``import litellm`` in
|
||||
``headroom/`` calls this first, in the same scope, because the switch is
|
||||
read when it is checked rather than when the package loads
|
||||
(``create_app(ProxyConfig(offline=True))`` turns it on after import). The
|
||||
chokepoint meta-test fails the build for an import that does not.
|
||||
``headroom/__init__.py`` also calls it, which covers a host process that
|
||||
imports litellm itself after importing Headroom; a host that imports
|
||||
litellm *before* Headroom has made that fetch already and must set the
|
||||
variable itself.
|
||||
|
||||
Forced and given back as described on :class:`_OfflineEnvPin`, so a
|
||||
process that leaves offline mode before its first litellm import gets the
|
||||
online pricing refresh again.
|
||||
"""
|
||||
_litellm_cost_map_pin()
|
||||
|
||||
|
||||
def pin_crewai_telemetry_off() -> None:
|
||||
"""Keep CrewAI's install tracking quiet under ``HEADROOM_OFFLINE``.
|
||||
|
||||
``import crewai`` starts a thread that requests a Scarf pixel from
|
||||
api.scarf.sh unless ``CREWAI_DISABLE_TELEMETRY`` (or ``OTEL_SDK_DISABLED``
|
||||
/ ``CREWAI_DISABLE_TRACKING``) is ``true``. Headroom imports CrewAI
|
||||
whenever it is installed, through ``headroom.integrations``, so even a
|
||||
LiteLLM callback import under the air-gap switch dialled out. As with
|
||||
litellm, the variable is the only lever, so every ``crewai`` import in
|
||||
``headroom/`` calls this first (the chokepoint meta-test enforces it), and
|
||||
it is forced and given back as described on :class:`_OfflineEnvPin`.
|
||||
"""
|
||||
_crewai_telemetry_pin()
|
||||
|
||||
|
||||
class OfflineEgressBlocked(BaseException):
|
||||
"""Raised by :func:`guard_egress` when ``HEADROOM_OFFLINE`` is in force.
|
||||
|
||||
|
||||
@@ -20,6 +20,7 @@ from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
from headroom import paths as _paths
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
from headroom.pricing.litellm_pricing import resolve_litellm_model
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
@@ -74,6 +75,7 @@ _STAGE_TIMINGS_RE = re.compile(
|
||||
# directly instead of maintaining our own pricing tables.
|
||||
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm as _litellm
|
||||
|
||||
_LITELLM_AVAILABLE = True
|
||||
|
||||
@@ -63,6 +63,7 @@ from typing import Any
|
||||
|
||||
from headroom.cache_economics import CACHE_ECONOMICS
|
||||
from headroom.log_safety import describe_exception, safe_id
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
from headroom.pricing.cache_ttl import CACHE_WRITE_MULTIPLIERS
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -338,6 +339,7 @@ def _litellm() -> Any | None:
|
||||
deployments often run without it. Callers degrade to list pricing.
|
||||
"""
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
except Exception: # pragma: no cover - environment-dependent
|
||||
return None
|
||||
|
||||
@@ -15,6 +15,7 @@ from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
from headroom.pricing.deepseek_tiers import (
|
||||
LEGACY_MODEL_IDS,
|
||||
OFF_PEAK_RATES_PER_1M,
|
||||
@@ -34,6 +35,11 @@ from headroom.pricing.litellm_model_resolution import (
|
||||
# litellm added. The module itself is fully imported and cached in
|
||||
# `sys.modules`; subsequent `import litellm` calls hit the cache and
|
||||
# don't re-run the dotenv side effect.
|
||||
#
|
||||
# Under HEADROOM_OFFLINE, litellm must also stay on its bundled pricing table
|
||||
# rather than fetch one from GitHub during that import.
|
||||
pin_litellm_cost_map_local()
|
||||
|
||||
try:
|
||||
import os as _os
|
||||
|
||||
|
||||
@@ -25,6 +25,7 @@ from datetime import datetime
|
||||
from typing import Any, cast
|
||||
|
||||
from headroom import paths as _paths
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
from headroom.pricing.deepseek_tiers import (
|
||||
LEGACY_MODEL_IDS,
|
||||
OFF_PEAK_RATES_PER_1M,
|
||||
@@ -49,6 +50,7 @@ def _get_litellm_clients() -> tuple[Any | None, Any | None]:
|
||||
return None, None
|
||||
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
|
||||
litellm.suppress_debug_info = True
|
||||
|
||||
@@ -22,12 +22,14 @@ from datetime import date
|
||||
from typing import Any
|
||||
|
||||
from headroom.log_safety import describe_exception, safe_id
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
from headroom.pricing.litellm_pricing import estimate_cost_from_tokens
|
||||
from headroom.tokenizers import EstimatingTokenCounter
|
||||
|
||||
from .base import Provider, TokenCounter
|
||||
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
|
||||
LITELLM_AVAILABLE = True
|
||||
|
||||
@@ -26,6 +26,7 @@ from datetime import date
|
||||
from typing import Any
|
||||
|
||||
from headroom.models.registry import ModelRegistry
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
from headroom.pricing.litellm_pricing import estimate_cost_from_tokens
|
||||
from headroom.tokenizers import EstimatingTokenCounter
|
||||
|
||||
@@ -33,6 +34,7 @@ from .base import Provider, TokenCounter
|
||||
|
||||
# Check if litellm is available for pricing/context limit lookups
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
|
||||
LITELLM_AVAILABLE = True
|
||||
|
||||
@@ -25,6 +25,7 @@ import logging
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
from headroom.pricing.litellm_pricing import estimate_cost_from_tokens
|
||||
from headroom.tokenizers import EstimatingTokenCounter
|
||||
|
||||
@@ -38,6 +39,7 @@ try:
|
||||
# module-level suppression flags below can be set.
|
||||
os.environ.setdefault("LITELLM_SUPPRESS_DEBUG_INFO", "True")
|
||||
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
|
||||
# Suppress litellm's startup banner ("Provider List: https://...") and
|
||||
|
||||
@@ -17,6 +17,7 @@ from typing import Any, cast
|
||||
|
||||
from headroom import paths as _paths
|
||||
from headroom.log_safety import describe_exception, safe_id
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
from headroom.pricing.litellm_pricing import estimate_cost_from_tokens
|
||||
from headroom.tokenizers.base import (
|
||||
TokenCountCache,
|
||||
@@ -49,6 +50,7 @@ def _get_litellm_module() -> Any | None:
|
||||
return None
|
||||
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
except ImportError:
|
||||
return None
|
||||
|
||||
@@ -17,6 +17,7 @@ from typing import TYPE_CHECKING, Any, NamedTuple
|
||||
|
||||
from headroom.cache_economics import CACHE_ECONOMICS as _CACHE_ECONOMICS
|
||||
from headroom.log_safety import WarnOnce, describe_exception, safe_id
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
from headroom.proxy.budget_basis_policy import (
|
||||
BUDGET_BASIS_BLOCK,
|
||||
BUDGET_BASIS_IGNORE,
|
||||
@@ -50,6 +51,7 @@ def _get_litellm_module() -> Any | None:
|
||||
return litellm
|
||||
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm as imported_litellm
|
||||
except ImportError:
|
||||
return None
|
||||
|
||||
@@ -34,6 +34,8 @@ import logging
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
STATE_ATTR = "headroom_route"
|
||||
@@ -202,6 +204,7 @@ def resolver_for(handler: Any) -> BackendResolver:
|
||||
def _known_provider(provider: str) -> bool:
|
||||
"""Is this a provider litellm actually knows about?"""
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
|
||||
names = {getattr(p, "value", None) or str(p) for p in getattr(litellm, "provider_list", [])}
|
||||
@@ -215,6 +218,7 @@ def _provider_of(model: str) -> str:
|
||||
if "/" in model:
|
||||
return model.split("/", 1)[0]
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
|
||||
entry = litellm.model_cost.get(model) or {}
|
||||
|
||||
@@ -25,6 +25,7 @@ from typing import Any
|
||||
|
||||
from headroom import paths as _paths
|
||||
from headroom.log_safety import WarnOnce, describe_exception, safe_id
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
from headroom.proxy import project_name_policy
|
||||
from headroom.proxy.persistent_metrics import (
|
||||
MAX_LABEL_LENGTH,
|
||||
@@ -156,6 +157,7 @@ def _get_litellm_module() -> Any | None:
|
||||
return litellm
|
||||
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm as imported_litellm
|
||||
except ImportError:
|
||||
return None
|
||||
|
||||
@@ -118,6 +118,8 @@ from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Where session events go. The receiver is open source: deploy/beacon/worker.js.
|
||||
@@ -532,6 +534,7 @@ def _public_model(model: str) -> str | None:
|
||||
|
||||
resolved: str | None = None
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
|
||||
registry = litellm.model_cost
|
||||
|
||||
@@ -191,6 +191,17 @@ class TestSocketTrap:
|
||||
@pytest.fixture
|
||||
def offline(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
|
||||
# An import of litellm or CrewAI under the switch writes its pin variable
|
||||
# directly, out of monkeypatch's sight. Record each variable (setenv saves
|
||||
# the original, delenv then clears it) so teardown puts it back...
|
||||
import headroom.offline
|
||||
|
||||
for pin in (headroom.offline._litellm_cost_map_pin, headroom.offline._crewai_telemetry_pin):
|
||||
monkeypatch.setenv(pin.name, "")
|
||||
monkeypatch.delenv(pin.name)
|
||||
# ...and the pin's memory of the value it replaced, which would
|
||||
# otherwise outlive the variable it describes.
|
||||
monkeypatch.setattr(pin, "before", None)
|
||||
|
||||
|
||||
# ``"example.com" in url`` is true of ``https://example.com.attacker.test``
|
||||
@@ -415,6 +426,374 @@ class TestLangfuseExporterOffline:
|
||||
assert configure_langfuse_tracing(LangfuseTracingConfig(enabled=False)) is not None
|
||||
|
||||
|
||||
# ────────────── path 2c: litellm's import-time pricing fetch ────────────────
|
||||
|
||||
# Runs in a fresh interpreter: the fetch happens once, on the first
|
||||
# ``import litellm``, and inside this pytest process an earlier test has
|
||||
# usually imported it already, which is how the leak hid behind test order.
|
||||
_LITELLM_IMPORT_PROBE = """
|
||||
import json, os, socket, sys
|
||||
|
||||
attempts = []
|
||||
|
||||
def _trap(address):
|
||||
host = address[0] if isinstance(address, tuple) else address
|
||||
if host not in ("127.0.0.1", "::1", "localhost"):
|
||||
attempts.append(str(host))
|
||||
raise OSError("outbound connection attempted while offline")
|
||||
|
||||
_create_connection = socket.create_connection
|
||||
_connect = socket.socket.connect
|
||||
socket.create_connection = lambda address, *a, **k: (
|
||||
_trap(address) or _create_connection(address, *a, **k)
|
||||
)
|
||||
socket.socket.connect = lambda self, address: _trap(address) or _connect(self, address)
|
||||
|
||||
import headroom
|
||||
|
||||
if sys.argv[1] == "late":
|
||||
os.environ["HEADROOM_OFFLINE"] = "1"
|
||||
import importlib
|
||||
|
||||
importlib.import_module(sys.argv[2])
|
||||
if "crewai" in sys.modules:
|
||||
# CrewAI tracks the install from a background thread; give it the chance
|
||||
# to dial before reporting, or a leak could race past the report.
|
||||
import threading, time
|
||||
|
||||
deadline = time.monotonic() + 3
|
||||
for thread in threading.enumerate():
|
||||
if thread is not threading.main_thread():
|
||||
thread.join(max(0.0, deadline - time.monotonic()))
|
||||
print(json.dumps({
|
||||
"attempts": attempts,
|
||||
"litellm": "litellm" in sys.modules,
|
||||
"crewai": "crewai" in sys.modules,
|
||||
}))
|
||||
"""
|
||||
|
||||
# Variables that turn a library's import-time phone-home off. The probe clears
|
||||
# them all, so each run starts from the libraries' own defaults.
|
||||
_PHONE_HOME_OPT_OUTS = (
|
||||
"LITELLM_LOCAL_MODEL_COST_MAP",
|
||||
"CREWAI_DISABLE_TELEMETRY",
|
||||
"CREWAI_DISABLE_TRACKING",
|
||||
"OTEL_SDK_DISABLED",
|
||||
)
|
||||
|
||||
|
||||
def _run_import_probe(
|
||||
when: str,
|
||||
module: str,
|
||||
env_overrides: dict[str, str] | None = None,
|
||||
pythonpath: Path | None = None,
|
||||
) -> dict[str, object]:
|
||||
"""Import ``module`` in a fresh interpreter with every outbound connect
|
||||
trapped. ``when`` is ``early`` (offline from the start), ``late`` (offline
|
||||
switched on after ``import headroom``) or ``online``."""
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
env = {k: v for k, v in os.environ.items() if k not in _PHONE_HOME_OPT_OUTS}
|
||||
env.pop("HEADROOM_OFFLINE", None)
|
||||
if when == "early":
|
||||
env["HEADROOM_OFFLINE"] = "1"
|
||||
env.update(env_overrides or {})
|
||||
if pythonpath is not None:
|
||||
env["PYTHONPATH"] = os.pathsep.join(
|
||||
p for p in (str(pythonpath), env.get("PYTHONPATH", "")) if p
|
||||
)
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", _LITELLM_IMPORT_PROBE, when, module],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=REPO_ROOT,
|
||||
env=env,
|
||||
timeout=120,
|
||||
check=False,
|
||||
)
|
||||
assert result.returncode == 0, result.stderr[-2000:]
|
||||
return json.loads(result.stdout.strip().splitlines()[-1])
|
||||
|
||||
|
||||
def _is_pin_call(statement: ast.stmt, pin: str) -> bool:
|
||||
if not (isinstance(statement, ast.Expr) and isinstance(statement.value, ast.Call)):
|
||||
return False
|
||||
func = statement.value.func
|
||||
name = func.id if isinstance(func, ast.Name) else getattr(func, "attr", None)
|
||||
return name == pin
|
||||
|
||||
|
||||
def _pinned_before(node: ast.AST, parents: dict[ast.AST, ast.AST], pin: str) -> bool:
|
||||
"""True when a ``pin()`` call precedes ``node`` in its own block or an
|
||||
enclosing block of the same function/module — i.e. it always runs first."""
|
||||
child = node
|
||||
while child in parents:
|
||||
parent = parents[child]
|
||||
for field in ("body", "orelse", "finalbody", "handlers"):
|
||||
block = getattr(parent, field, None)
|
||||
if isinstance(block, list) and child in block:
|
||||
if any(_is_pin_call(s, pin) for s in block[: block.index(child)]):
|
||||
return True
|
||||
if isinstance(parent, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda)):
|
||||
return False
|
||||
child = parent
|
||||
return False
|
||||
|
||||
|
||||
def _unpinned_imports(package: str, pin: str) -> list[str]:
|
||||
"""Every import of ``package`` (or a submodule) in ``headroom/`` that no
|
||||
``pin()`` call precedes in the same scope. A pin in another scope does not
|
||||
count: it ran at a different time from the import."""
|
||||
unpinned: list[str] = []
|
||||
for path in sorted(PACKAGE_ROOT.rglob("*.py")):
|
||||
tree = ast.parse(path.read_text(encoding="utf-8"))
|
||||
parents = {
|
||||
child: parent for parent in ast.walk(tree) for child in ast.iter_child_nodes(parent)
|
||||
}
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Import):
|
||||
names = [alias.name for alias in node.names]
|
||||
elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module:
|
||||
names = [node.module]
|
||||
else:
|
||||
continue
|
||||
if not any(n == package or n.startswith(f"{package}.") for n in names):
|
||||
continue
|
||||
if not _pinned_before(node, parents, pin):
|
||||
unpinned.append(f"{path.relative_to(REPO_ROOT)}:{node.lineno}")
|
||||
return unpinned
|
||||
|
||||
|
||||
class TestLiteLLMCostMapOffline:
|
||||
"""``import litellm`` refreshes its pricing table from
|
||||
raw.githubusercontent.com unless ``LITELLM_LOCAL_MODEL_COST_MAP`` is set,
|
||||
and every litellm-backed module (the pricing helpers, the LiteLLM provider
|
||||
and backend, both Headroom Cloud integrations) triggers it on first import.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def _probe(when: str, module: str, cost_map_env: str | None = None) -> dict[str, object]:
|
||||
pytest.importorskip("litellm")
|
||||
overrides = {} if cost_map_env is None else {"LITELLM_LOCAL_MODEL_COST_MAP": cost_map_env}
|
||||
return _run_import_probe(when, module, overrides)
|
||||
|
||||
_LITELLM_MODULES = [
|
||||
"headroom.providers.litellm",
|
||||
"headroom.backends.litellm",
|
||||
"headroom.pricing.litellm_pricing",
|
||||
"headroom.integrations.litellm_callback",
|
||||
]
|
||||
|
||||
@pytest.mark.parametrize("module", _LITELLM_MODULES)
|
||||
def test_import_with_offline_set_at_start_opens_no_socket(self, module: str) -> None:
|
||||
outcome = self._probe("early", module)
|
||||
assert outcome["litellm"] is True
|
||||
assert outcome["attempts"] == []
|
||||
|
||||
@pytest.mark.parametrize("module", _LITELLM_MODULES)
|
||||
def test_import_with_offline_set_after_package_load_opens_no_socket(self, module: str) -> None:
|
||||
"""The switch is read when it is checked, not when the package loads:
|
||||
``create_app(ProxyConfig(offline=True))`` turns it on after
|
||||
``import headroom``. So a pin in ``headroom/__init__.py`` is not enough
|
||||
on its own; each litellm import pins again first."""
|
||||
outcome = self._probe("late", module)
|
||||
assert outcome["litellm"] is True
|
||||
assert outcome["attempts"] == []
|
||||
|
||||
@pytest.mark.parametrize("operator_value", ["1", "False", ""])
|
||||
def test_operator_value_cannot_reopen_the_fetch(self, operator_value: str) -> None:
|
||||
"""litellm accepts only ``true``, so an operator ``1`` (the form every
|
||||
Headroom switch accepts), ``False`` or empty value would make it fetch.
|
||||
The air-gap switch wins over it."""
|
||||
outcome = self._probe("early", "headroom.providers.litellm", operator_value)
|
||||
assert outcome["litellm"] is True
|
||||
assert outcome["attempts"] == []
|
||||
|
||||
def test_every_litellm_import_is_pinned_first(self) -> None:
|
||||
"""Each ``import litellm`` / ``from litellm…`` in ``headroom/`` must
|
||||
follow a ``pin_litellm_cost_map_local()`` call in the same function (or
|
||||
at module level for a module-level import). A pin in another scope does
|
||||
not count: it ran at a different time from the import."""
|
||||
unpinned = _unpinned_imports("litellm", "pin_litellm_cost_map_local")
|
||||
assert unpinned == [], (
|
||||
"litellm imported without pin_litellm_cost_map_local() first in the same "
|
||||
f"scope; under HEADROOM_OFFLINE it fetches its pricing table: {unpinned}"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _fresh_pin_state(monkeypatch: pytest.MonkeyPatch, value: str | None) -> None:
|
||||
import headroom.offline
|
||||
|
||||
monkeypatch.setattr(headroom.offline._litellm_cost_map_pin, "before", None)
|
||||
# setenv first so teardown restores the real value even after the
|
||||
# pin writes the variable behind monkeypatch's back.
|
||||
monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "")
|
||||
if value is None:
|
||||
monkeypatch.delenv("LITELLM_LOCAL_MODEL_COST_MAP")
|
||||
else:
|
||||
monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", value)
|
||||
|
||||
def test_online_leaves_litellm_alone(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
|
||||
self._fresh_pin_state(monkeypatch, None)
|
||||
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
|
||||
pin_litellm_cost_map_local()
|
||||
assert "LITELLM_LOCAL_MODEL_COST_MAP" not in os.environ
|
||||
|
||||
@pytest.mark.parametrize("before", [None, "False"])
|
||||
def test_leaving_offline_puts_back_what_the_pin_replaced(
|
||||
self, monkeypatch: pytest.MonkeyPatch, before: str | None
|
||||
) -> None:
|
||||
"""Offline turned off before the first litellm import must give the
|
||||
online pricing refresh back, not leave Headroom's pin behind."""
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
|
||||
self._fresh_pin_state(monkeypatch, before)
|
||||
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
|
||||
pin_litellm_cost_map_local()
|
||||
pin_litellm_cost_map_local() # a second pin must not record its own value
|
||||
assert os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] == "True"
|
||||
|
||||
monkeypatch.delenv("HEADROOM_OFFLINE")
|
||||
pin_litellm_cost_map_local()
|
||||
assert os.environ.get("LITELLM_LOCAL_MODEL_COST_MAP") == before
|
||||
|
||||
def test_a_value_changed_after_the_pin_is_not_overwritten(
|
||||
self, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
from headroom.offline import pin_litellm_cost_map_local
|
||||
|
||||
self._fresh_pin_state(monkeypatch, None)
|
||||
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
|
||||
pin_litellm_cost_map_local()
|
||||
os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "false" # the operator's own change
|
||||
|
||||
monkeypatch.delenv("HEADROOM_OFFLINE")
|
||||
pin_litellm_cost_map_local()
|
||||
assert os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] == "false"
|
||||
|
||||
|
||||
# ─────────────── path 2d: CrewAI's import-time install tracking ─────────────
|
||||
|
||||
# Stands in for CrewAI where it is not installed (no extra or CI job installs
|
||||
# it): the same import-time behaviour as crewai 1.6.1's ``_track_install`` —
|
||||
# a background thread dialling api.scarf.sh unless one of its opt-out
|
||||
# variables is ``true`` — plus the one name Headroom imports from it.
|
||||
_CREWAI_STUB = {
|
||||
"crewai/__init__.py": (
|
||||
"import os, socket, threading\n"
|
||||
"\n"
|
||||
"def _track_install():\n"
|
||||
" names = ('OTEL_SDK_DISABLED', 'CREWAI_DISABLE_TELEMETRY', 'CREWAI_DISABLE_TRACKING')\n"
|
||||
" if any(os.getenv(n, 'false').lower() == 'true' for n in names):\n"
|
||||
" return\n"
|
||||
" try:\n"
|
||||
" socket.create_connection(('api.scarf.sh', 443), timeout=2).close()\n"
|
||||
" except OSError:\n"
|
||||
" pass\n"
|
||||
"\n"
|
||||
"threading.Thread(target=_track_install, daemon=True).start()\n"
|
||||
),
|
||||
"crewai/tools/__init__.py": "",
|
||||
"crewai/tools/base_tool.py": "class BaseTool:\n pass\n",
|
||||
}
|
||||
|
||||
# The modules a Headroom user imports that pull CrewAI in: the integration
|
||||
# itself, and any sibling under ``headroom.integrations``, whose package
|
||||
# ``__init__`` imports the CrewAI integration whenever CrewAI is installed.
|
||||
_CREWAI_REACHING_MODULES = [
|
||||
"headroom.integrations.crewai",
|
||||
"headroom.integrations.litellm_callback",
|
||||
]
|
||||
|
||||
|
||||
class TestCrewAITelemetryOffline:
|
||||
"""Importing ``headroom.integrations.litellm_callback`` under the air-gap
|
||||
switch dialled api.scarf.sh whenever CrewAI was installed, because the
|
||||
package ``__init__`` imports the CrewAI integration and ``import crewai``
|
||||
tracks the install from a background thread (found in review of #4242
|
||||
with crewai 1.6.1 and litellm 1.101.0)."""
|
||||
|
||||
@pytest.fixture
|
||||
def crewai_stub(self, tmp_path: Path) -> Path:
|
||||
for relative, source in _CREWAI_STUB.items():
|
||||
path = tmp_path / relative
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(source, encoding="utf-8")
|
||||
return tmp_path
|
||||
|
||||
def test_the_stub_phones_home_when_online(self, crewai_stub: Path) -> None:
|
||||
"""Without this, a stub that never dials would pass every test below."""
|
||||
outcome = _run_import_probe(
|
||||
"online", "headroom.integrations.crewai", pythonpath=crewai_stub
|
||||
)
|
||||
assert outcome["crewai"] is True
|
||||
# litellm's own pricing refresh also runs online; only the tracking matters here.
|
||||
assert "api.scarf.sh" in outcome["attempts"]
|
||||
|
||||
@pytest.mark.parametrize("when", ["early", "late"])
|
||||
@pytest.mark.parametrize("module", _CREWAI_REACHING_MODULES)
|
||||
def test_import_offline_opens_no_socket(
|
||||
self, crewai_stub: Path, module: str, when: str
|
||||
) -> None:
|
||||
if module.endswith("litellm_callback"):
|
||||
pytest.importorskip("litellm")
|
||||
outcome = _run_import_probe(when, module, pythonpath=crewai_stub)
|
||||
assert outcome["crewai"] is True
|
||||
assert outcome["attempts"] == []
|
||||
|
||||
@pytest.mark.parametrize("operator_value", ["1", "False", ""])
|
||||
def test_operator_value_cannot_reopen_the_tracking(
|
||||
self, crewai_stub: Path, operator_value: str
|
||||
) -> None:
|
||||
outcome = _run_import_probe(
|
||||
"early",
|
||||
"headroom.integrations.crewai",
|
||||
{"CREWAI_DISABLE_TELEMETRY": operator_value},
|
||||
pythonpath=crewai_stub,
|
||||
)
|
||||
assert outcome["attempts"] == []
|
||||
|
||||
@pytest.mark.parametrize("module", _CREWAI_REACHING_MODULES)
|
||||
def test_real_crewai_import_offline_opens_no_socket(self, module: str) -> None:
|
||||
"""The same against the installed CrewAI, telemetry at its default."""
|
||||
import importlib.util
|
||||
|
||||
if importlib.util.find_spec("crewai") is None:
|
||||
pytest.skip("crewai is not installed")
|
||||
if module.endswith("litellm_callback"):
|
||||
pytest.importorskip("litellm")
|
||||
outcome = _run_import_probe("early", module)
|
||||
assert outcome["crewai"] is True
|
||||
assert outcome["attempts"] == []
|
||||
|
||||
def test_every_crewai_import_is_pinned_first(self) -> None:
|
||||
unpinned = _unpinned_imports("crewai", "pin_crewai_telemetry_off")
|
||||
assert unpinned == [], (
|
||||
"crewai imported without pin_crewai_telemetry_off() first in the same "
|
||||
f"scope; under HEADROOM_OFFLINE it tracks the install over the network: {unpinned}"
|
||||
)
|
||||
|
||||
def test_leaving_offline_puts_back_what_the_pin_replaced(
|
||||
self, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
import headroom.offline
|
||||
from headroom.offline import pin_crewai_telemetry_off
|
||||
|
||||
monkeypatch.setattr(headroom.offline._crewai_telemetry_pin, "before", None)
|
||||
monkeypatch.setenv("CREWAI_DISABLE_TELEMETRY", "False")
|
||||
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
|
||||
pin_crewai_telemetry_off()
|
||||
assert os.environ["CREWAI_DISABLE_TELEMETRY"] == "true"
|
||||
|
||||
monkeypatch.delenv("HEADROOM_OFFLINE")
|
||||
pin_crewai_telemetry_off()
|
||||
assert os.environ["CREWAI_DISABLE_TELEMETRY"] == "False"
|
||||
|
||||
|
||||
# ────────── path 4: the Python half of the HuggingFace download ─────────────
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user