fix(offline): keep litellm's pricing table local under HEADROOM_OFFLINE (#4242)

## Description

`HEADROOM_OFFLINE=1` still lets litellm reach GitHub. `import litellm`
downloads `model_prices_and_context_window.json` from
`raw.githubusercontent.com` during the import (three attempts) unless
`LITELLM_LOCAL_MODEL_COST_MAP` is `true`. Headroom imports litellm from
24 places, among them the pricing helpers, the LiteLLM provider and
backend, the proxy's cost and savings trackers, and both Headroom Cloud
integrations. With the air-gap switch on, the first of those imports
dialled out.

Nothing in the chokepoint suite caught it, because no Headroom-built
client is involved, so no `guard_egress` site can dominate it.
`TestHeadroomCloudCompressionOffline` only passed when an earlier test
had already imported litellm. Run on its own (`pytest
tests/test_offline_egress_chokepoint.py -k HeadroomCloud`), it fails on
`main` with `SocketOpened: ... ('raw.githubusercontent.com', 443)`.

The only lever is the env var, and litellm reads it once, during its own
import. So:

- `pin_litellm_cost_map_local()` in `headroom/offline.py` sets
`LITELLM_LOCAL_MODEL_COST_MAP=True` while `HEADROOM_OFFLINE` is on.
- Every litellm import in `headroom/` calls it first, in the same scope.
A pin only at package import would miss a switch turned on later:
`is_offline()` is read when checked, and
`create_app(ProxyConfig(offline=True))` sets it after `import headroom`.
- The pin is Headroom's, so it is undone: called with the switch off, it
restores the value it replaced if the variable still holds the pin. A
process that leaves offline mode before its first litellm import gets
the online pricing refresh back.
- It overwrites the variable rather than using `setdefault`. litellm
only accepts `true`, so an operator value of `1` (the form every
Headroom switch takes), `False` or an empty string would otherwise
reopen the fetch.
- A meta-test fails the build for any litellm import without a preceding
pin, the same pattern as the existing site scanner.

No tracking issue; found while chasing a test that only failed when run
on its own.

Review found a second import-time leak on the same path:
`headroom/integrations/__init__.py` imports the CrewAI integration
whenever CrewAI is installed, and `import crewai` requests a Scarf pixel
from `api.scarf.sh` on a background thread unless
`CREWAI_DISABLE_TELEMETRY` is `true`. So `import
headroom.integrations.litellm_callback` still dialled out under the
switch. `pin_crewai_telemetry_off()` handles it the same way, through a
shared `_OfflineEnvPin`, and the meta-test now covers `crewai` imports
too.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/offline.py`: new `pin_litellm_cost_map_local()` (a no-op
unless `HEADROOM_OFFLINE` is on), documented in the module's list of
refused egress.
- All 24 litellm import sites call it immediately before importing:
`pricing/`, `providers/` (litellm, anthropic, openai, cohere, google),
`backends/litellm.py`, `proxy/` (cost, savings_tracker, route_advice),
`integrations/litellm_callback.py`, `telemetry/session.py`,
`perf/analyzer.py`, `learn/analyzer.py`, `cli/` (evals, learn) and
`evals/`.
- `pin_crewai_telemetry_off()` forces `CREWAI_DISABLE_TELEMETRY=true`
while offline (restored the same way), called before the one `crewai`
import (`headroom/integrations/crewai/agents.py`). Both pins share
`_OfflineEnvPin`.
- `headroom/__init__.py` also calls it, which covers a host that imports
litellm itself after importing Headroom.
- `tests/test_offline_egress_chokepoint.py`, new
`TestLiteLLMCostMapOffline`:
- subprocess probes (litellm only fetches on its first import, which an
earlier test usually did already) for four litellm-backed modules, with
the switch set both before and after `import headroom`;
  - operator values `1`, `False` and empty;
- leaving offline mode restores the replaced value (absent or `False`),
and an operator change made after the pin is kept;
  - `test_every_litellm_import_is_pinned_first`, the AST meta-test;
- the `offline` fixture now restores `LITELLM_LOCAL_MODEL_COST_MAP` at
teardown.
- `TestCrewAITelemetryOffline`: a stand-in `crewai` package with crewai
1.6.1's import-time tracking (no extra or CI job installs CrewAI),
probed early and late for `headroom.integrations.crewai` and
`headroom.integrations.litellm_callback`, plus operator values, a
restore test, the same probes against a real CrewAI when installed, and
`test_every_crewai_import_is_pinned_first`. A sanity test checks that
the stand-in does dial when online.
- `docs/metrics-technical-guide.md` and `docs/content/docs/proxy.mdx`
add the refresh and CrewAI's tracking to the refused list. The guide
also states the limits: a host that imports litellm *before* Headroom
has already made the fetch and must set the variable itself, and offline
prices come from the table bundled with the installed litellm release.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py
216 passed, 3 skipped, 1 warning in 38.44s

$ pytest -q --continue-on-collection-errors tests/test_providers tests/test_proxy tests/test_pricing* tests/test_perf* \
    tests/test_telemetry* tests/test_learn* tests/test_route_advice* tests/test_backend* tests/test_litellm* tests/test_cost* \
    tests/test_savings* tests/test_counterfactual* tests/test_evals* tests/test_integrations*
1742 passed, 337 skipped, 6 warnings, 1 error in 65.33s
# the 1 error is test_langchain_1x_compat.py failing collection (NameError: tool) because langchain_core
# is not installed in my venv; it fails identically on main df11f274

$ ruff check headroom tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py   # ruff 0.17.0
All checks passed!
$ ruff format --check headroom tests/test_offline_egress_chokepoint.py tests/test_proxy_hardening.py
573 files already formatted
$ mypy <the 23 changed headroom/ files>
Success: no issues found in 23 source files

# The new tests fail without the fix: removing one pin call, and reverting to setdefault, gives
FAILED ...::test_operator_value_cannot_reopen_the_fetch[1]       assert ['raw.githubusercontent.com', ...] == []
FAILED ...::test_operator_value_cannot_reopen_the_fetch[False]
FAILED ...::test_operator_value_cannot_reopen_the_fetch[]
FAILED ...::test_every_litellm_import_is_pinned_first           names the unpinned import in headroom/backends/litellm.py
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.10.18 venv, litellm 1.101.0 (the
`uv.lock` version), `main` at df11f274 in a worktree vs this branch, the
same venv for both, `LITELLM_LOCAL_MODEL_COST_MAP` unset unless shown.
- Exact command / steps: a probe script replaces
`socket.create_connection` with a trap that records the address and
raises, runs `import headroom`, sets `HEADROOM_OFFLINE=1` before
(`early`) or after (`late`) that import, then imports the module and
prints the recorded attempts. Run with `HEADROOM_OFFLINE=1 python
probe2.py early headroom.integrations.asgi`, then the same for
`headroom.backends.litellm`, then `python probe2.py late
headroom.backends.litellm`, then `HEADROOM_OFFLINE=1
LITELLM_LOCAL_MODEL_COST_MAP=1 python probe2.py early
headroom.providers.litellm`.
- Observed result: main df11f274 gives `attempts=3
['raw.githubusercontent.com:443']` for all four runs; this branch gives
`attempts=0 []` for all four, with litellm loaded in every case.
- CrewAI: in a venv with crewai 1.6.1 + litellm 1.101.0 (the review
setup) and CrewAI telemetry at its default, `pytest
tests/test_offline_egress_chokepoint.py -k 'LiteLLMCostMap or CrewAI'`
gives 28 passed, including both real-CrewAI imports. Without the
`agents.py` pin, 10 CrewAI checks fail with `attempts=['api.scarf.sh']`.
- Not tested: a live proxy on an air-gapped host. The probe covers the
import paths; the proxy's lazy imports (`proxy/cost.py`,
`proxy/savings_tracker.py`, `proxy/route_advice.py`) are covered by the
meta-test, not by a running proxy.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: stable.
- Stable/default behavior changed: only with HEADROOM_OFFLINE set;
litellm then uses its bundled pricing table, which is what it already
fell back to after the three failed fetches. Without the switch nothing
changes.
- Kill switch / disable path: unset HEADROOM_OFFLINE.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- A host process that imports litellm before Headroom (a LiteLLM proxy
that loads the Headroom callback) has already fetched by the time any
Headroom code runs. That connection isn't Headroom's, so the docs state
the condition instead of claiming it.
- `HF_HUB_OFFLINE` and `TRANSFORMERS_OFFLINE` keep their `setdefault` in
`apply_offline_env()`: the Hub path is protected by `guard_egress` in
`onnx_runtime.py`, not by those variables.
- Possible follow-up, not in this PR: show which cost map litellm loaded
(litellm's `get_model_cost_map_source_info()`) in the offline startup
banner or `headroom doctor`.

---------

Co-authored-by: Kenneth Wong <17061293+KennethWKZ@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Kenneth Wong
2026-10-24 08:45:38 -07:00
committed by GitHub
co-authored by Kenneth Wong JerrettDavis
parent 8c07cf4a0f
commit 3fef15d376
27 changed files with 534 additions and 9 deletions
+1 -1
View File
@@ -433,7 +433,7 @@ with a privileged job.
| `HEADROOM_PROXY_TOKEN` | none | Require a token from non-loopback callers, as `X-Headroom-Proxy-Token: <token>` or `Authorization: Bearer <token>`. The token is removed from the request before it is forwarded upstream; clients that also send a provider credential should use `X-Headroom-Proxy-Token` and keep `Authorization` for the provider. Also gates the read-only operator routes (`/stats-history`, `/quota`, `/subscription-window`, `/metrics`), which otherwise answer loopback and trusted-CIDR callers only. |
| `HEADROOM_ALLOW_UNAUTHENTICATED_BIND` | `false` | Acknowledge a non-loopback bind (`--host 0.0.0.0`, `::`, a LAN address) with **no** `HEADROOM_PROXY_TOKEN`. Without a token or this flag the proxy refuses to start, because `/v1/*` would relay upstream with the operator's credentials for every peer that can reach the port. Only set it when the runtime already restricts reachability (a container published on `127.0.0.1`). |
| `HEADROOM_COMPRESS_ALLOW_REMOTE` | `false` | Allow non-loopback callers to reach [`POST /v1/compress`](#post-v1compress) and `POST /v1/usage`. Required to run Headroom as a remote gateway/sidecar; without it remote callers get `404`. |
| `HEADROOM_OFFLINE` | `false` | Air-gap mode: refuse every connection Headroom itself initiates — telemetry, update checks, license reporting, model/tokenizer/binary/dataset downloads, remote Kompress, OTLP and Langfuse export, Copilot auth, the subscription pollers, the OpenAI embedders, Headroom Cloud compression and the TLS diagnostics (`headroom doctor --network` reports them as skipped). Four things still work on purpose: forwarding your own requests to the upstream you configured, operator-configured local endpoints (the Ollama embedder), loopback health probes, and paths an `is_offline()` check already disables. Refusals surface as messages, not tracebacks. See the air-gap notes in `docs/metrics-technical-guide.md`. |
| `HEADROOM_OFFLINE` | `false` | Air-gap mode: refuse every connection Headroom itself initiates — telemetry, update checks, license reporting, model/tokenizer/binary/dataset downloads, litellm's pricing-table refresh, CrewAI's install tracking, remote Kompress, OTLP and Langfuse export, Copilot auth, the subscription pollers, the OpenAI embedders, Headroom Cloud compression and the TLS diagnostics (`headroom doctor --network` reports them as skipped). Four things still work on purpose: forwarding your own requests to the upstream you configured, operator-configured local endpoints (the Ollama embedder), loopback health probes, and paths an `is_offline()` check already disables. Refusals surface as messages, not tracebacks. See the air-gap notes in `docs/metrics-technical-guide.md`. |
| `HEADROOM_LICENSE` | unset | Licence token, read by core and every licensed extension. `HEADROOM_LICENSE_KEY` is a deprecated alias for one release and logs a warning. A licence on its own sends nothing to Headroom. |
| `HEADROOM_USAGE_REPORTING` | `false` | Set `1` to let the proxy validate the licence and post aggregate usage counts (no content) to the Headroom cloud. Off unless set; `HEADROOM_OFFLINE` still overrides it. |
| `--stateless` / `HEADROOM_STATELESS` | `false` | Keep all state in memory; no filesystem writes (disables logs, memory, TOIN). |
+3 -1
View File
@@ -268,7 +268,7 @@ Verify with `curl -s localhost:8787/stats | jq .otel`.
**Multi-tenant labels:** `register_otel_metric_attribute_provider()` adds request-scoped attributes (tenant, team, cost centre) to every OTel datapoint. Max 16 attributes, 256 chars each.
**Air-gapped deployments:** `HEADROOM_OFFLINE=1` refuses every connection Headroom itself decides to make to a destination Headroom itself chose. That is the whole list, not a sample: the anonymous usage beacon (which is **on by default**), the update check, the license/usage reporter, **OTLP metric and Langfuse trace export**, HuggingFace / Kompress / fastembed model and tokenizer downloads (Python and Rust), the remote Kompress endpoint, the `headroom install` release-binary and codebase-memory-mcp downloads, the BFCL eval-dataset fetch and the provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI embedders, the Headroom Cloud compression modes in the ASGI and LiteLLM integrations, and the TLS diagnostics (`headroom doctor --network` endpoint checks and the certificate-chain re-probe after an upstream TLS failure).
**Air-gapped deployments:** `HEADROOM_OFFLINE=1` refuses every connection Headroom itself decides to make to a destination Headroom itself chose. That is the whole list, not a sample: the anonymous usage beacon (which is **on by default**), the update check, the license/usage reporter, **OTLP metric and Langfuse trace export**, HuggingFace / Kompress / fastembed model and tokenizer downloads (Python and Rust), the remote Kompress endpoint, the `headroom install` release-binary and codebase-memory-mcp downloads, litellm's import-time pricing-table refresh, CrewAI's import-time install tracking, the BFCL eval-dataset fetch and the provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI embedders, the Headroom Cloud compression modes in the ASGI and LiteLLM integrations, and the TLS diagnostics (`headroom doctor --network` endpoint checks and the certificate-chain re-probe after an upstream TLS failure).
Four things are deliberately still allowed, and they are the complete set of exceptions:
@@ -279,6 +279,8 @@ Four things are deliberately still allowed, and they are the complete set of exc
Those four are enumerated with written reasons in `_EGRESS_ALLOWLIST` in `tests/test_offline_egress_chokepoint.py`, which fails the build if a new egress path appears that is neither guarded nor one of them. There is no category for "known violation": a path that can dial the internet with the flag set is a bug. A refusal reaches you as a message — a Click error on the CLI, a named "model unavailable" from a model loader, a logged line from a background poller — never as a traceback and never as a silent degradation. Enforcing the same policy at the network layer as well is still good practice; it is no longer the only thing standing between you and Headroom-initiated egress.
litellm's pricing-table refresh is refused by setting `LITELLM_LOCAL_MODEL_COST_MAP=True` before litellm is imported, so it holds only when Headroom is the first thing in the process to import litellm. A host that imports litellm before Headroom, such as a LiteLLM proxy loading the Headroom callback, has made that fetch already and must set the variable itself. Offline, cost and savings figures use the pricing table bundled with the installed litellm release, so a model released after it prices as unknown.
OTLP export is refused loudly: with `HEADROOM_OFFLINE=1` set, `HEADROOM_OTEL_METRICS_ENABLED=1` plus the default `otlp_http` exporter makes the proxy exit 78 at startup with an explanation, rather than quietly dropping metrics. There is no exemption for a collector that looks local — an in-cluster address is not reliably distinguishable from an internet one. For metrics under an air-gap, either scrape `localhost:8787/metrics` or set `HEADROOM_OTEL_METRICS_EXPORTER=console`, both of which stay on-box. `HEADROOM_KOMPRESS_ENDPOINT` and `HEADROOM_LANGFUSE_ENABLED` are refused the same way, for the same reason.
---
+7
View File
@@ -75,6 +75,7 @@ from typing import Any
from ._ort import ensure_ort_dylib_pinned
from ._version import __version__ # noqa: F401
from .offline import pin_litellm_cost_map_local
# Must run before anything can import `headroom._core`: on Windows the
# Rust core resolves onnxruntime.dll at runtime (ort load-dynamic), and
@@ -83,6 +84,12 @@ from ._version import __version__ # noqa: F401
# ~microseconds. See `headroom/_ort.py` for the full story.
ensure_ort_dylib_pinned()
# Under HEADROOM_OFFLINE, litellm must not fetch its pricing table from GitHub
# at import time. Each litellm import in this package pins that itself; this
# call also covers a host that imports litellm after Headroom. A no-op unless
# the switch is set. See `headroom/offline.py`.
pin_litellm_cost_map_local()
from .compress import ( # noqa: E402
CompressConfig,
CompressResult,
+3
View File
@@ -21,6 +21,7 @@ from collections.abc import AsyncIterator
from dataclasses import dataclass, field
from typing import Any
from headroom.offline import pin_litellm_cost_map_local
from headroom.proxy.public_errors import client_message
from headroom.utils import format_exception_message
@@ -67,6 +68,7 @@ try:
import os as _os
_env_snapshot = set(_os.environ)
pin_litellm_cost_map_local()
import litellm
from litellm import acompletion
from litellm.utils import supports_prompt_caching
@@ -536,6 +538,7 @@ def _caller_key_travels_to(model: str, key: str) -> bool:
if not key.startswith("sk-ant-"):
return True
try:
pin_litellm_cost_map_local()
from litellm import get_llm_provider
provider = (get_llm_provider(model)[1] or "").lower()
+3
View File
@@ -7,6 +7,8 @@ from pathlib import Path
import click
from headroom.offline import pin_litellm_cost_map_local
from .main import main
@@ -480,6 +482,7 @@ def _run_memory_eval(
answer_fn = None
if answer_model:
try:
pin_litellm_cost_map_local()
import litellm
def answer_fn(question: str, memories: list[str]) -> str:
+3
View File
@@ -7,6 +7,8 @@ from typing import TYPE_CHECKING, Any, NamedTuple
import click
from headroom.offline import pin_litellm_cost_map_local
if TYPE_CHECKING:
from ..learn.base import LearnPlugin
@@ -390,6 +392,7 @@ def _make_llm_judge(model: str) -> Any:
try:
import json
pin_litellm_cost_map_local()
import litellm
except ImportError:
return None
+3 -1
View File
@@ -15,7 +15,7 @@ from collections.abc import Callable
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any
from headroom.offline import guard_egress
from headroom.offline import guard_egress, pin_litellm_cost_map_local
if TYPE_CHECKING:
from headroom.transforms.html_extractor import HTMLExtractor
@@ -316,6 +316,7 @@ class HTMLExtractionEvaluator:
else:
try:
pin_litellm_cost_map_local()
import litellm
def judge(question: str, ground_truth: str, prediction: str) -> tuple[float, str]:
@@ -396,6 +397,7 @@ Answer concisely and factually based only on the content provided."""
)
else:
pin_litellm_cost_map_local()
import litellm
litellm_response = litellm.completion(
+2 -1
View File
@@ -17,7 +17,7 @@ from __future__ import annotations
import logging
from collections.abc import Callable
from headroom.offline import guard_egress
from headroom.offline import guard_egress, pin_litellm_cost_map_local
logger = logging.getLogger(__name__)
@@ -169,6 +169,7 @@ def create_litellm_judge(
score, reasoning = judge_fn(question, ground_truth, prediction)
"""
try:
pin_litellm_cost_map_local()
import litellm
except ImportError as e:
raise ImportError(
+2
View File
@@ -30,6 +30,7 @@ from headroom.evals.memory.locomo import (
)
from headroom.evals.metrics import compute_exact_match, compute_f1
from headroom.memory import HierarchicalMemory, MemoryConfig
from headroom.offline import pin_litellm_cost_map_local
logger = logging.getLogger(__name__)
@@ -277,6 +278,7 @@ class LoCoMoEvaluator:
List of extracted memory dicts with 'content' and 'category'.
"""
try:
pin_litellm_cost_map_local()
import litellm
except ImportError:
logger.warning("litellm not available, falling back to raw dialogue storage")
+6 -2
View File
@@ -24,8 +24,6 @@ from datetime import datetime
from pathlib import Path
from typing import Any
import litellm
from headroom.evals.memory.locomo import (
LOCOMO_CATEGORIES,
LoCoMoCase,
@@ -37,6 +35,12 @@ from headroom.memory.backends.local import LocalBackend, LocalBackendConfig
from headroom.memory.models import Memory
from headroom.memory.system import MemoryBackend
from headroom.memory.tools import MEMORY_TOOLS
from headroom.offline import pin_litellm_cost_map_local
# Before the import: litellm fetches its pricing table at import time unless
# told not to, and HEADROOM_OFFLINE forbids that fetch.
pin_litellm_cost_map_local()
import litellm # noqa: E402
logger = logging.getLogger(__name__)
+3
View File
@@ -27,7 +27,10 @@ from __future__ import annotations
import logging
from typing import Any
from headroom.offline import pin_crewai_telemetry_off
try:
pin_crewai_telemetry_off()
from crewai.tools.base_tool import BaseTool
CREWAI_AVAILABLE = True
+2 -1
View File
@@ -24,12 +24,13 @@ from headroom.integrations._compress_backend import ( # noqa: F401
_DEFAULT_CLOUD_URL,
CompressBackend,
)
from headroom.offline import guard_egress # noqa: F401
from headroom.offline import guard_egress, pin_litellm_cost_map_local # noqa: F401
logger = logging.getLogger(__name__)
try:
pin_litellm_cost_map_local()
from litellm.integrations.custom_logger import CustomLogger as _CustomLogger
except ImportError: # litellm not installed — fall back to plain object
_CustomLogger = object # type: ignore[assignment,misc]
+2
View File
@@ -25,6 +25,7 @@ import time
import typing
from headroom._subprocess import Popen, run
from headroom.offline import pin_litellm_cost_map_local
from .loops import LoopPattern, apply_loop_weighting, detect_loops, format_loops_for_digest
from .models import (
@@ -1095,6 +1096,7 @@ def _call_llm(
if model in _CLI_MODEL_IDS:
return _call_cli_llm(digest, model, on_progress=on_progress)
pin_litellm_cost_map_local()
import litellm
# Suppress LiteLLM's verbose logging
+84 -2
View File
@@ -8,8 +8,9 @@ deployment exists to carry.
Refused (this is not a sample — it is the rule, and the meta-test enforces it):
the telemetry beacon, the update check, the license/usage reporter, OTLP and
Langfuse export, HuggingFace/Kompress/fastembed model downloads, release-binary
downloads (codebase-memory-mcp among them), eval dataset downloads and the provider SDK
clients the eval harness drives, GitHub Copilot device-flow auth and token
downloads (codebase-memory-mcp among them), litellm's import-time pricing-table
refresh, CrewAI's import-time install tracking, eval dataset downloads and the
provider SDK clients the eval harness drives, GitHub Copilot device-flow auth and token
exchange, the Anthropic / Codex / Copilot subscription pollers, the OpenAI
embedders, both Headroom Cloud compression integrations, and the TLS
diagnostics (``doctor --network`` endpoint checks and the certificate-chain
@@ -101,6 +102,87 @@ def apply_offline_env() -> None:
os.environ.setdefault("TRANSFORMERS_OFFLINE", "1")
class _OfflineEnvPin:
"""An environment variable Headroom forces while ``HEADROOM_OFFLINE`` is
on, for a third-party library that dials out during its own import and
only listens to that variable.
Overwritten rather than ``setdefault``: these libraries accept only
``true``, so an operator ``1``, ``False`` or empty value would reopen a
connection the air-gap switch exists to refuse. The pin is Headroom's, so
it is also undone: called with the switch off, it puts back the value it
replaced, provided the variable still holds the pin. An operator's own
value is never touched outside offline mode.
"""
def __init__(self, name: str, value: str) -> None:
self.name = name
self.value = value
# Unset while no pin is in place; otherwise a 1-tuple of the value the
# pin replaced (None when the variable was absent).
self.before: tuple[str | None] | None = None
def __call__(self) -> None:
if is_offline():
if self.before is None:
self.before = (os.environ.get(self.name),)
os.environ[self.name] = self.value
elif self.before is not None:
(before,) = self.before
self.before = None
if os.environ.get(self.name) == self.value:
if before is None:
os.environ.pop(self.name, None)
else:
os.environ[self.name] = before
LITELLM_LOCAL_COST_MAP_ENV = "LITELLM_LOCAL_MODEL_COST_MAP"
_litellm_cost_map_pin = _OfflineEnvPin(LITELLM_LOCAL_COST_MAP_ENV, "True")
CREWAI_TELEMETRY_ENV = "CREWAI_DISABLE_TELEMETRY"
_crewai_telemetry_pin = _OfflineEnvPin(CREWAI_TELEMETRY_ENV, "true")
def pin_litellm_cost_map_local() -> None:
"""Keep litellm on its bundled pricing table under ``HEADROOM_OFFLINE``.
``import litellm`` fetches ``model_prices_and_context_window.json`` from
raw.githubusercontent.com at import time (three attempts) unless
``LITELLM_LOCAL_MODEL_COST_MAP`` is ``true``. That is not a client Headroom
builds, so no ``guard_egress`` site can dominate it; the only lever is the
env var, read once during that import. So every ``import litellm`` in
``headroom/`` calls this first, in the same scope, because the switch is
read when it is checked rather than when the package loads
(``create_app(ProxyConfig(offline=True))`` turns it on after import). The
chokepoint meta-test fails the build for an import that does not.
``headroom/__init__.py`` also calls it, which covers a host process that
imports litellm itself after importing Headroom; a host that imports
litellm *before* Headroom has made that fetch already and must set the
variable itself.
Forced and given back as described on :class:`_OfflineEnvPin`, so a
process that leaves offline mode before its first litellm import gets the
online pricing refresh again.
"""
_litellm_cost_map_pin()
def pin_crewai_telemetry_off() -> None:
"""Keep CrewAI's install tracking quiet under ``HEADROOM_OFFLINE``.
``import crewai`` starts a thread that requests a Scarf pixel from
api.scarf.sh unless ``CREWAI_DISABLE_TELEMETRY`` (or ``OTEL_SDK_DISABLED``
/ ``CREWAI_DISABLE_TRACKING``) is ``true``. Headroom imports CrewAI
whenever it is installed, through ``headroom.integrations``, so even a
LiteLLM callback import under the air-gap switch dialled out. As with
litellm, the variable is the only lever, so every ``crewai`` import in
``headroom/`` calls this first (the chokepoint meta-test enforces it), and
it is forced and given back as described on :class:`_OfflineEnvPin`.
"""
_crewai_telemetry_pin()
class OfflineEgressBlocked(BaseException):
"""Raised by :func:`guard_egress` when ``HEADROOM_OFFLINE`` is in force.
+2
View File
@@ -20,6 +20,7 @@ from datetime import datetime, timedelta
from pathlib import Path
from headroom import paths as _paths
from headroom.offline import pin_litellm_cost_map_local
from headroom.pricing.litellm_pricing import resolve_litellm_model
log = logging.getLogger(__name__)
@@ -74,6 +75,7 @@ _STAGE_TIMINGS_RE = re.compile(
# directly instead of maintaining our own pricing tables.
try:
pin_litellm_cost_map_local()
import litellm as _litellm
_LITELLM_AVAILABLE = True
+2
View File
@@ -63,6 +63,7 @@ from typing import Any
from headroom.cache_economics import CACHE_ECONOMICS
from headroom.log_safety import describe_exception, safe_id
from headroom.offline import pin_litellm_cost_map_local
from headroom.pricing.cache_ttl import CACHE_WRITE_MULTIPLIERS
logger = logging.getLogger(__name__)
@@ -338,6 +339,7 @@ def _litellm() -> Any | None:
deployments often run without it. Callers degrade to list pricing.
"""
try:
pin_litellm_cost_map_local()
import litellm
except Exception: # pragma: no cover - environment-dependent
return None
+6
View File
@@ -15,6 +15,7 @@ from dataclasses import dataclass
from datetime import datetime
from typing import Any
from headroom.offline import pin_litellm_cost_map_local
from headroom.pricing.deepseek_tiers import (
LEGACY_MODEL_IDS,
OFF_PEAK_RATES_PER_1M,
@@ -34,6 +35,11 @@ from headroom.pricing.litellm_model_resolution import (
# litellm added. The module itself is fully imported and cached in
# `sys.modules`; subsequent `import litellm` calls hit the cache and
# don't re-run the dotenv side effect.
#
# Under HEADROOM_OFFLINE, litellm must also stay on its bundled pricing table
# rather than fetch one from GitHub during that import.
pin_litellm_cost_map_local()
try:
import os as _os
+2
View File
@@ -25,6 +25,7 @@ from datetime import datetime
from typing import Any, cast
from headroom import paths as _paths
from headroom.offline import pin_litellm_cost_map_local
from headroom.pricing.deepseek_tiers import (
LEGACY_MODEL_IDS,
OFF_PEAK_RATES_PER_1M,
@@ -49,6 +50,7 @@ def _get_litellm_clients() -> tuple[Any | None, Any | None]:
return None, None
try:
pin_litellm_cost_map_local()
import litellm
litellm.suppress_debug_info = True
+2
View File
@@ -22,12 +22,14 @@ from datetime import date
from typing import Any
from headroom.log_safety import describe_exception, safe_id
from headroom.offline import pin_litellm_cost_map_local
from headroom.pricing.litellm_pricing import estimate_cost_from_tokens
from headroom.tokenizers import EstimatingTokenCounter
from .base import Provider, TokenCounter
try:
pin_litellm_cost_map_local()
import litellm
LITELLM_AVAILABLE = True
+2
View File
@@ -26,6 +26,7 @@ from datetime import date
from typing import Any
from headroom.models.registry import ModelRegistry
from headroom.offline import pin_litellm_cost_map_local
from headroom.pricing.litellm_pricing import estimate_cost_from_tokens
from headroom.tokenizers import EstimatingTokenCounter
@@ -33,6 +34,7 @@ from .base import Provider, TokenCounter
# Check if litellm is available for pricing/context limit lookups
try:
pin_litellm_cost_map_local()
import litellm
LITELLM_AVAILABLE = True
+2
View File
@@ -25,6 +25,7 @@ import logging
import os
from typing import Any
from headroom.offline import pin_litellm_cost_map_local
from headroom.pricing.litellm_pricing import estimate_cost_from_tokens
from headroom.tokenizers import EstimatingTokenCounter
@@ -38,6 +39,7 @@ try:
# module-level suppression flags below can be set.
os.environ.setdefault("LITELLM_SUPPRESS_DEBUG_INFO", "True")
pin_litellm_cost_map_local()
import litellm
# Suppress litellm's startup banner ("Provider List: https://...") and
+2
View File
@@ -17,6 +17,7 @@ from typing import Any, cast
from headroom import paths as _paths
from headroom.log_safety import describe_exception, safe_id
from headroom.offline import pin_litellm_cost_map_local
from headroom.pricing.litellm_pricing import estimate_cost_from_tokens
from headroom.tokenizers.base import (
TokenCountCache,
@@ -49,6 +50,7 @@ def _get_litellm_module() -> Any | None:
return None
try:
pin_litellm_cost_map_local()
import litellm
except ImportError:
return None
+2
View File
@@ -17,6 +17,7 @@ from typing import TYPE_CHECKING, Any, NamedTuple
from headroom.cache_economics import CACHE_ECONOMICS as _CACHE_ECONOMICS
from headroom.log_safety import WarnOnce, describe_exception, safe_id
from headroom.offline import pin_litellm_cost_map_local
from headroom.proxy.budget_basis_policy import (
BUDGET_BASIS_BLOCK,
BUDGET_BASIS_IGNORE,
@@ -50,6 +51,7 @@ def _get_litellm_module() -> Any | None:
return litellm
try:
pin_litellm_cost_map_local()
import litellm as imported_litellm
except ImportError:
return None
+4
View File
@@ -34,6 +34,8 @@ import logging
from dataclasses import dataclass
from typing import Any
from headroom.offline import pin_litellm_cost_map_local
log = logging.getLogger(__name__)
STATE_ATTR = "headroom_route"
@@ -202,6 +204,7 @@ def resolver_for(handler: Any) -> BackendResolver:
def _known_provider(provider: str) -> bool:
"""Is this a provider litellm actually knows about?"""
try:
pin_litellm_cost_map_local()
import litellm
names = {getattr(p, "value", None) or str(p) for p in getattr(litellm, "provider_list", [])}
@@ -215,6 +218,7 @@ def _provider_of(model: str) -> str:
if "/" in model:
return model.split("/", 1)[0]
try:
pin_litellm_cost_map_local()
import litellm
entry = litellm.model_cost.get(model) or {}
+2
View File
@@ -25,6 +25,7 @@ from typing import Any
from headroom import paths as _paths
from headroom.log_safety import WarnOnce, describe_exception, safe_id
from headroom.offline import pin_litellm_cost_map_local
from headroom.proxy import project_name_policy
from headroom.proxy.persistent_metrics import (
MAX_LABEL_LENGTH,
@@ -156,6 +157,7 @@ def _get_litellm_module() -> Any | None:
return litellm
try:
pin_litellm_cost_map_local()
import litellm as imported_litellm
except ImportError:
return None
+3
View File
@@ -118,6 +118,8 @@ from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
from headroom.offline import pin_litellm_cost_map_local
logger = logging.getLogger(__name__)
# Where session events go. The receiver is open source: deploy/beacon/worker.js.
@@ -532,6 +534,7 @@ def _public_model(model: str) -> str | None:
resolved: str | None = None
try:
pin_litellm_cost_map_local()
import litellm
registry = litellm.model_cost
+379
View File
@@ -191,6 +191,17 @@ class TestSocketTrap:
@pytest.fixture
def offline(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
# An import of litellm or CrewAI under the switch writes its pin variable
# directly, out of monkeypatch's sight. Record each variable (setenv saves
# the original, delenv then clears it) so teardown puts it back...
import headroom.offline
for pin in (headroom.offline._litellm_cost_map_pin, headroom.offline._crewai_telemetry_pin):
monkeypatch.setenv(pin.name, "")
monkeypatch.delenv(pin.name)
# ...and the pin's memory of the value it replaced, which would
# otherwise outlive the variable it describes.
monkeypatch.setattr(pin, "before", None)
# ``"example.com" in url`` is true of ``https://example.com.attacker.test``
@@ -415,6 +426,374 @@ class TestLangfuseExporterOffline:
assert configure_langfuse_tracing(LangfuseTracingConfig(enabled=False)) is not None
# ────────────── path 2c: litellm's import-time pricing fetch ────────────────
# Runs in a fresh interpreter: the fetch happens once, on the first
# ``import litellm``, and inside this pytest process an earlier test has
# usually imported it already, which is how the leak hid behind test order.
_LITELLM_IMPORT_PROBE = """
import json, os, socket, sys
attempts = []
def _trap(address):
host = address[0] if isinstance(address, tuple) else address
if host not in ("127.0.0.1", "::1", "localhost"):
attempts.append(str(host))
raise OSError("outbound connection attempted while offline")
_create_connection = socket.create_connection
_connect = socket.socket.connect
socket.create_connection = lambda address, *a, **k: (
_trap(address) or _create_connection(address, *a, **k)
)
socket.socket.connect = lambda self, address: _trap(address) or _connect(self, address)
import headroom
if sys.argv[1] == "late":
os.environ["HEADROOM_OFFLINE"] = "1"
import importlib
importlib.import_module(sys.argv[2])
if "crewai" in sys.modules:
# CrewAI tracks the install from a background thread; give it the chance
# to dial before reporting, or a leak could race past the report.
import threading, time
deadline = time.monotonic() + 3
for thread in threading.enumerate():
if thread is not threading.main_thread():
thread.join(max(0.0, deadline - time.monotonic()))
print(json.dumps({
"attempts": attempts,
"litellm": "litellm" in sys.modules,
"crewai": "crewai" in sys.modules,
}))
"""
# Variables that turn a library's import-time phone-home off. The probe clears
# them all, so each run starts from the libraries' own defaults.
_PHONE_HOME_OPT_OUTS = (
"LITELLM_LOCAL_MODEL_COST_MAP",
"CREWAI_DISABLE_TELEMETRY",
"CREWAI_DISABLE_TRACKING",
"OTEL_SDK_DISABLED",
)
def _run_import_probe(
when: str,
module: str,
env_overrides: dict[str, str] | None = None,
pythonpath: Path | None = None,
) -> dict[str, object]:
"""Import ``module`` in a fresh interpreter with every outbound connect
trapped. ``when`` is ``early`` (offline from the start), ``late`` (offline
switched on after ``import headroom``) or ``online``."""
import json
import subprocess
import sys
env = {k: v for k, v in os.environ.items() if k not in _PHONE_HOME_OPT_OUTS}
env.pop("HEADROOM_OFFLINE", None)
if when == "early":
env["HEADROOM_OFFLINE"] = "1"
env.update(env_overrides or {})
if pythonpath is not None:
env["PYTHONPATH"] = os.pathsep.join(
p for p in (str(pythonpath), env.get("PYTHONPATH", "")) if p
)
result = subprocess.run(
[sys.executable, "-c", _LITELLM_IMPORT_PROBE, when, module],
capture_output=True,
text=True,
cwd=REPO_ROOT,
env=env,
timeout=120,
check=False,
)
assert result.returncode == 0, result.stderr[-2000:]
return json.loads(result.stdout.strip().splitlines()[-1])
def _is_pin_call(statement: ast.stmt, pin: str) -> bool:
if not (isinstance(statement, ast.Expr) and isinstance(statement.value, ast.Call)):
return False
func = statement.value.func
name = func.id if isinstance(func, ast.Name) else getattr(func, "attr", None)
return name == pin
def _pinned_before(node: ast.AST, parents: dict[ast.AST, ast.AST], pin: str) -> bool:
"""True when a ``pin()`` call precedes ``node`` in its own block or an
enclosing block of the same function/module — i.e. it always runs first."""
child = node
while child in parents:
parent = parents[child]
for field in ("body", "orelse", "finalbody", "handlers"):
block = getattr(parent, field, None)
if isinstance(block, list) and child in block:
if any(_is_pin_call(s, pin) for s in block[: block.index(child)]):
return True
if isinstance(parent, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda)):
return False
child = parent
return False
def _unpinned_imports(package: str, pin: str) -> list[str]:
"""Every import of ``package`` (or a submodule) in ``headroom/`` that no
``pin()`` call precedes in the same scope. A pin in another scope does not
count: it ran at a different time from the import."""
unpinned: list[str] = []
for path in sorted(PACKAGE_ROOT.rglob("*.py")):
tree = ast.parse(path.read_text(encoding="utf-8"))
parents = {
child: parent for parent in ast.walk(tree) for child in ast.iter_child_nodes(parent)
}
for node in ast.walk(tree):
if isinstance(node, ast.Import):
names = [alias.name for alias in node.names]
elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module:
names = [node.module]
else:
continue
if not any(n == package or n.startswith(f"{package}.") for n in names):
continue
if not _pinned_before(node, parents, pin):
unpinned.append(f"{path.relative_to(REPO_ROOT)}:{node.lineno}")
return unpinned
class TestLiteLLMCostMapOffline:
"""``import litellm`` refreshes its pricing table from
raw.githubusercontent.com unless ``LITELLM_LOCAL_MODEL_COST_MAP`` is set,
and every litellm-backed module (the pricing helpers, the LiteLLM provider
and backend, both Headroom Cloud integrations) triggers it on first import.
"""
@staticmethod
def _probe(when: str, module: str, cost_map_env: str | None = None) -> dict[str, object]:
pytest.importorskip("litellm")
overrides = {} if cost_map_env is None else {"LITELLM_LOCAL_MODEL_COST_MAP": cost_map_env}
return _run_import_probe(when, module, overrides)
_LITELLM_MODULES = [
"headroom.providers.litellm",
"headroom.backends.litellm",
"headroom.pricing.litellm_pricing",
"headroom.integrations.litellm_callback",
]
@pytest.mark.parametrize("module", _LITELLM_MODULES)
def test_import_with_offline_set_at_start_opens_no_socket(self, module: str) -> None:
outcome = self._probe("early", module)
assert outcome["litellm"] is True
assert outcome["attempts"] == []
@pytest.mark.parametrize("module", _LITELLM_MODULES)
def test_import_with_offline_set_after_package_load_opens_no_socket(self, module: str) -> None:
"""The switch is read when it is checked, not when the package loads:
``create_app(ProxyConfig(offline=True))`` turns it on after
``import headroom``. So a pin in ``headroom/__init__.py`` is not enough
on its own; each litellm import pins again first."""
outcome = self._probe("late", module)
assert outcome["litellm"] is True
assert outcome["attempts"] == []
@pytest.mark.parametrize("operator_value", ["1", "False", ""])
def test_operator_value_cannot_reopen_the_fetch(self, operator_value: str) -> None:
"""litellm accepts only ``true``, so an operator ``1`` (the form every
Headroom switch accepts), ``False`` or empty value would make it fetch.
The air-gap switch wins over it."""
outcome = self._probe("early", "headroom.providers.litellm", operator_value)
assert outcome["litellm"] is True
assert outcome["attempts"] == []
def test_every_litellm_import_is_pinned_first(self) -> None:
"""Each ``import litellm`` / ``from litellm…`` in ``headroom/`` must
follow a ``pin_litellm_cost_map_local()`` call in the same function (or
at module level for a module-level import). A pin in another scope does
not count: it ran at a different time from the import."""
unpinned = _unpinned_imports("litellm", "pin_litellm_cost_map_local")
assert unpinned == [], (
"litellm imported without pin_litellm_cost_map_local() first in the same "
f"scope; under HEADROOM_OFFLINE it fetches its pricing table: {unpinned}"
)
@staticmethod
def _fresh_pin_state(monkeypatch: pytest.MonkeyPatch, value: str | None) -> None:
import headroom.offline
monkeypatch.setattr(headroom.offline._litellm_cost_map_pin, "before", None)
# setenv first so teardown restores the real value even after the
# pin writes the variable behind monkeypatch's back.
monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "")
if value is None:
monkeypatch.delenv("LITELLM_LOCAL_MODEL_COST_MAP")
else:
monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", value)
def test_online_leaves_litellm_alone(self, monkeypatch: pytest.MonkeyPatch) -> None:
from headroom.offline import pin_litellm_cost_map_local
self._fresh_pin_state(monkeypatch, None)
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
pin_litellm_cost_map_local()
assert "LITELLM_LOCAL_MODEL_COST_MAP" not in os.environ
@pytest.mark.parametrize("before", [None, "False"])
def test_leaving_offline_puts_back_what_the_pin_replaced(
self, monkeypatch: pytest.MonkeyPatch, before: str | None
) -> None:
"""Offline turned off before the first litellm import must give the
online pricing refresh back, not leave Headroom's pin behind."""
from headroom.offline import pin_litellm_cost_map_local
self._fresh_pin_state(monkeypatch, before)
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
pin_litellm_cost_map_local()
pin_litellm_cost_map_local() # a second pin must not record its own value
assert os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] == "True"
monkeypatch.delenv("HEADROOM_OFFLINE")
pin_litellm_cost_map_local()
assert os.environ.get("LITELLM_LOCAL_MODEL_COST_MAP") == before
def test_a_value_changed_after_the_pin_is_not_overwritten(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
from headroom.offline import pin_litellm_cost_map_local
self._fresh_pin_state(monkeypatch, None)
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
pin_litellm_cost_map_local()
os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "false" # the operator's own change
monkeypatch.delenv("HEADROOM_OFFLINE")
pin_litellm_cost_map_local()
assert os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] == "false"
# ─────────────── path 2d: CrewAI's import-time install tracking ─────────────
# Stands in for CrewAI where it is not installed (no extra or CI job installs
# it): the same import-time behaviour as crewai 1.6.1's ``_track_install`` —
# a background thread dialling api.scarf.sh unless one of its opt-out
# variables is ``true`` — plus the one name Headroom imports from it.
_CREWAI_STUB = {
"crewai/__init__.py": (
"import os, socket, threading\n"
"\n"
"def _track_install():\n"
" names = ('OTEL_SDK_DISABLED', 'CREWAI_DISABLE_TELEMETRY', 'CREWAI_DISABLE_TRACKING')\n"
" if any(os.getenv(n, 'false').lower() == 'true' for n in names):\n"
" return\n"
" try:\n"
" socket.create_connection(('api.scarf.sh', 443), timeout=2).close()\n"
" except OSError:\n"
" pass\n"
"\n"
"threading.Thread(target=_track_install, daemon=True).start()\n"
),
"crewai/tools/__init__.py": "",
"crewai/tools/base_tool.py": "class BaseTool:\n pass\n",
}
# The modules a Headroom user imports that pull CrewAI in: the integration
# itself, and any sibling under ``headroom.integrations``, whose package
# ``__init__`` imports the CrewAI integration whenever CrewAI is installed.
_CREWAI_REACHING_MODULES = [
"headroom.integrations.crewai",
"headroom.integrations.litellm_callback",
]
class TestCrewAITelemetryOffline:
"""Importing ``headroom.integrations.litellm_callback`` under the air-gap
switch dialled api.scarf.sh whenever CrewAI was installed, because the
package ``__init__`` imports the CrewAI integration and ``import crewai``
tracks the install from a background thread (found in review of #4242
with crewai 1.6.1 and litellm 1.101.0)."""
@pytest.fixture
def crewai_stub(self, tmp_path: Path) -> Path:
for relative, source in _CREWAI_STUB.items():
path = tmp_path / relative
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(source, encoding="utf-8")
return tmp_path
def test_the_stub_phones_home_when_online(self, crewai_stub: Path) -> None:
"""Without this, a stub that never dials would pass every test below."""
outcome = _run_import_probe(
"online", "headroom.integrations.crewai", pythonpath=crewai_stub
)
assert outcome["crewai"] is True
# litellm's own pricing refresh also runs online; only the tracking matters here.
assert "api.scarf.sh" in outcome["attempts"]
@pytest.mark.parametrize("when", ["early", "late"])
@pytest.mark.parametrize("module", _CREWAI_REACHING_MODULES)
def test_import_offline_opens_no_socket(
self, crewai_stub: Path, module: str, when: str
) -> None:
if module.endswith("litellm_callback"):
pytest.importorskip("litellm")
outcome = _run_import_probe(when, module, pythonpath=crewai_stub)
assert outcome["crewai"] is True
assert outcome["attempts"] == []
@pytest.mark.parametrize("operator_value", ["1", "False", ""])
def test_operator_value_cannot_reopen_the_tracking(
self, crewai_stub: Path, operator_value: str
) -> None:
outcome = _run_import_probe(
"early",
"headroom.integrations.crewai",
{"CREWAI_DISABLE_TELEMETRY": operator_value},
pythonpath=crewai_stub,
)
assert outcome["attempts"] == []
@pytest.mark.parametrize("module", _CREWAI_REACHING_MODULES)
def test_real_crewai_import_offline_opens_no_socket(self, module: str) -> None:
"""The same against the installed CrewAI, telemetry at its default."""
import importlib.util
if importlib.util.find_spec("crewai") is None:
pytest.skip("crewai is not installed")
if module.endswith("litellm_callback"):
pytest.importorskip("litellm")
outcome = _run_import_probe("early", module)
assert outcome["crewai"] is True
assert outcome["attempts"] == []
def test_every_crewai_import_is_pinned_first(self) -> None:
unpinned = _unpinned_imports("crewai", "pin_crewai_telemetry_off")
assert unpinned == [], (
"crewai imported without pin_crewai_telemetry_off() first in the same "
f"scope; under HEADROOM_OFFLINE it tracks the install over the network: {unpinned}"
)
def test_leaving_offline_puts_back_what_the_pin_replaced(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
import headroom.offline
from headroom.offline import pin_crewai_telemetry_off
monkeypatch.setattr(headroom.offline._crewai_telemetry_pin, "before", None)
monkeypatch.setenv("CREWAI_DISABLE_TELEMETRY", "False")
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
pin_crewai_telemetry_off()
assert os.environ["CREWAI_DISABLE_TELEMETRY"] == "true"
monkeypatch.delenv("HEADROOM_OFFLINE")
pin_crewai_telemetry_off()
assert os.environ["CREWAI_DISABLE_TELEMETRY"] == "False"
# ────────── path 4: the Python half of the HuggingFace download ─────────────