mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(offline): guard the learn and memory-eval LLM calls under HEADROOM_OFFLINE (#4344)
## Description `HEADROOM_OFFLINE=1` didn't cover `headroom learn` or the memory evals. Every one of these dialed out with no `guard_egress` on the path: - `learn/analyzer.py` `_call_llm`: `litellm.completion` for `headroom learn` (dry run included, since the analysis runs before the `--apply` gate) - `cli/learn.py` `_make_llm_judge`: `litellm.completion` for `headroom learn --verbosity --llm-judge` - `cli/evals.py` `_run_memory_eval`: `litellm.completion` for `--answer-model` - `evals/memory/runner.py` `_extract_memories_from_session`: `litellm.completion` for memory extraction (on by default) - `evals/memory/runner_v2.py` `_replay_conversation` and `_answer_question`: `litellm.completion` passed to `asyncio.to_thread` - `evals/memory/judge.py` `create_litellm_judge`: `litellm.completion`. The OpenAI and Anthropic builders in the same file were already guarded. - `evals/memory/locomo.py` `download_locomo`: `urllib.request.urlretrieve` for the dataset The chokepoint meta-test (`test_every_egress_site_is_guarded_or_allowlisted`) passed on `main` because `_EGRESS_CALLEES` matched neither `litellm.completion` nor `urlretrieve`. This PR adds a `guard_egress(...)` call before each of those calls. Where the callable gets built up front (the `--llm-judge` judge, the `--answer-model` answer function, the litellm judge factory), the guard sits in the builder, so the command refuses before the eval starts and not halfway through it. The LoCoMo guard comes after the cache check, so a pre-seeded dataset still loads offline. I also added `litellm\.a?completion` and `(?:urllib\.request\.)?urlretrieve` to the meta-test's callee pattern so the next site like these can't slip through. Fixes #4338 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/learn/analyzer.py`, `headroom/cli/learn.py`, `headroom/cli/evals.py`, `headroom/evals/memory/runner.py`, `headroom/evals/memory/runner_v2.py`, `headroom/evals/memory/judge.py`, `headroom/evals/memory/locomo.py`: add a `guard_egress(purpose, destination)` call that dominates each call site. Each purpose string names the model and feature, so the refusal tells the operator what to turn off. - `tests/test_offline_egress_chokepoint.py`: - `_EGRESS_CALLEES` now also matches `litellm.completion` / `litellm.acompletion` and `urlretrieve`. - Widening the pattern also exposes the four `acompletion` sites in `headroom/backends/litellm.py`, which are `LiteLLMBackend`'s send/stream paths. Those forward the caller's request to the provider the operator configured as the proxy backend (Bedrock, Vertex, OpenRouter, ...), so I allowlisted them as `user-traffic` with a count of 4, the same treatment as the proxy's own forwarding in `proxy/server.py`. Please check that you agree with this classification. - New `TestLearnAndMemoryEvalOffline`: one test per path, with `litellm` swapped for a recorder. Each test asserts that `OfflineEgressBlocked` is raised and that no completion call was made. The LoCoMo test checks that no socket opens and that the refusal names `raw.githubusercontent.com`. There is also a test that a cached LoCoMo file still loads offline. - Follow-up from review: `headroom learn`'s provider-CLI backends (claude-cli, gemini-cli, codex-cli, agy-cli) are now refused before `_call_cli_llm` runs, and the chokepoint inventory treats `_call_cli_llm` as an egress call. ## Testing - [x] Unit tests pass (`pytest`): `tests/test_offline_egress_chokepoint.py tests/test_learn tests/test_evals tests/test_cli_learn.py tests/test_memory_eval.py tests/test_verbosity_learn.py tests/test_learn_grok_plugin.py`: 628 passed, 6 skipped - [x] Linting passes (`ruff check .`): `ruff check` and `ruff format --check` on the 8 changed files - [x] Type checking passes (`mypy headroom`): `mypy` on the 7 changed source files, no issues - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output With the source changes reverted and the new tests kept (`main` behavior): ```text $ pytest tests/test_offline_egress_chokepoint.py -q -k "TestLearnAndMemoryEvalOffline or TestEgressChokepointCoverage" E headroom/cli/evals.py E line 510: response = litellm.completion( E headroom/cli/learn.py E line 406: resp = litellm.completion( E headroom/evals/memory/judge.py E line 185: response = litellm.completion( E headroom/evals/memory/locomo.py E line 235: urllib.request.urlretrieve(LOCOMO_URL, cache_path) # nosec B310 E headroom/evals/memory/runner.py E line 325: response = litellm.completion( E headroom/learn/analyzer.py E line 1109: response = litellm.completion( FAILED ...::TestLearnAndMemoryEvalOffline::test_learn_analysis_is_refused FAILED ...::TestLearnAndMemoryEvalOffline::test_learn_verbosity_llm_judge_is_refused FAILED ...::TestLearnAndMemoryEvalOffline::test_memory_eval_answer_model_is_refused FAILED ...::TestLearnAndMemoryEvalOffline::test_memory_eval_litellm_judge_is_refused FAILED ...::TestLearnAndMemoryEvalOffline::test_memory_eval_extraction_is_refused FAILED ...::TestLearnAndMemoryEvalOffline::test_memory_eval_v2_save_and_answer_are_refused FAILED ...::TestLearnAndMemoryEvalOffline::test_locomo_download_opens_no_socket FAILED ...::TestEgressChokepointCoverage::test_every_egress_site_is_guarded_or_allowlisted ================= 8 failed, 5 passed, 137 deselected in 23.81s ================= ``` With the fix: ```text $ pytest tests/test_offline_egress_chokepoint.py -q -k "TestLearnAndMemoryEvalOffline or TestEgressChokepointCoverage" ====================== 13 passed, 137 deselected in 9.93s ====================== $ pytest tests/test_offline_egress_chokepoint.py tests/test_learn tests/test_evals tests/test_cli_learn.py tests/test_memory_eval.py tests/test_verbosity_learn.py tests/test_learn_grok_plugin.py -q ============= 628 passed, 6 skipped, 1 warning in 73.68s (0:01:13) ============= ``` ## Real Behavior Proof - Environment: macOS 26.2, Python 3.14.7, repo deps via `uv sync --extra dev`. To avoid sending a real session digest, I built a throwaway `--agent claude` project with `CLAUDE_CONFIG_DIR` pointing at a temp dir: one synthetic session with a single failed `Read`. `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` were unset for every run. - Exact command / steps: the four `HEADROOM_OFFLINE=1` commands below, run against a synthetic session. - Observed result: each command now stops with the HEADROOM_OFFLINE refusal instead of reaching litellm. Output per command: `HEADROOM_OFFLINE=1 headroom learn --agent claude --project <tmp>/proj --model claude-sonnet-4-6` Before (`main`): the analysis went straight to litellm. It only stopped because no key was set: ```text Analyzing with claude-sonnet-4-6... LLM analysis failed: litellm.AuthenticationError: Missing Anthropic API Key - A call is being made to anthropic but no key is set ... ``` After (exit code 1): ```text Analyzing with claude-sonnet-4-6... Error: HEADROOM_OFFLINE is set: refusing outbound network access for claude-sonnet-4-6 API for headroom learn session analysis to https://api.anthropic.com. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. ``` `HEADROOM_OFFLINE=1 headroom learn --agent claude --project <tmp>/proj --verbosity --llm-judge` ```text Error: HEADROOM_OFFLINE is set: refusing outbound network access for claude-sonnet-4-6 API for the headroom learn --llm-judge verbosity judge. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. ``` `HEADROOM_OFFLINE=1 headroom evals memory -n 1 --answer-model gpt-4o` (exit code 1) ```text Error: HEADROOM_OFFLINE is set: refusing outbound network access for gpt-4o API for the memory eval answers. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. ``` `HEADROOM_OFFLINE=1 headroom evals memory -n 1 --no-extract` with an empty LoCoMo cache (exit code 1) ```text Running evaluation... Error: HEADROOM_OFFLINE is set: refusing outbound network access for LoCoMo eval-dataset download to https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json. Unset HEADROOM_OFFLINE, or turn off the feature that needs this connection. ``` Without `HEADROOM_OFFLINE`, the same `headroom learn` command behaves as before (it reaches litellm, then fails on the missing key), so the online path is unchanged. - Not tested: any run with a real API key or real session data, and the `memory-v2` CLI end to end. The v2 save/answer guards are covered by the unit test only. Windows was not tested. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a - Stable/default behavior changed: only under `HEADROOM_OFFLINE=1`. `headroom learn` (LiteLLM models), `--llm-judge`, the memory evals' LiteLLM calls and the LoCoMo download now refuse with the standard air-gap error instead of dialing out. Nothing changes with the switch unset. - Kill switch / disable path: unset `HEADROOM_OFFLINE` - Unsafe override required: no - Qualification impact: none - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`, it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - **Left out on purpose: the `api_base` point.** The issue also points out that `_call_llm` forces `api_base="https://api.anthropic.com"` for `claude*` models, which bypasses an operator-configured `ANTHROPIC_BASE_URL`. That override exists to stop `learn` from going through the user's local Headroom proxy, so changing it is a design decision and I'd rather keep it out of this fix. The guard still names that host as the destination. - **Overlap with #4242.** #4242 (litellm's import-time pricing fetch) touches most of the same files, adding `pin_litellm_cost_map_local()` and its import. To stay out of its hunks, I import `guard_egress` inside the function in those files and put each guard away from the `import litellm` lines. A trial merge of #4242 on top of this branch merged cleanly, and the full `tests/test_offline_egress_chokepoint.py` passed on the merged tree (176 passed, 2 skipped). - **What the scanner still can't see.** `runner_v2.py` passes `litellm.completion` as an argument to `asyncio.to_thread`, so the AST scanner doesn't see those two sites as calls. They're guarded anyway and covered by `test_memory_eval_v2_save_and_answer_are_refused`. - **Not touched: the `-cli` models in learn.** `_call_cli_llm` (the `claude -p` / `-cli` model path in learn) spawns the user's own agent CLI as a subprocess, not a Headroom HTTP client. It's outside what this issue and the meta-test cover, so I didn't change it. - **Pre-existing flake (not caused by this PR).** On `main`, `TestHeadroomCloudCompressionOffline::test_asgi_middleware_opens_no_socket` fails when that class runs on its own, because importing litellm under the socket trap tries to fetch `raw.githubusercontent.com`. It passes in the full run. That fetch is the import-time download that #4242 addresses. --------- Co-authored-by: Abelo9996 <Abelo9996@users.noreply.github.com> Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
co-authored by
Abelo9996
JD Davis
parent
911c75ffdd
commit
dd361bb7a3
@@ -481,6 +481,9 @@ def _run_memory_eval(
|
||||
# Create answer function based on provider
|
||||
answer_fn = None
|
||||
if answer_model:
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
guard_egress(f"{answer_model} API for the memory eval answers")
|
||||
try:
|
||||
pin_litellm_cost_map_local()
|
||||
import litellm
|
||||
|
||||
@@ -387,6 +387,9 @@ def _make_llm_judge(model: str) -> Any:
|
||||
best-effort: any failure (no key, parse error) returns None so the caller
|
||||
falls back to the heuristic.
|
||||
"""
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
guard_egress(f"{model} API for the headroom learn --llm-judge verbosity judge")
|
||||
|
||||
def judge(signals: dict) -> tuple[int, str] | None:
|
||||
try:
|
||||
|
||||
@@ -176,6 +176,8 @@ def create_litellm_judge(
|
||||
"LiteLLM package required for LLM judge. Install with: pip install litellm"
|
||||
) from e
|
||||
|
||||
guard_egress(f"{model} API for the memory eval LLM judge")
|
||||
|
||||
def judge(question: str, ground_truth: str, prediction: str) -> tuple[float, str]:
|
||||
prompt = JUDGE_PROMPT.format(
|
||||
question=question,
|
||||
|
||||
@@ -29,6 +29,8 @@ from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# LoCoMo dataset URL
|
||||
@@ -231,6 +233,7 @@ def download_locomo(cache_dir: Path | None = None) -> Path:
|
||||
logger.info(f"Using cached LoCoMo dataset: {cache_path}")
|
||||
return cache_path
|
||||
|
||||
guard_egress("LoCoMo eval-dataset download", LOCOMO_URL)
|
||||
logger.info(f"Downloading LoCoMo dataset from {LOCOMO_URL}...")
|
||||
urllib.request.urlretrieve(LOCOMO_URL, cache_path) # nosec B310
|
||||
logger.info(f"Downloaded to {cache_path}")
|
||||
|
||||
@@ -323,6 +323,9 @@ OUTPUT FORMAT (JSON only):
|
||||
|
||||
IMPORTANT: Every event MUST have a specific date. If you cannot determine the date, state it as "around {session_datetime}"."""
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
guard_egress(f"{self.config.extraction_model} API for memory eval extraction")
|
||||
try:
|
||||
response = litellm.completion(
|
||||
model=self.config.extraction_model,
|
||||
|
||||
@@ -467,6 +467,9 @@ Categories to use:
|
||||
# Get just the save tool
|
||||
save_tool = next(t for t in MEMORY_TOOLS if t["function"]["name"] == "memory_save")
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
guard_egress(f"{self._config.save_model} API for the memory eval save phase")
|
||||
for session in conversation.sessions:
|
||||
# Format session as dialogue
|
||||
session_text = f"\n[Session Date: {session.datetime}]\n"
|
||||
@@ -689,6 +692,9 @@ The answer should contain ONLY the specific information requested, nothing more.
|
||||
max_rounds = 3
|
||||
final_answer = "Information not found"
|
||||
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
guard_egress(f"{self._config.answer_model} API for the memory eval answers")
|
||||
for round_num in range(max_rounds):
|
||||
try:
|
||||
response = await asyncio.to_thread(
|
||||
|
||||
@@ -1093,7 +1093,12 @@ def _call_llm(
|
||||
the provider: "claude-*" → Anthropic, "gpt-*" → OpenAI, "gemini/*" → Google, etc.
|
||||
For CLI-based models (ending in "-cli"), delegates to ``_call_cli_llm``.
|
||||
"""
|
||||
from headroom.offline import guard_egress
|
||||
|
||||
if model in _CLI_MODEL_IDS:
|
||||
# The provider CLI sends the digest to its own API, so refuse before
|
||||
# spawning it rather than letting the subprocess reach the network.
|
||||
guard_egress(f"{model} provider CLI for headroom learn session analysis")
|
||||
return _call_cli_llm(digest, model, on_progress=on_progress)
|
||||
|
||||
pin_litellm_cost_map_local()
|
||||
@@ -1108,6 +1113,7 @@ def _call_llm(
|
||||
if model.startswith("claude"):
|
||||
api_base = "https://api.anthropic.com"
|
||||
|
||||
guard_egress(f"{model} API for headroom learn session analysis", api_base)
|
||||
response = litellm.completion(
|
||||
model=model,
|
||||
messages=[
|
||||
|
||||
@@ -2147,6 +2147,167 @@ class TestEvalRunnerLocalProvider:
|
||||
runner._init_llm_client()
|
||||
|
||||
|
||||
class TestLearnAndMemoryEvalOffline:
|
||||
"""`headroom learn` and the memory evals reach their LLM through litellm,
|
||||
and the LoCoMo loader downloads with urlretrieve. None of them consulted
|
||||
the switch (#4338). litellm is replaced by a recorder so a refused path
|
||||
is seen to stop before the request, not just to raise somewhere."""
|
||||
|
||||
@pytest.fixture
|
||||
def litellm_calls(self, monkeypatch: pytest.MonkeyPatch) -> list[dict]:
|
||||
import sys
|
||||
|
||||
calls: list[dict] = []
|
||||
fake = SimpleNamespace(
|
||||
completion=lambda **kwargs: calls.append(kwargs),
|
||||
suppress_debug_info=False,
|
||||
)
|
||||
monkeypatch.setitem(sys.modules, "litellm", fake)
|
||||
return calls
|
||||
|
||||
def test_learn_analysis_is_refused(
|
||||
self, offline: None, no_sockets: None, litellm_calls: list[dict]
|
||||
) -> None:
|
||||
from headroom.learn import analyzer
|
||||
|
||||
with pytest.raises(OfflineEgressBlocked) as excinfo:
|
||||
analyzer._call_llm("digest", "claude-sonnet-4-6")
|
||||
assert _refused_hostname(excinfo.value) == "api.anthropic.com"
|
||||
assert litellm_calls == []
|
||||
|
||||
@pytest.fixture
|
||||
def cli_calls(self, monkeypatch: pytest.MonkeyPatch) -> list[str]:
|
||||
"""Record the provider-CLI delegate and fail on any spawned process."""
|
||||
from headroom.learn import analyzer
|
||||
|
||||
calls: list[str] = []
|
||||
|
||||
def no_process(*args: object, **kwargs: object) -> None:
|
||||
raise AssertionError(f"learn spawned a provider CLI: {args!r}")
|
||||
|
||||
def fake_cli(digest: str, model: str, on_progress: object = None) -> dict:
|
||||
calls.append(model)
|
||||
return {"context_file_rules": []}
|
||||
|
||||
monkeypatch.setattr(analyzer, "_call_cli_llm", fake_cli)
|
||||
monkeypatch.setattr(analyzer, "Popen", no_process)
|
||||
monkeypatch.setattr(analyzer, "run", no_process)
|
||||
monkeypatch.setattr(analyzer.subprocess, "Popen", no_process)
|
||||
monkeypatch.setattr(analyzer.subprocess, "run", no_process)
|
||||
return calls
|
||||
|
||||
@pytest.mark.parametrize("model", ["claude-cli", "gemini-cli", "codex-cli", "agy-cli"])
|
||||
def test_learn_cli_analysis_is_refused_before_the_cli_runs(
|
||||
self, offline: None, no_sockets: None, cli_calls: list[str], model: str
|
||||
) -> None:
|
||||
from headroom.learn import analyzer
|
||||
|
||||
assert model in analyzer._CLI_MODEL_IDS
|
||||
with pytest.raises(OfflineEgressBlocked, match=model):
|
||||
analyzer._call_llm("digest", model)
|
||||
assert cli_calls == []
|
||||
|
||||
def test_learn_cli_analysis_still_runs_online(
|
||||
self, monkeypatch: pytest.MonkeyPatch, cli_calls: list[str]
|
||||
) -> None:
|
||||
from headroom.learn import analyzer
|
||||
|
||||
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
|
||||
assert analyzer._call_llm("digest", "claude-cli") == {"context_file_rules": []}
|
||||
assert cli_calls == ["claude-cli"]
|
||||
|
||||
def test_learn_verbosity_llm_judge_is_refused(
|
||||
self, offline: None, no_sockets: None, litellm_calls: list[dict]
|
||||
) -> None:
|
||||
from headroom.cli import learn
|
||||
|
||||
with pytest.raises(OfflineEgressBlocked):
|
||||
learn._make_llm_judge("gpt-4o")
|
||||
assert litellm_calls == []
|
||||
|
||||
def test_memory_eval_answer_model_is_refused(
|
||||
self, offline: None, no_sockets: None, litellm_calls: list[dict]
|
||||
) -> None:
|
||||
from headroom.cli import evals
|
||||
|
||||
with pytest.raises(OfflineEgressBlocked):
|
||||
evals._run_memory_eval(
|
||||
n_conversations=1,
|
||||
categories=None,
|
||||
include_adversarial=False,
|
||||
top_k=10,
|
||||
f1_threshold=0.5,
|
||||
answer_model="gpt-4o",
|
||||
llm_judge=False,
|
||||
judge_provider="simple",
|
||||
judge_model="gpt-4o",
|
||||
output=None,
|
||||
no_extract=True,
|
||||
extraction_model="gpt-4o-mini",
|
||||
pass_all=False,
|
||||
parallel=1,
|
||||
debug=False,
|
||||
)
|
||||
assert litellm_calls == []
|
||||
|
||||
def test_memory_eval_litellm_judge_is_refused(
|
||||
self, offline: None, no_sockets: None, litellm_calls: list[dict]
|
||||
) -> None:
|
||||
from headroom.evals.memory.judge import create_litellm_judge
|
||||
|
||||
with pytest.raises(OfflineEgressBlocked):
|
||||
create_litellm_judge(model="gpt-4o")
|
||||
assert litellm_calls == []
|
||||
|
||||
def test_memory_eval_extraction_is_refused(
|
||||
self, offline: None, no_sockets: None, litellm_calls: list[dict]
|
||||
) -> None:
|
||||
from headroom.evals.memory.runner import LoCoMoEvaluator
|
||||
|
||||
evaluator = LoCoMoEvaluator()
|
||||
with pytest.raises(OfflineEgressBlocked):
|
||||
evaluator._extract_memories_from_session("A: hi", "1 May 2023", "A", "B")
|
||||
assert litellm_calls == []
|
||||
|
||||
def test_memory_eval_v2_save_and_answer_are_refused(
|
||||
self, offline: None, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
import asyncio
|
||||
|
||||
from headroom.evals.memory import runner_v2
|
||||
|
||||
calls: list[dict] = []
|
||||
monkeypatch.setattr(
|
||||
runner_v2, "litellm", SimpleNamespace(completion=lambda **kw: calls.append(kw))
|
||||
)
|
||||
evaluator = runner_v2.LoCoMoEvaluatorV2()
|
||||
conversation = SimpleNamespace(speaker_a="A", speaker_b="B", sessions=[])
|
||||
with pytest.raises(OfflineEgressBlocked):
|
||||
asyncio.run(evaluator._replay_conversation(conversation))
|
||||
with pytest.raises(OfflineEgressBlocked):
|
||||
asyncio.run(evaluator._answer_question(SimpleNamespace(question="Who?"), conversation))
|
||||
assert calls == []
|
||||
|
||||
def test_locomo_download_opens_no_socket(
|
||||
self, offline: None, no_sockets: None, tmp_path: Path
|
||||
) -> None:
|
||||
from headroom.evals.memory.locomo import download_locomo
|
||||
|
||||
with pytest.raises(OfflineEgressBlocked) as excinfo:
|
||||
download_locomo(cache_dir=tmp_path)
|
||||
assert _refused_hostname(excinfo.value) == "raw.githubusercontent.com"
|
||||
assert not (tmp_path / "locomo10.json").exists()
|
||||
|
||||
def test_a_cached_locomo_dataset_still_loads_offline(
|
||||
self, offline: None, no_sockets: None, tmp_path: Path
|
||||
) -> None:
|
||||
from headroom.evals.memory.locomo import download_locomo
|
||||
|
||||
cached = tmp_path / "locomo10.json"
|
||||
cached.write_text("[]", encoding="utf-8")
|
||||
assert download_locomo(cache_dir=tmp_path) == cached
|
||||
|
||||
|
||||
class TestHeadroomCloudCompressionOffline:
|
||||
"""Both Headroom Cloud integrations.
|
||||
|
||||
@@ -2407,6 +2568,7 @@ _EGRESS_CALLEES = re.compile(
|
||||
| httpx\.(?:get|post|put|patch|delete|head|request|stream) # module-level verbs
|
||||
| requests\.(?:get|post|put|patch|delete|head|request|Session)
|
||||
| (?:urllib\.request\.)?urlopen | _urlopen # urllib, bare or wrapped
|
||||
| (?:urllib\.request\.)?urlretrieve # urllib file download
|
||||
| aiohttp\.ClientSession
|
||||
| urllib3\.PoolManager
|
||||
| (?:huggingface_hub\.)?hf_hub_download # the Python half of the HF fetch
|
||||
@@ -2418,6 +2580,8 @@ _EGRESS_CALLEES = re.compile(
|
||||
| OTLP(?:Metric|Span|Log)Exporter # OTEL export, incl. its background timer
|
||||
| (?:openai\.)?(?:Async)?(?:OpenAI|AzureOpenAI) # provider SDKs build their own
|
||||
| (?:anthropic\.)?(?:Async)?Anthropic(?:Bedrock|Vertex)?
|
||||
| litellm\.a?completion # litellm-routed provider calls
|
||||
| _call_cli_llm # learn's provider CLIs call their APIs
|
||||
)$""",
|
||||
re.VERBOSE,
|
||||
)
|
||||
@@ -2668,6 +2832,13 @@ _EGRESS_ALLOWLIST: dict[str, tuple[int, str]] = {
|
||||
"deployment that points Headroom at an on-prem model endpoint, which "
|
||||
"is the main reason such a deployment exists.",
|
||||
),
|
||||
"backends/litellm.py": (
|
||||
4,
|
||||
"user-traffic: LiteLLMBackend's send/stream paths (Anthropic and "
|
||||
"OpenAI shaped) forward the caller's request to the provider the "
|
||||
"operator configured as the proxy backend (Bedrock, Vertex, "
|
||||
"OpenRouter, ...). Same role as the proxy's own forwarding above.",
|
||||
),
|
||||
"telemetry/session.py": (
|
||||
1,
|
||||
"gated: the beacon upload is reached only via is_telemetry_enabled(), "
|
||||
|
||||
Reference in New Issue
Block a user