fix(offline): guard the learn and memory-eval LLM calls under HEADROOM_OFFLINE (#4344)

## Description

`HEADROOM_OFFLINE=1` didn't cover `headroom learn` or the memory evals.
Every one of these dialed out with no `guard_egress` on the path:

- `learn/analyzer.py` `_call_llm`: `litellm.completion` for `headroom
learn` (dry run included, since the analysis runs before the `--apply`
gate)
- `cli/learn.py` `_make_llm_judge`: `litellm.completion` for `headroom
learn --verbosity --llm-judge`
- `cli/evals.py` `_run_memory_eval`: `litellm.completion` for
`--answer-model`
- `evals/memory/runner.py` `_extract_memories_from_session`:
`litellm.completion` for memory extraction (on by default)
- `evals/memory/runner_v2.py` `_replay_conversation` and
`_answer_question`: `litellm.completion` passed to `asyncio.to_thread`
- `evals/memory/judge.py` `create_litellm_judge`: `litellm.completion`.
The OpenAI and Anthropic builders in the same file were already guarded.
- `evals/memory/locomo.py` `download_locomo`:
`urllib.request.urlretrieve` for the dataset

The chokepoint meta-test
(`test_every_egress_site_is_guarded_or_allowlisted`) passed on `main`
because `_EGRESS_CALLEES` matched neither `litellm.completion` nor
`urlretrieve`.

This PR adds a `guard_egress(...)` call before each of those calls.
Where the callable gets built up front (the `--llm-judge` judge, the
`--answer-model` answer function, the litellm judge factory), the guard
sits in the builder, so the command refuses before the eval starts and
not halfway through it. The LoCoMo guard comes after the cache check, so
a pre-seeded dataset still loads offline. I also added
`litellm\.a?completion` and `(?:urllib\.request\.)?urlretrieve` to the
meta-test's callee pattern so the next site like these can't slip
through.

Fixes #4338

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/learn/analyzer.py`, `headroom/cli/learn.py`,
`headroom/cli/evals.py`, `headroom/evals/memory/runner.py`,
`headroom/evals/memory/runner_v2.py`, `headroom/evals/memory/judge.py`,
`headroom/evals/memory/locomo.py`: add a `guard_egress(purpose,
destination)` call that dominates each call site. Each purpose string
names the model and feature, so the refusal tells the operator what to
turn off.
- `tests/test_offline_egress_chokepoint.py`:
- `_EGRESS_CALLEES` now also matches `litellm.completion` /
`litellm.acompletion` and `urlretrieve`.
- Widening the pattern also exposes the four `acompletion` sites in
`headroom/backends/litellm.py`, which are `LiteLLMBackend`'s send/stream
paths. Those forward the caller's request to the provider the operator
configured as the proxy backend (Bedrock, Vertex, OpenRouter, ...), so I
allowlisted them as `user-traffic` with a count of 4, the same treatment
as the proxy's own forwarding in `proxy/server.py`. Please check that
you agree with this classification.
- New `TestLearnAndMemoryEvalOffline`: one test per path, with `litellm`
swapped for a recorder. Each test asserts that `OfflineEgressBlocked` is
raised and that no completion call was made. The LoCoMo test checks that
no socket opens and that the refusal names `raw.githubusercontent.com`.
There is also a test that a cached LoCoMo file still loads offline.
- Follow-up from review: `headroom learn`'s provider-CLI backends
(claude-cli, gemini-cli, codex-cli, agy-cli) are now refused before
`_call_cli_llm` runs, and the chokepoint inventory treats
`_call_cli_llm` as an egress call.

## Testing

- [x] Unit tests pass (`pytest`):
`tests/test_offline_egress_chokepoint.py tests/test_learn
tests/test_evals tests/test_cli_learn.py tests/test_memory_eval.py
tests/test_verbosity_learn.py tests/test_learn_grok_plugin.py`: 628
passed, 6 skipped
- [x] Linting passes (`ruff check .`): `ruff check` and `ruff format
--check` on the 8 changed files
- [x] Type checking passes (`mypy headroom`): `mypy` on the 7 changed
source files, no issues
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

With the source changes reverted and the new tests kept (`main`
behavior):

```text
$ pytest tests/test_offline_egress_chokepoint.py -q -k "TestLearnAndMemoryEvalOffline or TestEgressChokepointCoverage"
E       headroom/cli/evals.py
E             line 510: response = litellm.completion(
E       headroom/cli/learn.py
E             line 406: resp = litellm.completion(
E       headroom/evals/memory/judge.py
E             line 185: response = litellm.completion(
E       headroom/evals/memory/locomo.py
E             line 235: urllib.request.urlretrieve(LOCOMO_URL, cache_path)  # nosec B310
E       headroom/evals/memory/runner.py
E             line 325: response = litellm.completion(
E       headroom/learn/analyzer.py
E             line 1109: response = litellm.completion(
FAILED ...::TestLearnAndMemoryEvalOffline::test_learn_analysis_is_refused
FAILED ...::TestLearnAndMemoryEvalOffline::test_learn_verbosity_llm_judge_is_refused
FAILED ...::TestLearnAndMemoryEvalOffline::test_memory_eval_answer_model_is_refused
FAILED ...::TestLearnAndMemoryEvalOffline::test_memory_eval_litellm_judge_is_refused
FAILED ...::TestLearnAndMemoryEvalOffline::test_memory_eval_extraction_is_refused
FAILED ...::TestLearnAndMemoryEvalOffline::test_memory_eval_v2_save_and_answer_are_refused
FAILED ...::TestLearnAndMemoryEvalOffline::test_locomo_download_opens_no_socket
FAILED ...::TestEgressChokepointCoverage::test_every_egress_site_is_guarded_or_allowlisted
================= 8 failed, 5 passed, 137 deselected in 23.81s =================
```

With the fix:

```text
$ pytest tests/test_offline_egress_chokepoint.py -q -k "TestLearnAndMemoryEvalOffline or TestEgressChokepointCoverage"
====================== 13 passed, 137 deselected in 9.93s ======================

$ pytest tests/test_offline_egress_chokepoint.py tests/test_learn tests/test_evals tests/test_cli_learn.py tests/test_memory_eval.py tests/test_verbosity_learn.py tests/test_learn_grok_plugin.py -q
============= 628 passed, 6 skipped, 1 warning in 73.68s (0:01:13) =============
```

## Real Behavior Proof

- Environment: macOS 26.2, Python 3.14.7, repo deps via `uv sync --extra
dev`. To avoid sending a real session digest, I built a throwaway
`--agent claude` project with `CLAUDE_CONFIG_DIR` pointing at a temp
dir: one synthetic session with a single failed `Read`.
`ANTHROPIC_API_KEY` / `OPENAI_API_KEY` were unset for every run.
- Exact command / steps: the four `HEADROOM_OFFLINE=1` commands below,
run against a synthetic session.
- Observed result: each command now stops with the HEADROOM_OFFLINE
refusal instead of reaching litellm. Output per command:

`HEADROOM_OFFLINE=1 headroom learn --agent claude --project <tmp>/proj
--model claude-sonnet-4-6`

Before (`main`): the analysis went straight to litellm. It only stopped
because no key was set:
  ```text
    Analyzing with claude-sonnet-4-6...
LLM analysis failed: litellm.AuthenticationError: Missing Anthropic API
Key - A call is being made to anthropic but no key is set ...
  ```
  After (exit code 1):
  ```text
    Analyzing with claude-sonnet-4-6...
Error: HEADROOM_OFFLINE is set: refusing outbound network access for
claude-sonnet-4-6 API for headroom learn session analysis to
https://api.anthropic.com. Unset HEADROOM_OFFLINE, or turn off the
feature that needs this connection.
  ```

`HEADROOM_OFFLINE=1 headroom learn --agent claude --project <tmp>/proj
--verbosity --llm-judge`
  ```text
Error: HEADROOM_OFFLINE is set: refusing outbound network access for
claude-sonnet-4-6 API for the headroom learn --llm-judge verbosity
judge. Unset HEADROOM_OFFLINE, or turn off the feature that needs this
connection.
  ```

`HEADROOM_OFFLINE=1 headroom evals memory -n 1 --answer-model gpt-4o`
(exit code 1)
  ```text
Error: HEADROOM_OFFLINE is set: refusing outbound network access for
gpt-4o API for the memory eval answers. Unset HEADROOM_OFFLINE, or turn
off the feature that needs this connection.
  ```

`HEADROOM_OFFLINE=1 headroom evals memory -n 1 --no-extract` with an
empty LoCoMo cache (exit code 1)
  ```text
  Running evaluation...
Error: HEADROOM_OFFLINE is set: refusing outbound network access for
LoCoMo eval-dataset download to
https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json.
Unset HEADROOM_OFFLINE, or turn off the feature that needs this
connection.
  ```

Without `HEADROOM_OFFLINE`, the same `headroom learn` command behaves as
before (it reaches litellm, then fails on the missing key), so the
online path is unchanged.

- Not tested: any run with a real API key or real session data, and the
`memory-v2` CLI end to end. The v2 save/answer guards are covered by the
unit test only. Windows was not tested.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: n/a
- Stable/default behavior changed: only under `HEADROOM_OFFLINE=1`.
`headroom learn` (LiteLLM models), `--llm-judge`, the memory evals'
LiteLLM calls and the LoCoMo download now refuse with the standard
air-gap error instead of dialing out. Nothing changes with the switch
unset.
- Kill switch / disable path: unset `HEADROOM_OFFLINE`
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert this commit

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`, it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A

## Additional Notes

- **Left out on purpose: the `api_base` point.** The issue also points
out that `_call_llm` forces `api_base="https://api.anthropic.com"` for
`claude*` models, which bypasses an operator-configured
`ANTHROPIC_BASE_URL`. That override exists to stop `learn` from going
through the user's local Headroom proxy, so changing it is a design
decision and I'd rather keep it out of this fix. The guard still names
that host as the destination.
- **Overlap with #4242.** #4242 (litellm's import-time pricing fetch)
touches most of the same files, adding `pin_litellm_cost_map_local()`
and its import. To stay out of its hunks, I import `guard_egress` inside
the function in those files and put each guard away from the `import
litellm` lines. A trial merge of #4242 on top of this branch merged
cleanly, and the full `tests/test_offline_egress_chokepoint.py` passed
on the merged tree (176 passed, 2 skipped).
- **What the scanner still can't see.** `runner_v2.py` passes
`litellm.completion` as an argument to `asyncio.to_thread`, so the AST
scanner doesn't see those two sites as calls. They're guarded anyway and
covered by `test_memory_eval_v2_save_and_answer_are_refused`.
- **Not touched: the `-cli` models in learn.** `_call_cli_llm` (the
`claude -p` / `-cli` model path in learn) spawns the user's own agent
CLI as a subprocess, not a Headroom HTTP client. It's outside what this
issue and the meta-test cover, so I didn't change it.
- **Pre-existing flake (not caused by this PR).** On `main`,
`TestHeadroomCloudCompressionOffline::test_asgi_middleware_opens_no_socket`
fails when that class runs on its own, because importing litellm under
the socket trap tries to fetch `raw.githubusercontent.com`. It passes in
the full run. That fetch is the import-time download that #4242
addresses.

---------

Co-authored-by: Abelo9996 <Abelo9996@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This commit is contained in:
Abel Yagubyan
2026-11-04 22:55:08 -06:00
committed by GitHub
co-authored by Abelo9996 JD Davis
parent 911c75ffdd
commit dd361bb7a3
8 changed files with 197 additions and 0 deletions
+3
View File
@@ -481,6 +481,9 @@ def _run_memory_eval(
# Create answer function based on provider
answer_fn = None
if answer_model:
from headroom.offline import guard_egress
guard_egress(f"{answer_model} API for the memory eval answers")
try:
pin_litellm_cost_map_local()
import litellm
+3
View File
@@ -387,6 +387,9 @@ def _make_llm_judge(model: str) -> Any:
best-effort: any failure (no key, parse error) returns None so the caller
falls back to the heuristic.
"""
from headroom.offline import guard_egress
guard_egress(f"{model} API for the headroom learn --llm-judge verbosity judge")
def judge(signals: dict) -> tuple[int, str] | None:
try:
+2
View File
@@ -176,6 +176,8 @@ def create_litellm_judge(
"LiteLLM package required for LLM judge. Install with: pip install litellm"
) from e
guard_egress(f"{model} API for the memory eval LLM judge")
def judge(question: str, ground_truth: str, prediction: str) -> tuple[float, str]:
prompt = JUDGE_PROMPT.format(
question=question,
+3
View File
@@ -29,6 +29,8 @@ from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
from headroom.offline import guard_egress
logger = logging.getLogger(__name__)
# LoCoMo dataset URL
@@ -231,6 +233,7 @@ def download_locomo(cache_dir: Path | None = None) -> Path:
logger.info(f"Using cached LoCoMo dataset: {cache_path}")
return cache_path
guard_egress("LoCoMo eval-dataset download", LOCOMO_URL)
logger.info(f"Downloading LoCoMo dataset from {LOCOMO_URL}...")
urllib.request.urlretrieve(LOCOMO_URL, cache_path) # nosec B310
logger.info(f"Downloaded to {cache_path}")
+3
View File
@@ -323,6 +323,9 @@ OUTPUT FORMAT (JSON only):
IMPORTANT: Every event MUST have a specific date. If you cannot determine the date, state it as "around {session_datetime}"."""
from headroom.offline import guard_egress
guard_egress(f"{self.config.extraction_model} API for memory eval extraction")
try:
response = litellm.completion(
model=self.config.extraction_model,
+6
View File
@@ -467,6 +467,9 @@ Categories to use:
# Get just the save tool
save_tool = next(t for t in MEMORY_TOOLS if t["function"]["name"] == "memory_save")
from headroom.offline import guard_egress
guard_egress(f"{self._config.save_model} API for the memory eval save phase")
for session in conversation.sessions:
# Format session as dialogue
session_text = f"\n[Session Date: {session.datetime}]\n"
@@ -689,6 +692,9 @@ The answer should contain ONLY the specific information requested, nothing more.
max_rounds = 3
final_answer = "Information not found"
from headroom.offline import guard_egress
guard_egress(f"{self._config.answer_model} API for the memory eval answers")
for round_num in range(max_rounds):
try:
response = await asyncio.to_thread(
+6
View File
@@ -1093,7 +1093,12 @@ def _call_llm(
the provider: "claude-*" → Anthropic, "gpt-*" → OpenAI, "gemini/*" → Google, etc.
For CLI-based models (ending in "-cli"), delegates to ``_call_cli_llm``.
"""
from headroom.offline import guard_egress
if model in _CLI_MODEL_IDS:
# The provider CLI sends the digest to its own API, so refuse before
# spawning it rather than letting the subprocess reach the network.
guard_egress(f"{model} provider CLI for headroom learn session analysis")
return _call_cli_llm(digest, model, on_progress=on_progress)
pin_litellm_cost_map_local()
@@ -1108,6 +1113,7 @@ def _call_llm(
if model.startswith("claude"):
api_base = "https://api.anthropic.com"
guard_egress(f"{model} API for headroom learn session analysis", api_base)
response = litellm.completion(
model=model,
messages=[
+171
View File
@@ -2147,6 +2147,167 @@ class TestEvalRunnerLocalProvider:
runner._init_llm_client()
class TestLearnAndMemoryEvalOffline:
"""`headroom learn` and the memory evals reach their LLM through litellm,
and the LoCoMo loader downloads with urlretrieve. None of them consulted
the switch (#4338). litellm is replaced by a recorder so a refused path
is seen to stop before the request, not just to raise somewhere."""
@pytest.fixture
def litellm_calls(self, monkeypatch: pytest.MonkeyPatch) -> list[dict]:
import sys
calls: list[dict] = []
fake = SimpleNamespace(
completion=lambda **kwargs: calls.append(kwargs),
suppress_debug_info=False,
)
monkeypatch.setitem(sys.modules, "litellm", fake)
return calls
def test_learn_analysis_is_refused(
self, offline: None, no_sockets: None, litellm_calls: list[dict]
) -> None:
from headroom.learn import analyzer
with pytest.raises(OfflineEgressBlocked) as excinfo:
analyzer._call_llm("digest", "claude-sonnet-4-6")
assert _refused_hostname(excinfo.value) == "api.anthropic.com"
assert litellm_calls == []
@pytest.fixture
def cli_calls(self, monkeypatch: pytest.MonkeyPatch) -> list[str]:
"""Record the provider-CLI delegate and fail on any spawned process."""
from headroom.learn import analyzer
calls: list[str] = []
def no_process(*args: object, **kwargs: object) -> None:
raise AssertionError(f"learn spawned a provider CLI: {args!r}")
def fake_cli(digest: str, model: str, on_progress: object = None) -> dict:
calls.append(model)
return {"context_file_rules": []}
monkeypatch.setattr(analyzer, "_call_cli_llm", fake_cli)
monkeypatch.setattr(analyzer, "Popen", no_process)
monkeypatch.setattr(analyzer, "run", no_process)
monkeypatch.setattr(analyzer.subprocess, "Popen", no_process)
monkeypatch.setattr(analyzer.subprocess, "run", no_process)
return calls
@pytest.mark.parametrize("model", ["claude-cli", "gemini-cli", "codex-cli", "agy-cli"])
def test_learn_cli_analysis_is_refused_before_the_cli_runs(
self, offline: None, no_sockets: None, cli_calls: list[str], model: str
) -> None:
from headroom.learn import analyzer
assert model in analyzer._CLI_MODEL_IDS
with pytest.raises(OfflineEgressBlocked, match=model):
analyzer._call_llm("digest", model)
assert cli_calls == []
def test_learn_cli_analysis_still_runs_online(
self, monkeypatch: pytest.MonkeyPatch, cli_calls: list[str]
) -> None:
from headroom.learn import analyzer
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
assert analyzer._call_llm("digest", "claude-cli") == {"context_file_rules": []}
assert cli_calls == ["claude-cli"]
def test_learn_verbosity_llm_judge_is_refused(
self, offline: None, no_sockets: None, litellm_calls: list[dict]
) -> None:
from headroom.cli import learn
with pytest.raises(OfflineEgressBlocked):
learn._make_llm_judge("gpt-4o")
assert litellm_calls == []
def test_memory_eval_answer_model_is_refused(
self, offline: None, no_sockets: None, litellm_calls: list[dict]
) -> None:
from headroom.cli import evals
with pytest.raises(OfflineEgressBlocked):
evals._run_memory_eval(
n_conversations=1,
categories=None,
include_adversarial=False,
top_k=10,
f1_threshold=0.5,
answer_model="gpt-4o",
llm_judge=False,
judge_provider="simple",
judge_model="gpt-4o",
output=None,
no_extract=True,
extraction_model="gpt-4o-mini",
pass_all=False,
parallel=1,
debug=False,
)
assert litellm_calls == []
def test_memory_eval_litellm_judge_is_refused(
self, offline: None, no_sockets: None, litellm_calls: list[dict]
) -> None:
from headroom.evals.memory.judge import create_litellm_judge
with pytest.raises(OfflineEgressBlocked):
create_litellm_judge(model="gpt-4o")
assert litellm_calls == []
def test_memory_eval_extraction_is_refused(
self, offline: None, no_sockets: None, litellm_calls: list[dict]
) -> None:
from headroom.evals.memory.runner import LoCoMoEvaluator
evaluator = LoCoMoEvaluator()
with pytest.raises(OfflineEgressBlocked):
evaluator._extract_memories_from_session("A: hi", "1 May 2023", "A", "B")
assert litellm_calls == []
def test_memory_eval_v2_save_and_answer_are_refused(
self, offline: None, monkeypatch: pytest.MonkeyPatch
) -> None:
import asyncio
from headroom.evals.memory import runner_v2
calls: list[dict] = []
monkeypatch.setattr(
runner_v2, "litellm", SimpleNamespace(completion=lambda **kw: calls.append(kw))
)
evaluator = runner_v2.LoCoMoEvaluatorV2()
conversation = SimpleNamespace(speaker_a="A", speaker_b="B", sessions=[])
with pytest.raises(OfflineEgressBlocked):
asyncio.run(evaluator._replay_conversation(conversation))
with pytest.raises(OfflineEgressBlocked):
asyncio.run(evaluator._answer_question(SimpleNamespace(question="Who?"), conversation))
assert calls == []
def test_locomo_download_opens_no_socket(
self, offline: None, no_sockets: None, tmp_path: Path
) -> None:
from headroom.evals.memory.locomo import download_locomo
with pytest.raises(OfflineEgressBlocked) as excinfo:
download_locomo(cache_dir=tmp_path)
assert _refused_hostname(excinfo.value) == "raw.githubusercontent.com"
assert not (tmp_path / "locomo10.json").exists()
def test_a_cached_locomo_dataset_still_loads_offline(
self, offline: None, no_sockets: None, tmp_path: Path
) -> None:
from headroom.evals.memory.locomo import download_locomo
cached = tmp_path / "locomo10.json"
cached.write_text("[]", encoding="utf-8")
assert download_locomo(cache_dir=tmp_path) == cached
class TestHeadroomCloudCompressionOffline:
"""Both Headroom Cloud integrations.
@@ -2407,6 +2568,7 @@ _EGRESS_CALLEES = re.compile(
| httpx\.(?:get|post|put|patch|delete|head|request|stream) # module-level verbs
| requests\.(?:get|post|put|patch|delete|head|request|Session)
| (?:urllib\.request\.)?urlopen | _urlopen # urllib, bare or wrapped
| (?:urllib\.request\.)?urlretrieve # urllib file download
| aiohttp\.ClientSession
| urllib3\.PoolManager
| (?:huggingface_hub\.)?hf_hub_download # the Python half of the HF fetch
@@ -2418,6 +2580,8 @@ _EGRESS_CALLEES = re.compile(
| OTLP(?:Metric|Span|Log)Exporter # OTEL export, incl. its background timer
| (?:openai\.)?(?:Async)?(?:OpenAI|AzureOpenAI) # provider SDKs build their own
| (?:anthropic\.)?(?:Async)?Anthropic(?:Bedrock|Vertex)?
| litellm\.a?completion # litellm-routed provider calls
| _call_cli_llm # learn's provider CLIs call their APIs
)$""",
re.VERBOSE,
)
@@ -2668,6 +2832,13 @@ _EGRESS_ALLOWLIST: dict[str, tuple[int, str]] = {
"deployment that points Headroom at an on-prem model endpoint, which "
"is the main reason such a deployment exists.",
),
"backends/litellm.py": (
4,
"user-traffic: LiteLLMBackend's send/stream paths (Anthropic and "
"OpenAI shaped) forward the caller's request to the provider the "
"operator configured as the proxy backend (Bedrock, Vertex, "
"OpenRouter, ...). Same role as the proxy's own forwarding above.",
),
"telemetry/session.py": (
1,
"gated: the beacon upload is reached only via is_telemetry_enabled(), "