fix(wrap): never reuse a proxy with incompatible routing config (#3201)

## Description

When `headroom wrap <tool> --port P` finds a proxy already listening,
`_ensure_proxy_unlocked` decides reuse-vs-restart by comparing feature
flags (`memory`, `learn`, `code_graph`, savings profile) and, only when
the caller explicitly requests them, `openai_api_url`/`vertex_api_url`.
Two gaps make this silently dangerous:

1. **The backend is never compared** in the shared path (`wrap copilot`
has its own guard; every other tool does not).
2. **URL checks are asymmetric**: a proxy started with a non-default
upstream (e.g. a Copilot tenant URL, or an augment/factory upstream from
a feature-branch build) is "compatible" with every plain wrap that
requests nothing, so unrelated sessions silently inherit captive
routing.

Observed in production: `headroom wrap codex --port 8788` reused a
21-day-old augment-routed proxy (`backend=anthropic`, `augment_api_url`
set, zero clients) because nothing in the reuse check objected; the
session ended up outside Headroom while the wrap reported "Proxy already
running".

This PR adds `_proxy_routing_mismatches` (symmetric comparison of
effective backend, including `HEADROOM_BACKEND`, plus all upstream-URL
keys, where "running has it set, caller did not request it" is a
mismatch) and, on mismatch:

- **zero live clients**: kill the proxy via the `pid` exposed in
`/health` config and restart with the requested config (existing
missing-flags mechanics);
- **live clients attached**: never reuse and never disrupt — start a
dedicated proxy on the next free port (the `wrap auggie` "different
session" precedent);
- **persistent deployment with routing mismatch**: leave the deployment
alone (its routing comes from the manifest; killing its proxy from the
ephemeral path orphans the manifest) and start a dedicated proxy on a
free port;
- **proxy exposes no config block at all**: never reused (it cannot be
compatibility-checked and reports no PID to restart with) — dedicated
port instead.

Closes #3198

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/cli/wrap.py`: new `_PROXY_ROUTING_URL_KEYS` +
`_proxy_routing_mismatches` helper (module-level so tests and future
callers share it; URL keys absent from a running proxy's config compare
equal to unrequested/`None`, so they are inert, never wrong, against
builds that do not expose them). Wired into the persistent-deployment
reuse gate (mismatch now diverts to a dedicated port instead of falling
through to the ephemeral kill path) and into the ephemeral/shared branch
(mismatch restarts when idle, port-bumps when clients are attached;
configless proxies are no longer reused).
- `tests/test_cli/test_wrap_persistent.py`: helper unit tests (backend
via arg and `HEADROOM_BACKEND`, symmetric URL mismatch both directions,
normalization tolerance, missing-key tolerance),
restart-on-routing-mismatch, port-bump-with-clients,
configless-no-reuse, persistent-divert, happy-path reuse. The
pre-existing `..._restarts_persistent_deployment_for_feature_mismatch`
test asserted the old kill-and-hijack behavior for a routing mismatch;
updated to the divert semantic with an explanatory comment.
- `tests/test_cli/test_wrap_helpers.py`: the dashboard-URL regression
test now mocks a config-bearing proxy (configless is covered by the new
dedicated-port test).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_cli/test_wrap_persistent.py -q
38 passed in 0.45s
$ uv run pytest tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_persistent.py -q
96 passed in 0.56s
$ uv run pytest tests/test_cli -q -k 'wrap or unwrap or proxy'
525 passed, 1 skipped, 182 deselected in 12.27s
$ uv run ruff format headroom/cli/wrap.py tests/test_cli/test_wrap_persistent.py tests/test_cli/test_wrap_helpers.py
3 files left unchanged
$ uv run ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_persistent.py tests/test_cli/test_wrap_helpers.py
All checks passed!
$ uv run mypy headroom/cli/wrap.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 26.5 arm64, Python 3.13, local source build
0.37.0-dev (integration overlay with this PR applied), live ports: 8789
(plain 0.37.0-dev proxy), 8791 free.
- Exact command / steps: with the plain proxy healthy on 8789, `python
-c "from headroom.cli import wrap as w; w._ensure_proxy(8789, False,
agent_type='auggie',
augment_api_url='https://xlb.api.augmentcode.com')"`, then `GET /health`
on the resulting port.
- Observed result: `Proxy on port 8789 has incompatible routing config
(augment_api_url); restarting with the requested config...` — the plain
proxy was killed and an augment-routed proxy came up (on 8791: the
just-freed 8789 lost the bind race and 8790 was held by another stale
listener; the fallback messaging named the port change). `/health` on
8791: `version 0.37.0-dev, augment_api_url
https://xlb.api.augmentcode.com`. Before this PR the identical call
printed "Proxy already running on port 8789" and reused the plain proxy.
- Not tested: the attached-clients port-bump path against real live
traffic (unit-tested with mocked markers); the persistent-deployment
divert against a real `install agent` deployment (unit-tested); Windows.

## Runtime Rollout Safety

- Rollout-managed feature(s): none
- Minimum rollout channel: N/A
- Stable/default behavior changed: yes — wraps no longer silently reuse
proxies whose backend/upstream routing differs, and no longer reuse
configless proxies. Previously-correct reuse (identical config) is
unchanged and still prints the dashboard URL.
- Kill switch / disable path: `headroom wrap <tool> --no-proxy` bypasses
the check entirely (pre-existing).
- Unsafe override required: no
- Qualification impact: none
- Rollback path: revert the commit; the previous reuse heuristic
returns.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Adversarially reviewed by a second model (glm-5.2, high reasoning);
initial verdict needs-work, all accepted findings applied (configless
no-reuse, persistent-divert, honest docstring about which URL keys this
build's own `/health` exposes) and re-tested green. Companion PRs (same
incident): `-dev` version-restart fix, orphan-proxy watchdog. Follow-up
for the auggie/droid feature branches: expose
`augment_api_url`/`factory_api_url` in their `/health` config payloads
so same-build detection works there too (this PR already catches
older/foreign builds that expose them).
This commit is contained in:
rNoz
2026-09-28 13:32:34 -05:00
committed by GitHub
parent 4227bd2419
commit f69e2463a0
3 changed files with 1095 additions and 151 deletions
+280 -121
View File
@@ -3668,12 +3668,12 @@ def _proxy_active_session_count(payload: dict[str, Any] | None) -> int:
return max(counts, default=0)
def _normalize_proxy_api_url(url: object) -> str | None:
def _normalize_proxy_api_url(url: object, *, strip_provider_v1: bool = True) -> str | None:
"""Normalize configured upstream URLs for running-proxy comparisons."""
if not isinstance(url, str):
return None
normalized = url.strip().rstrip("/")
if normalized.endswith("/v1"):
if strip_provider_v1 and normalized.endswith("/v1"):
normalized = normalized[:-3]
return normalized or None
@@ -3700,6 +3700,95 @@ def _proxy_needs_version_restart(payload: dict[str, Any] | None) -> bool:
)
# Upstream-URL config keys that decide where a running proxy forwards
# traffic. A running proxy is only reusable when every key matches the
# requested value. This build's /health config payload only serializes the
# first five keys; the augment/factory entries catch proxies built from
# feature branches or other versions that DO expose them (exactly the
# mixed-fleet shape of the 2026-08-22 incident, where a 0.34-era proxy
# reported augment_api_url). Keys absent from the running config simply
# compare equal to an unrequested (None) value, so they are inert, never
# wrong, against proxies that do not expose them.
_PROXY_ROUTING_URL_KEYS = (
"openai_api_url",
"anthropic_api_url",
"vertex_api_url",
"cloudcode_api_url",
"gemini_api_url",
"augment_api_url",
"factory_api_url",
)
def _proxy_routing_mismatches(
running_config: Mapping[str, Any],
*,
backend: str | None,
requested_api_urls: Mapping[str, str | None] | None = None,
) -> list[str]:
"""Return routing-level keys where a running proxy differs from the request.
Unlike the feature-flag check in ``_ensure_proxy_unlocked`` (memory,
learn, code graph), a mismatch here is never "degraded but usable": the
proxy would forward this session's traffic to a backend or upstream the
caller did not ask for. URL comparison is symmetric: a running proxy with
a non-default upstream the caller did not request is a mismatch too.
"""
mismatches: list[str] = []
effective_backend = backend or os.environ.get("HEADROOM_BACKEND") or "anthropic"
running_backend = running_config.get("backend")
if (
isinstance(running_backend, str)
and running_backend
and running_backend != effective_backend
):
mismatches.append("backend")
requested_api_urls = requested_api_urls or {}
for key in _PROXY_ROUTING_URL_KEYS:
strip_provider_v1 = key not in {"augment_api_url", "factory_api_url"}
if _normalize_proxy_api_url(
running_config.get(key),
strip_provider_v1=strip_provider_v1,
) != _normalize_proxy_api_url(
requested_api_urls.get(key),
strip_provider_v1=strip_provider_v1,
):
mismatches.append(key)
return mismatches
def _effective_requested_proxy_routing(
*,
backend: str | None,
openai_api_url: str | None,
anthropic_api_url: str | None,
vertex_api_url: str | None,
clear_vertex_api_url: bool,
) -> tuple[str, dict[str, str | None]]:
"""Resolve the routing a newly started proxy would inherit."""
from headroom.providers.registry import resolve_api_overrides
overrides = resolve_api_overrides(
anthropic_api_url=anthropic_api_url,
openai_api_url=openai_api_url,
gemini_api_url=None,
cloudcode_api_url=None,
vertex_api_url=vertex_api_url,
)
return (
backend or os.environ.get("HEADROOM_BACKEND") or "anthropic",
{
"anthropic_api_url": overrides.anthropic,
"openai_api_url": overrides.openai,
"gemini_api_url": overrides.gemini,
"cloudcode_api_url": overrides.cloudcode,
"vertex_api_url": None if clear_vertex_api_url else overrides.vertex,
"augment_api_url": os.environ.get("AUGMENT_TARGET_API_URL"),
"factory_api_url": os.environ.get("FACTORY_TARGET_API_URL"),
},
)
def _detect_running_proxy_backend(port: int) -> str | None:
"""Read the backend of an already-running proxy from its health endpoint."""
return _copilot_detect_running_proxy_backend(port)
@@ -4161,6 +4250,18 @@ def _ensure_proxy_unlocked(
or bool(copilot_refresh_oauth_token)
or copilot_api_token_expires_at is not None
)
requested_backend, requested_api_urls = helpers._effective_requested_proxy_routing(
backend=backend,
openai_api_url=openai_api_url,
anthropic_api_url=anthropic_api_url,
vertex_api_url=vertex_api_url,
clear_vertex_api_url=clear_vertex_api_url,
)
# Set True when the proxy on the requested port belongs to a persistent
# deployment whose routing config does not match this wrap: the deployment
# owns the port and must not be reused, killed, or restarted from the
# shared ephemeral path; the wrap starts a dedicated proxy elsewhere.
persistent_routing_mismatch = False
# --no-proxy reuses an already-running proxy, so backend/region/provider
# flags (which only apply when we start one) would be silently dropped.
if no_proxy and (backend or anyllm_provider or region):
@@ -4183,7 +4284,34 @@ def _ensure_proxy_unlocked(
if probe_ready(manifest.health_url):
health_payload = helpers._query_proxy_health(port)
if helpers._proxy_needs_version_restart(health_payload):
running_config = helpers._proxy_health_config(health_payload)
if running_config is None:
running_config = helpers._query_proxy_config(port)
routing_mismatches = (
None
if running_config is None
else helpers._proxy_routing_mismatches(
running_config,
backend=requested_backend,
requested_api_urls=requested_api_urls,
)
)
if running_config is None:
click.echo(
f" Persistent deployment '{manifest.profile}' on port {port} "
"did not expose routing config; starting a dedicated proxy "
"on a different port instead."
)
persistent_routing_mismatch = True
elif routing_mismatches:
keys_str = ", ".join(routing_mismatches)
click.echo(
f" Persistent deployment '{manifest.profile}' on port {port} "
f"has incompatible routing config ({keys_str}); "
"starting a dedicated proxy on a different port instead."
)
persistent_routing_mismatch = True
elif helpers._proxy_needs_version_restart(health_payload):
running_version = helpers._proxy_version(health_payload) or "unknown"
active_sessions = helpers._proxy_active_session_count(health_payload)
other_wrappers = helpers._live_proxy_clients(port, exclude_self=True)
@@ -4208,14 +4336,8 @@ def _ensure_proxy_unlocked(
f"Persistent deployment '{manifest.profile}' on port {port} "
f"is running stale Headroom {running_version} and could not be restarted."
)
# Check if the running proxy has the features we need.
# Without this, a persistent deployment started for one use case
# (e.g. --backend anthropic) would be silently reused for another
# (e.g. --subscription --provider-type openai) causing auth failures.
running_config = helpers._proxy_health_config(health_payload)
if running_config is None:
running_config = helpers._query_proxy_config(port)
if running_config is not None:
else:
# Check if the running proxy has the features we need.
missing = []
if memory and not running_config.get("memory"):
missing.append("memory")
@@ -4223,46 +4345,18 @@ def _ensure_proxy_unlocked(
missing.append("learn")
if code_graph and not running_config.get("code_graph"):
missing.append("code_graph")
if copilot_subscription_seed_requested:
missing.append("copilot-subscription-auth")
if openai_api_url:
running_openai_url = _normalize_proxy_api_url(
running_config.get("openai_api_url")
)
requested_openai_url = _normalize_proxy_api_url(openai_api_url)
if running_openai_url != requested_openai_url:
missing.append("openai-api-url")
if not missing:
click.echo(f" Proxy already running on port {port}")
click.echo(f" Dashboard: http://127.0.0.1:{port}/dashboard")
return None, port
# Features mismatch or config unavailable — fall through to
# the non-persistent path which handles proxy restart.
# Features mismatch or config unavailable — fall through to the
# non-persistent path which handles proxy restart. A routing
# mismatch skips that path entirely (see flag above).
else:
if helpers._recover_persistent_proxy(port):
# If the caller requested feature-sensitive config (e.g.
# openai_api_url for Copilot subscription), continue into
# the shared running-proxy checks below so mismatch-driven
# restart logic can run. For plain recover-only calls,
# preserve the historical fast return.
if not any(
(
memory,
learn,
code_graph,
openai_api_url,
copilot_subscription_seed_requested,
)
):
return None, port
if not helpers._check_proxy(port):
return None, port
# A freshly recovered persistent proxy may not expose
# a full config payload yet. In feature-sensitive flows
# (e.g. Copilot subscription), treat missing or mismatched
# config as restart-required and refresh the persistent
# deployment directly instead of silently reusing it.
health_payload = helpers._query_proxy_health(port)
running_config = helpers._proxy_health_config(health_payload)
if running_config is None:
@@ -4271,61 +4365,93 @@ def _ensure_proxy_unlocked(
if running_config is None:
click.echo(
f" Recovered persistent deployment '{manifest.profile}' "
"did not expose config; restarting with requested features..."
"did not expose routing config; starting a dedicated proxy "
"on a different port instead."
)
if helpers._restart_persistent_proxy(manifest, port):
return None, port
raise click.ClickException(
f"Persistent deployment '{manifest.profile}' on port {port} "
"could not be restarted after recovery."
persistent_routing_mismatch = True
else:
routing_mismatches = helpers._proxy_routing_mismatches(
running_config,
backend=requested_backend,
requested_api_urls=requested_api_urls,
)
if routing_mismatches:
keys_str = ", ".join(routing_mismatches)
click.echo(
f" Recovered persistent deployment '{manifest.profile}' "
f"has incompatible routing config ({keys_str}); "
"starting a dedicated proxy on a different port instead."
)
persistent_routing_mismatch = True
else:
missing = []
if memory and not running_config.get("memory"):
missing.append("memory")
if learn and not running_config.get("learn"):
missing.append("learn")
if code_graph and not running_config.get("code_graph"):
missing.append("code-graph")
missing = []
if memory and not running_config.get("memory"):
missing.append("memory")
if learn and not running_config.get("learn"):
missing.append("learn")
if code_graph and not running_config.get("code_graph"):
missing.append("code-graph")
if copilot_subscription_seed_requested:
missing.append("copilot-subscription-auth")
if openai_api_url:
running_openai_url = _normalize_proxy_api_url(
running_config.get("openai_api_url")
)
requested_openai_url = _normalize_proxy_api_url(openai_api_url)
if running_openai_url != requested_openai_url:
missing.append("openai-api-url")
if missing:
flags_str = ", ".join(f"--{f}" for f in missing)
click.echo(
f" Recovered persistent deployment '{manifest.profile}' is missing: "
f"{flags_str}; restarting..."
)
if helpers._restart_persistent_proxy(manifest, port):
return None, port
raise click.ClickException(
f"Persistent deployment '{manifest.profile}' on port {port} "
"could not be restarted with requested features."
)
return None, port
if not missing:
return None, port
flags_str = ", ".join(f"--{f}" for f in missing)
click.echo(
f" Recovered persistent deployment '{manifest.profile}' "
f"is missing: {flags_str}; restarting..."
)
if helpers._restart_persistent_proxy(manifest, port):
return None, port
raise click.ClickException(
f"Persistent deployment '{manifest.profile}' on port {port} "
"could not be restarted with requested features."
)
elif helpers._check_proxy(port):
raise click.ClickException(
f"Persistent deployment '{manifest.profile}' on port {port} is not healthy."
)
click.echo(
f" Warning: persistent deployment '{manifest.profile}' on port {port} "
"is stale; starting a fresh proxy instead."
)
if not persistent_routing_mismatch:
click.echo(
f" Warning: persistent deployment '{manifest.profile}' on port {port} "
"is stale; starting a fresh proxy instead."
)
if not isolated_copilot_subscription_proxy and helpers._check_proxy(port):
if (
not isolated_copilot_subscription_proxy
and not persistent_routing_mismatch
and helpers._check_proxy(port)
):
# Proxy is running — check if it has the features we need
needs_restart = False
# Set False when the running proxy must not serve this session at
# all (routing-level mismatch with live clients attached): fall
# through to a fresh start on a different port.
reuse_running = True
health_payload = helpers._query_proxy_health(port)
running_config = helpers._proxy_health_config(health_payload)
if running_config is None:
running_config = helpers._query_proxy_config(port)
routing_mismatches = (
None
if running_config is None
else helpers._proxy_routing_mismatches(
running_config,
backend=requested_backend,
requested_api_urls=requested_api_urls,
)
)
if running_config is None and not helpers._proxy_needs_version_restart(health_payload):
# A proxy that exposes no config block cannot be checked for
# feature or routing compatibility, and without a reported PID
# it cannot be restarted either. Reusing it blindly is the
# silent-misrouting incident class; start a dedicated proxy on
# a free port instead (mirrors the persistent branch, which
# already treats config-unavailable as not-reusable).
click.echo(
f" Proxy on port {port} did not expose its config; "
"starting a dedicated proxy on a different port instead of "
"reusing it unchecked."
)
reuse_running = False
if helpers._proxy_needs_version_restart(health_payload):
running_version = helpers._proxy_version(health_payload) or "unknown"
@@ -4345,28 +4471,37 @@ def _ensure_proxy_unlocked(
f" Proxy on port {port} is running Headroom {running_version}; "
f"current CLI is {_HEADROOM_VERSION}."
)
click.echo(
f" Leaving it running because {detail} "
"are still attached; it will be restarted when idle."
)
return None, port
if routing_mismatches or running_config is None:
click.echo(
f" Not reusing it because its routing "
f"{'cannot be verified' if running_config is None else 'is incompatible'}; "
"starting a dedicated proxy on a different port instead."
)
reuse_running = False
else:
click.echo(
f" Leaving it running because {detail} "
"are still attached; it will be restarted when idle."
)
return None, port
click.echo(
f" Proxy on port {port} is running Headroom {running_version}; "
f"restarting with {_HEADROOM_VERSION}..."
)
proxy_pid = running_config.get("pid") if running_config is not None else None
if proxy_pid is None:
raise click.ClickException(
f"Proxy on port {port} is stale but did not expose a PID. "
"Stop it manually and retry."
else:
click.echo(
f" Proxy on port {port} is running Headroom {running_version}; "
f"restarting with {_HEADROOM_VERSION}..."
)
if not helpers._kill_proxy_by_pid(int(proxy_pid), port):
raise click.ClickException(
f"Failed to stop stale proxy (PID {proxy_pid}) on port {port}. "
"Stop it manually and retry."
)
needs_restart = True
proxy_pid = running_config.get("pid") if running_config is not None else None
if proxy_pid is None:
raise click.ClickException(
f"Proxy on port {port} is stale but did not expose a PID. "
"Stop it manually and retry."
)
if not helpers._kill_proxy_by_pid(int(proxy_pid), port):
raise click.ClickException(
f"Failed to stop stale proxy (PID {proxy_pid}) on port {port}. "
"Stop it manually and retry."
)
needs_restart = True
if running_config is not None:
missing = []
@@ -4376,30 +4511,14 @@ def _ensure_proxy_unlocked(
missing.append("learn")
if code_graph and not running_config.get("code_graph"):
missing.append("code_graph")
if copilot_subscription_seed_requested:
missing.append("copilot-subscription-auth")
expected_savings_profile = helpers._wrap_agent_savings_profile(agent_type)
if (
expected_savings_profile is not None
and running_config.get("savings_profile") != expected_savings_profile
):
missing.append("savings-profile")
if openai_api_url:
running_openai_url = _normalize_proxy_api_url(
running_config.get("openai_api_url")
)
requested_openai_url = _normalize_proxy_api_url(openai_api_url)
if running_openai_url != requested_openai_url:
missing.append("openai-api-url")
if vertex_api_url or clear_vertex_api_url:
running_vertex_url = _normalize_proxy_api_url(
running_config.get("vertex_api_url")
)
requested_vertex_url = _normalize_proxy_api_url(vertex_api_url)
if running_vertex_url != requested_vertex_url:
missing.append("vertex-api-url")
if missing:
if missing and not routing_mismatches:
flags_str = ", ".join(
f if f.startswith("--") else f"--{f.replace('_', '-')}" for f in missing
)
@@ -4444,7 +4563,47 @@ def _ensure_proxy_unlocked(
)
return None, port
if not needs_restart:
# Routing-level config (backend, upstream URLs) must match in
# both directions. Unlike missing feature flags, reusing a
# proxy that forwards to a backend/upstream the caller did not
# request is broken, not degraded, so "reuse as-is" is never an
# option here (cf. the factory-api-url fail-loud precedent on
# the droid branch and the tenant check in wrap auggie).
routing_mismatches = [] if needs_restart else (routing_mismatches or [])
if routing_mismatches:
keys_str = ", ".join(routing_mismatches)
other_wrappers = helpers._live_proxy_clients(port, exclude_self=True)
active_sessions = helpers._proxy_active_session_count(health_payload)
if other_wrappers or active_sessions > 0:
click.echo(
f" Proxy on port {port} has incompatible routing config "
f"({keys_str}), but live session(s) are attached."
)
click.echo(
" Starting a dedicated proxy on a different port instead "
"of reusing or disrupting it."
)
reuse_running = False
else:
click.echo(
f" Proxy on port {port} has incompatible routing config "
f"({keys_str}); restarting with the requested config..."
)
proxy_pid = running_config.get("pid")
if proxy_pid is None:
raise click.ClickException(
f"Proxy on port {port} has incompatible routing config "
f"({keys_str}) but did not expose a PID. "
"Stop it manually and retry."
)
if not helpers._kill_proxy_by_pid(int(proxy_pid), port):
raise click.ClickException(
f"Failed to stop incompatible proxy (PID {proxy_pid}) "
f"on port {port}. Stop it manually and retry."
)
needs_restart = True
if not needs_restart and reuse_running:
click.echo(f" Proxy already running on port {port}")
click.echo(f" Dashboard: http://127.0.0.1:{port}/dashboard")
return None, port
+6 -2
View File
@@ -847,12 +847,16 @@ def test_ensure_proxy_already_running_prints_dashboard_url(
wraps (the common case) never told the user where the dashboard lives.
"""
port = 1234
# A compatible running proxy exposes its config; proxies without a config
# block are no longer reused (they cannot be compatibility-checked) and
# are covered by test_ensure_proxy_never_reuses_configless_proxy.
running_config = {"pid": 4321, "backend": "anthropic"}
monkeypatch.setattr(wrap_mod, "_find_persistent_manifest", lambda _p: None)
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda _p: True)
monkeypatch.setattr(wrap_mod, "_query_proxy_health", lambda _p: {})
monkeypatch.setattr(wrap_mod, "_proxy_needs_version_restart", lambda _h: False)
monkeypatch.setattr(wrap_mod, "_proxy_health_config", lambda _h: None)
monkeypatch.setattr(wrap_mod, "_query_proxy_config", lambda _p: None)
monkeypatch.setattr(wrap_mod, "_proxy_health_config", lambda _h: running_config)
monkeypatch.setattr(wrap_mod, "_live_proxy_clients", lambda *a, **kw: [])
output = _run_in_click_context(lambda: wrap_mod._ensure_proxy(port, no_proxy=False))
+809 -28
View File
@@ -17,6 +17,17 @@ def _no_attached_wrappers(monkeypatch: pytest.MonkeyPatch) -> None:
flaky. Individual tests override this to simulate attached wrappers.
"""
monkeypatch.setattr(wrap_cli, "_live_proxy_clients", lambda *a, **kw: [])
for name in (
"ANTHROPIC_TARGET_API_URL",
"OPENAI_TARGET_API_URL",
"GEMINI_TARGET_API_URL",
"CLOUDCODE_TARGET_API_URL",
"VERTEX_TARGET_API_URL",
"AUGMENT_TARGET_API_URL",
"FACTORY_TARGET_API_URL",
"HEADROOM_BACKEND",
):
monkeypatch.delenv(name, raising=False)
class _Manifest:
@@ -67,6 +78,18 @@ def test_ensure_proxy_recovers_persistent_deployment_when_socket_is_bound(monkey
monkeypatch.setattr(
"headroom.install.runtime.wait_ready", lambda manifest, timeout_seconds=45: True
)
monkeypatch.setattr(
wrap_cli,
"_query_proxy_health",
lambda port: {"version": wrap_cli._HEADROOM_VERSION, "config": {"backend": "anthropic"}},
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("matching recovered proxy should be reused")
),
)
proc, actual_port = wrap_cli._ensure_proxy(8787, False)
@@ -374,6 +397,694 @@ def test_ensure_proxy_restarts_ephemeral_proxy_for_openai_api_url_mismatch(monke
assert calls[1][2]["openai_api_url"] == "https://api.individual.githubcopilot.com"
def test_proxy_routing_mismatches_detects_backend_and_urls(monkeypatch) -> None:
monkeypatch.delenv("HEADROOM_BACKEND", raising=False)
check = wrap_cli._proxy_routing_mismatches
# Stock proxy vs stock request: compatible.
assert check({"backend": "anthropic"}, backend=None) == []
# Running non-default backend vs default request: mismatch.
assert check({"backend": "anyllm"}, backend=None) == ["backend"]
# Requested backend vs different running backend: mismatch.
assert check({"backend": "anthropic"}, backend="anyllm") == ["backend"]
# HEADROOM_BACKEND env feeds the effective requested backend.
monkeypatch.setenv("HEADROOM_BACKEND", "anyllm")
assert check({"backend": "anyllm"}, backend=None) == []
monkeypatch.delenv("HEADROOM_BACKEND")
# Running proxy carries an upstream the caller never requested (the
# 2026-08-22 incident: a wrap-auggie proxy reused by wrap codex).
assert check(
{"backend": "anthropic", "augment_api_url": "https://xlb.api.augmentcode.com"},
backend=None,
) == ["augment_api_url"]
# Requested upstream the running proxy does not have: mismatch.
assert check(
{"backend": "anthropic"},
backend=None,
requested_api_urls={"openai_api_url": "https://api.githubcopilot.com"},
) == ["openai_api_url"]
# Same upstream on both sides, modulo normalization noise: compatible.
assert (
check(
{"backend": "anthropic", "openai_api_url": "https://api.githubcopilot.com/v1/"},
backend=None,
requested_api_urls={"openai_api_url": "https://api.githubcopilot.com"},
)
== []
)
# Captive provider paths are semantic; /v1 is not a removable transport
# convention for their REST catch-all routing.
assert check(
{"backend": "anthropic", "factory_api_url": "https://factory.example/v1"},
backend=None,
requested_api_urls={"factory_api_url": "https://factory.example"},
) == ["factory_api_url"]
# Missing backend key (older proxy): treated as unknown, not a mismatch.
assert check({"openai_api_url": None}, backend=None) == []
def test_effective_requested_proxy_routing_uses_startup_precedence(monkeypatch) -> None:
monkeypatch.setenv("HEADROOM_BACKEND", "anyllm")
monkeypatch.setenv("OPENAI_TARGET_API_URL", "https://env.openai.test")
monkeypatch.setenv("VERTEX_TARGET_API_URL", "https://env.vertex.test")
monkeypatch.setenv("AUGMENT_TARGET_API_URL", "https://env.augment.test/v1")
monkeypatch.setenv("FACTORY_TARGET_API_URL", "https://env.factory.test/v1")
backend, api_urls = wrap_cli._effective_requested_proxy_routing(
backend=None,
openai_api_url="https://explicit.openai.test",
anthropic_api_url=None,
vertex_api_url=None,
clear_vertex_api_url=True,
)
assert backend == "anyllm"
assert api_urls["openai_api_url"] == "https://explicit.openai.test"
assert api_urls["vertex_api_url"] is None
assert api_urls["augment_api_url"] == "https://env.augment.test/v1"
assert api_urls["factory_api_url"] == "https://env.factory.test/v1"
def test_ensure_proxy_restarts_idle_proxy_for_routing_mismatch(monkeypatch) -> None:
"""A clientless proxy with captive routing (augment upstream) is restarted."""
calls: list[object] = []
health = {
"version": wrap_cli._HEADROOM_VERSION,
"runtime": {"websocket_sessions": {"active_sessions": 0, "active_relay_tasks": 0}},
"config": {
"pid": "12345",
"memory": False,
"learn": False,
"code_graph": False,
"backend": "anthropic",
"augment_api_url": "https://xlb.api.augmentcode.com",
},
}
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: None)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: len(calls) == 0)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(wrap_cli, "_live_proxy_clients", lambda *a, **kw: [])
monkeypatch.setattr(wrap_cli, "_port_bind_error", lambda port: None)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda pid, port: calls.append(("kill", pid, port)) or True,
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: calls.append(("start", args, kwargs)),
)
proc, actual_port = wrap_cli._ensure_proxy(8787, False)
assert proc is None
assert actual_port == 8787
assert calls[0] == ("kill", 12345, 8787)
assert calls[1][0] == "start"
def _routing_mismatch_health(config_overrides: dict) -> dict:
"""Shared fixture: running proxy with captive augment routing (mismatch)."""
config = {
"pid": "12345",
"memory": False,
"learn": False,
"code_graph": False,
"backend": "anthropic",
"augment_api_url": "https://xlb.api.augmentcode.com",
}
config.update(config_overrides)
return {
"version": wrap_cli._HEADROOM_VERSION,
"runtime": {"websocket_sessions": {"active_sessions": 0, "active_relay_tasks": 0}},
"config": config,
}
def test_ensure_proxy_routing_mismatch_without_pid_raises(monkeypatch) -> None:
"""Incompatible proxy that exposes no PID cannot be restarted: fail loud."""
health = _routing_mismatch_health({})
del health["config"]["pid"]
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: None)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(wrap_cli, "_port_bind_error", lambda port: None)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *a, **kw: (_ for _ in ()).throw(AssertionError("must not start")),
)
with pytest.raises(click.ClickException, match="did not expose a PID"):
wrap_cli._ensure_proxy(8787, False)
def test_ensure_proxy_routing_mismatch_kill_failure_raises(monkeypatch) -> None:
"""When the incompatible proxy survives the kill attempt: fail loud."""
health = _routing_mismatch_health({})
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: None)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(wrap_cli, "_port_bind_error", lambda port: None)
monkeypatch.setattr(wrap_cli, "_kill_proxy_by_pid", lambda pid, port: False)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *a, **kw: (_ for _ in ()).throw(AssertionError("must not start")),
)
with pytest.raises(click.ClickException, match="Failed to stop incompatible proxy"):
wrap_cli._ensure_proxy(8787, False)
def test_ensure_proxy_bumps_port_for_routing_mismatch_with_attached_clients(
monkeypatch,
) -> None:
"""With live clients attached, an incompatible proxy is neither reused
nor restarted: the wrap starts a dedicated proxy on a free port."""
calls: list[object] = []
health = {
"version": wrap_cli._HEADROOM_VERSION,
"runtime": {"websocket_sessions": {"active_sessions": 0, "active_relay_tasks": 0}},
"config": {
"pid": "12345",
"backend": "anthropic",
"augment_api_url": "https://xlb.api.augmentcode.com",
},
}
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: None)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: port == 8787)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(wrap_cli, "_live_proxy_clients", lambda *a, **kw: [999])
monkeypatch.setattr(
wrap_cli,
"_find_available_port",
lambda start_port, **kw: calls.append(("find_port", start_port)) or 8799,
)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("incompatible proxy with live clients must not be killed")
),
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: calls.append(("start", args, kwargs)),
)
proc, actual_port = wrap_cli._ensure_proxy(8787, False)
assert proc is None
assert actual_port == 8799
assert calls[0] == ("find_port", 8787)
assert calls[1][0] == "start"
assert calls[1][1][0] == 8799
def test_ensure_proxy_reuses_proxy_with_matching_routing_config(monkeypatch) -> None:
"""Happy path: identical routing config reuses the running proxy."""
health = {
"version": wrap_cli._HEADROOM_VERSION,
"runtime": {"websocket_sessions": {"active_sessions": 0, "active_relay_tasks": 0}},
"config": {
"pid": "12345",
"memory": False,
"learn": False,
"code_graph": False,
"backend": "anthropic",
"openai_api_url": None,
},
}
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: None)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("must not kill")),
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: (_ for _ in ()).throw(AssertionError("must not start")),
)
proc, actual_port = wrap_cli._ensure_proxy(8787, False)
assert proc is None
assert actual_port == 8787
def test_ensure_proxy_restarts_idle_proxy_for_backend_mismatch(monkeypatch) -> None:
calls: list[object] = []
health = {
"version": wrap_cli._HEADROOM_VERSION,
"runtime": {"websocket_sessions": {"active_sessions": 0, "active_relay_tasks": 0}},
"config": {"pid": "12345", "backend": "anyllm"},
}
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: None)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: len(calls) == 0)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(wrap_cli, "_live_proxy_clients", lambda *a, **kw: [])
monkeypatch.setattr(wrap_cli, "_port_bind_error", lambda port: None)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda pid, port: calls.append(("kill", pid, port)) or True,
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: calls.append(("start", args, kwargs)),
)
proc, actual_port = wrap_cli._ensure_proxy(8787, False)
assert proc is None
assert actual_port == 8787
assert calls[0] == ("kill", 12345, 8787)
assert calls[1][0] == "start"
def test_ensure_proxy_never_reuses_configless_proxy(monkeypatch) -> None:
"""A proxy exposing no config block cannot be compatibility-checked; it
must not be silently reused (the persistent branch already treats
config-unavailable as fall-through). No PID is known, so the wrap starts
a dedicated proxy on a free port instead of killing blindly."""
calls: list[object] = []
health = {
"version": wrap_cli._HEADROOM_VERSION,
"runtime": {"websocket_sessions": {"active_sessions": 0, "active_relay_tasks": 0}},
}
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: None)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: port == 8787)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(wrap_cli, "_query_proxy_config", lambda port: None)
monkeypatch.setattr(
wrap_cli,
"_find_available_port",
lambda start_port, **kw: calls.append(("find_port", start_port)) or 8799,
)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("configless proxy exposes no PID; must not kill")
),
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: calls.append(("start", args, kwargs)),
)
proc, actual_port = wrap_cli._ensure_proxy(8787, False)
assert proc is None
assert actual_port == 8799
assert calls[0] == ("find_port", 8787)
assert calls[1][0] == "start"
def test_ensure_proxy_routing_mismatch_on_persistent_deployment_uses_dedicated_port(
monkeypatch,
) -> None:
"""A persistent deployment's routing comes from its manifest; a wrap with
different routing must not reuse it AND must not let the ephemeral path
kill it (the manifest would point at a hijacked port). The wrap starts a
dedicated proxy on a free port and the deployment keeps running."""
calls: list[object] = []
health = {
"version": wrap_cli._HEADROOM_VERSION,
"runtime": {"websocket_sessions": {"active_sessions": 0, "active_relay_tasks": 0}},
"config": {
"pid": "12345",
"backend": "anthropic",
"augment_api_url": "https://xlb.api.augmentcode.com",
},
}
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: _Manifest())
monkeypatch.setattr("headroom.install.health.probe_ready", lambda url: True)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: port == 8787)
monkeypatch.setattr(
wrap_cli,
"_find_available_port",
lambda start_port, **kw: calls.append(("find_port", start_port)) or 8799,
)
monkeypatch.setattr(
wrap_cli,
"_restart_persistent_proxy",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("routing mismatch must not restart the deployment")
),
)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("routing mismatch must not kill the deployment's proxy")
),
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: calls.append(("start", args, kwargs)),
)
proc, actual_port = wrap_cli._ensure_proxy(8787, False)
assert proc is None
assert actual_port == 8799
assert calls[0] == ("find_port", 8787)
assert calls[1][0] == "start"
assert calls[1][1][0] == 8799
def test_ensure_proxy_healthy_persistent_uses_config_fallback(monkeypatch) -> None:
config = {"pid": 12345, "backend": "anthropic"}
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: _Manifest())
monkeypatch.setattr("headroom.install.health.probe_ready", lambda url: True)
monkeypatch.setattr(
wrap_cli,
"_query_proxy_health",
lambda port: {"version": wrap_cli._HEADROOM_VERSION},
)
monkeypatch.setattr(wrap_cli, "_query_proxy_config", lambda port: config)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("matching persistent proxy should be reused")
),
)
assert wrap_cli._ensure_proxy(8787, False) == (None, 8787)
def test_ensure_proxy_healthy_configless_persistent_uses_dedicated_port(monkeypatch) -> None:
calls: list[object] = []
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: _Manifest())
monkeypatch.setattr("headroom.install.health.probe_ready", lambda url: True)
monkeypatch.setattr(
wrap_cli,
"_query_proxy_health",
lambda port: {"version": wrap_cli._HEADROOM_VERSION},
)
monkeypatch.setattr(wrap_cli, "_query_proxy_config", lambda port: None)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: port == 8787)
monkeypatch.setattr(
wrap_cli,
"_find_available_port",
lambda start_port, **kwargs: calls.append(("find_port", start_port)) or 8799,
)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("configless persistent proxy must not be killed")
),
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: calls.append(("start", args, kwargs)),
)
assert wrap_cli._ensure_proxy(8787, False) == (None, 8799)
assert calls[0] == ("find_port", 8787)
assert calls[1][0] == "start"
@pytest.mark.parametrize(
("env_name", "config_key"),
[
("ANTHROPIC_TARGET_API_URL", "anthropic_api_url"),
("OPENAI_TARGET_API_URL", "openai_api_url"),
("GEMINI_TARGET_API_URL", "gemini_api_url"),
("CLOUDCODE_TARGET_API_URL", "cloudcode_api_url"),
("VERTEX_TARGET_API_URL", "vertex_api_url"),
("AUGMENT_TARGET_API_URL", "augment_api_url"),
("FACTORY_TARGET_API_URL", "factory_api_url"),
],
)
def test_ensure_proxy_reuses_matching_inherited_routing_target(
monkeypatch,
env_name: str,
config_key: str,
) -> None:
target = "https://tenant.example/api"
for name in (
"ANTHROPIC_TARGET_API_URL",
"OPENAI_TARGET_API_URL",
"GEMINI_TARGET_API_URL",
"CLOUDCODE_TARGET_API_URL",
"VERTEX_TARGET_API_URL",
"AUGMENT_TARGET_API_URL",
"FACTORY_TARGET_API_URL",
):
monkeypatch.delenv(name, raising=False)
monkeypatch.setenv(env_name, target)
health = {
"version": wrap_cli._HEADROOM_VERSION,
"runtime": {"websocket_sessions": {"active_sessions": 0, "active_relay_tasks": 0}},
"config": {"pid": 12345, "backend": "anthropic", config_key: target},
}
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: None)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(
wrap_cli,
"_find_available_port",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("matching inherited routing must reuse the proxy")
),
)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("matching inherited routing must not kill the proxy")
),
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("matching inherited routing must not start another proxy")
),
)
assert wrap_cli._ensure_proxy(8787, False) == (None, 8787)
@pytest.mark.parametrize(
("ensure_kwargs", "config_overrides", "env_override"),
[
(
{"anthropic_api_url": "https://expected.anthropic.test"},
{"anthropic_api_url": "https://running.anthropic.test"},
None,
),
(
{"vertex_api_url": "https://expected.vertex.test"},
{"vertex_api_url": "https://running.vertex.test"},
None,
),
(
{"backend": "anyllm"},
{"backend": "anthropic"},
None,
),
(
{},
{"factory_api_url": "https://running.factory.test"},
("FACTORY_TARGET_API_URL", "https://expected.factory.test"),
),
],
)
def test_ensure_proxy_recovered_routing_mismatch_uses_dedicated_port(
monkeypatch,
ensure_kwargs: dict[str, object],
config_overrides: dict[str, object],
env_override: tuple[str, str] | None,
) -> None:
calls: list[object] = []
if env_override is not None:
monkeypatch.setenv(*env_override)
config = {"pid": 12345, "backend": "anthropic"}
config.update(config_overrides)
health = {
"version": wrap_cli._HEADROOM_VERSION,
"runtime": {"websocket_sessions": {"active_sessions": 0, "active_relay_tasks": 0}},
"config": config,
}
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: _Manifest())
monkeypatch.setattr("headroom.install.health.probe_ready", lambda url: False)
monkeypatch.setattr(wrap_cli, "_recover_persistent_proxy", lambda port: True)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: port == 8787)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(
wrap_cli,
"_find_available_port",
lambda start_port, **kwargs: calls.append(("find_port", start_port)) or 8799,
)
monkeypatch.setattr(
wrap_cli,
"_restart_persistent_proxy",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("routing mismatch must not restart the persistent proxy")
),
)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("routing mismatch must not kill the persistent proxy")
),
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: calls.append(("start", args, kwargs)),
)
assert wrap_cli._ensure_proxy(8787, False, **ensure_kwargs) == (None, 8799)
assert calls[0] == ("find_port", 8787)
assert calls[1][0] == "start"
def test_ensure_proxy_stale_attached_persistent_routing_mismatch_uses_dedicated_port(
monkeypatch,
) -> None:
calls: list[object] = []
health = _routing_mismatch_health({})
health["version"] = "0.0.1"
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: _Manifest())
monkeypatch.setattr("headroom.install.health.probe_ready", lambda url: True)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(wrap_cli, "_live_proxy_clients", lambda *args, **kwargs: [999])
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: port == 8787)
monkeypatch.setattr(
wrap_cli,
"_find_available_port",
lambda start_port, **kwargs: calls.append(("find_port", start_port)) or 8799,
)
monkeypatch.setattr(
wrap_cli,
"_restart_persistent_proxy",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("active persistent proxy must not restart")
),
)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("active persistent proxy must not be killed")
),
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: calls.append(("start", args, kwargs)),
)
assert wrap_cli._ensure_proxy(8787, False, memory=True) == (None, 8799)
assert calls[0] == ("find_port", 8787)
assert calls[1][0] == "start"
def test_ensure_proxy_stale_attached_ephemeral_routing_mismatch_uses_dedicated_port(
monkeypatch,
) -> None:
calls: list[object] = []
health = _routing_mismatch_health({})
health["version"] = "0.0.1"
health["runtime"]["websocket_sessions"]["active_sessions"] = 1
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: None)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(wrap_cli, "_live_proxy_clients", lambda *args, **kwargs: [])
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: port == 8787)
monkeypatch.setattr(
wrap_cli,
"_find_available_port",
lambda start_port, **kwargs: calls.append(("find_port", start_port)) or 8799,
)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("active proxy must not be killed")
),
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: calls.append(("start", args, kwargs)),
)
assert wrap_cli._ensure_proxy(8787, False, memory=True) == (None, 8799)
assert calls[0] == ("find_port", 8787)
assert calls[1][0] == "start"
@pytest.mark.parametrize(
("pid", "kill_result", "message"),
[
(None, True, "did not expose a PID"),
(12345, False, "Failed to stop stale proxy"),
],
)
def test_ensure_proxy_stale_idle_restart_failures_are_actionable(
monkeypatch,
pid: int | None,
kill_result: bool,
message: str,
) -> None:
config: dict[str, object] = {"backend": "anthropic"}
if pid is not None:
config["pid"] = pid
health = {
"version": "0.0.1",
"runtime": {"websocket_sessions": {"active_sessions": 0, "active_relay_tasks": 0}},
"config": config,
}
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: None)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(wrap_cli, "_kill_proxy_by_pid", lambda found_pid, port: kill_result)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("failed stale restart must not start another proxy")
),
)
with pytest.raises(click.ClickException, match=message):
wrap_cli._ensure_proxy(8787, False)
def test_ensure_proxy_starts_isolated_ephemeral_proxy_for_copilot_subscription_seed(
monkeypatch,
) -> None:
@@ -741,8 +1452,14 @@ def test_ensure_proxy_restarts_for_flags_when_no_other_wrapper(monkeypatch) -> N
assert calls[1][0] == "start"
def test_ensure_proxy_restarts_persistent_deployment_for_feature_mismatch(monkeypatch) -> None:
"""Persistent deployment should restart when requested features differ from running config."""
def test_ensure_proxy_diverts_persistent_deployment_for_routing_mismatch(
monkeypatch,
) -> None:
"""A routing-URL mismatch against a persistent deployment starts a
dedicated proxy on a free port. Killing the deployment's proxy and
starting an ephemeral one with the requested routing (the old behavior)
orphaned the manifest: it still pointed at the port while the routing it
declares no longer matched what served it."""
calls: list[object] = []
health = {
"version": wrap_cli._HEADROOM_VERSION,
@@ -761,11 +1478,17 @@ def test_ensure_proxy_restarts_persistent_deployment_for_feature_mismatch(monkey
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
# Persistent proxy is running, so _check_proxy returns True
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
monkeypatch.setattr(wrap_cli, "_port_bind_error", lambda port: None)
monkeypatch.setattr(
wrap_cli,
"_find_available_port",
lambda start_port, **kw: calls.append(("find_port", start_port)) or 8799,
)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda pid, port: calls.append(("kill", pid, port)) or True,
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("routing mismatch must not kill the deployment's proxy")
),
)
monkeypatch.setattr(
wrap_cli,
@@ -781,10 +1504,12 @@ def test_ensure_proxy_restarts_persistent_deployment_for_feature_mismatch(monkey
)
assert proc is None
assert actual_port == 8787
# Proxy should be killed and restarted due to openai_api_url mismatch
assert calls[0] == ("kill", 12345, 8787)
assert actual_port == 8799
# Deployment left running; dedicated proxy started on the free port with
# the requested routing.
assert calls[0] == ("find_port", 8787)
assert calls[1][0] == "start"
assert calls[1][1][0] == 8799
assert calls[1][2]["openai_api_url"] == "https://api.githubcopilot.com"
@@ -852,17 +1577,29 @@ def test_ensure_proxy_restarts_recovered_persistent_for_openai_api_url_mismatch(
monkeypatch.setattr(wrap_cli, "_recover_persistent_proxy", lambda port: True)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(
wrap_cli,
"_find_available_port",
lambda start_port, **kwargs: calls.append(("find_port", start_port)) or 8799,
)
monkeypatch.setattr(
wrap_cli,
"_restart_persistent_proxy",
lambda manifest, port: calls.append(("restart", manifest.profile, port)) or True,
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("routing mismatch must not restart the persistent proxy")
),
)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("routing mismatch must not kill the persistent proxy")
),
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("ephemeral proxy should not start")
),
lambda *args, **kwargs: calls.append(("start", args, kwargs)),
)
proc, actual_port = wrap_cli._ensure_proxy(
@@ -872,8 +1609,9 @@ def test_ensure_proxy_restarts_recovered_persistent_for_openai_api_url_mismatch(
)
assert proc is None
assert actual_port == 8787
assert calls == [("restart", "default", 8787)]
assert actual_port == 8799
assert calls[0] == ("find_port", 8787)
assert calls[1][0] == "start"
def test_ensure_proxy_restarts_recovered_persistent_when_config_unavailable(monkeypatch) -> None:
@@ -885,17 +1623,29 @@ def test_ensure_proxy_restarts_recovered_persistent_when_config_unavailable(monk
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: {"version": "x"})
monkeypatch.setattr(wrap_cli, "_query_proxy_config", lambda port: None)
monkeypatch.setattr(
wrap_cli,
"_find_available_port",
lambda start_port, **kwargs: calls.append(("find_port", start_port)) or 8799,
)
monkeypatch.setattr(
wrap_cli,
"_restart_persistent_proxy",
lambda manifest, port: calls.append(("restart", manifest.profile, port)) or True,
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("configless recovered proxy must not restart")
),
)
monkeypatch.setattr(
wrap_cli,
"_kill_proxy_by_pid",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("configless recovered proxy must not be killed")
),
)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("ephemeral proxy should not start")
),
lambda *args, **kwargs: calls.append(("start", args, kwargs)),
)
proc, actual_port = wrap_cli._ensure_proxy(
@@ -905,8 +1655,9 @@ def test_ensure_proxy_restarts_recovered_persistent_when_config_unavailable(monk
)
assert proc is None
assert actual_port == 8787
assert calls == [("restart", "default", 8787)]
assert actual_port == 8799
assert calls[0] == ("find_port", 8787)
assert calls[1][0] == "start"
def test_ensure_proxy_reuses_persistent_deployment_when_features_match(monkeypatch) -> None:
@@ -953,12 +1704,22 @@ def test_ensure_proxy_reuses_persistent_deployment_when_features_match(monkeypat
assert actual_port == 8787
def test_ensure_proxy_recovered_persistent_deployment_checks_feature_mismatch(monkeypatch) -> None:
@pytest.mark.parametrize(
"ensure_kwargs",
[
{"memory": True},
{"learn": True},
{"code_graph": True},
],
)
def test_ensure_proxy_recovered_persistent_deployment_checks_feature_mismatch(
monkeypatch,
ensure_kwargs: dict[str, bool],
) -> None:
"""Recovered persistent deployments must still restart on feature mismatch.
Regression guard for the recover path: when wrap requests a different
openai_api_url (Copilot subscription), do not early-return right after
recover; run the shared mismatch checks and restart if needed.
Regression guard for the recover path: when wrap requests memory, do not
early-return right after recover; restart the deployment if needed.
"""
calls: list[object] = []
@@ -992,12 +1753,32 @@ def test_ensure_proxy_recovered_persistent_deployment_checks_feature_mismatch(mo
),
)
proc, actual_port = wrap_cli._ensure_proxy(
8787,
False,
openai_api_url="https://api.githubcopilot.com",
)
proc, actual_port = wrap_cli._ensure_proxy(8787, False, **ensure_kwargs)
assert proc is None
assert actual_port == 8787
assert calls == [("restart", "default", 8787)]
def test_ensure_proxy_recovered_feature_restart_failure_raises(monkeypatch) -> None:
health = {
"version": wrap_cli._HEADROOM_VERSION,
"config": {"pid": 12345, "backend": "anthropic", "memory": False},
}
monkeypatch.setattr(wrap_cli, "_find_persistent_manifest", lambda port: _Manifest())
monkeypatch.setattr("headroom.install.health.probe_ready", lambda url: False)
monkeypatch.setattr(wrap_cli, "_recover_persistent_proxy", lambda port: True)
monkeypatch.setattr(wrap_cli, "_check_proxy", lambda port: True)
monkeypatch.setattr(wrap_cli, "_query_proxy_health", lambda port: health)
monkeypatch.setattr(wrap_cli, "_restart_persistent_proxy", lambda manifest, port: False)
monkeypatch.setattr(
wrap_cli,
"_start_proxy",
lambda *args, **kwargs: (_ for _ in ()).throw(
AssertionError("failed persistent restart must not start an ephemeral proxy")
),
)
with pytest.raises(click.ClickException, match="could not be restarted"):
wrap_cli._ensure_proxy(8787, False, memory=True)