Files
ComfyUI/tests/execution/test_force_output_rerun.py
Christian Byrne c226dc23a3 Add --force-output-rerun so output nodes write their files again on a cache hit (#17809)
## Problem

A server that keeps one `PromptExecutor` across jobs and deletes output
files after each job (Comfy Cloud uploads, then removes them) answers a
repeated prompt from the cache: `SaveImage` is not run, the cached
`executed` message replays the first run's filename, that file no longer
exists, and the job finishes as a success with no output. Cloud's own
telemetry counted 2,368 such zero-output successes for 273 users in prod
over 7 days at the start of November (not all of them are this cause).

## Change

New opt-in flag `--force-output-rerun` (default off, so nothing changes
for local users):

- On a cache hit, a node with `OUTPUT_NODE = True` is executed again, so
it writes its files again. Its inputs still come from the cache, so
models, latents and decoded images are not recomputed: the cost is one
encode and write per output node.
- API nodes (`API_NODE = True`) are excluded. Some are output nodes, and
re-running one repeats a paid external call.
- Forced nodes are left out of the `execution_cached` message, so
clients are not told a node was cached when it ran.

## Tests

`tests/execution/test_force_output_rerun.py` starts a real server:

- With the flag: run a prompt, delete its saved file, run the identical
prompt again. The save node writes a new file that exists, its upstream
`EmptyImage` stays cached, and a test output node marked `API_NODE`
stays cached with its first output.
- Control, without the flag: the second run replays the first filename
and every node is cached.

Red first: on the commit with only the flag, the flagged test failed
because the second run replayed the first filename
(`force_rerun_00001_.png`), while the no-flag control passed:
https://github.com/Comfy-Org/ComfyUI/actions/runs/44228691563/job/134548112592.
A follow-up commit corrects one assertion (the counter can reuse a name
after the file is deleted, so the test checks the file exists instead),
and the last commit wires in the behaviour.

Tracking: https://linear.app/comfyorg/issue/BE-4802 (cloud enables the
flag after this ships: https://linear.app/comfyorg/issue/BE-4803).

## Review notes

An independent review raised three points:

- **Upstream evicted from the RAM cache.** If memory pressure evicted an
upstream result but kept the output node's entry, the upstream nodes run
again. That already happens without this flag: `ExecutionList.add_node`
schedules every uncached input of an output node whether or not the
output node itself is cached, so the flag adds only the output node's
own write. No change made.
- **Output nodes created by an expanding node that is itself cached**
are not re-run, because the cached parent does not expand again. This
flag covers output nodes in the submitted prompt, which is how the
frontend sends subgraphs (flattened). Known limit, noted here rather
than widened in this PR.
- **The test only used `--cache-classic`.** Fixed: the flagged test now
also runs under the default RAM-pressure cache.

The Cursor review panel then found that a node reading a forced output
node's result could run before the re-run and read the old value. Fixed:
`ExecutionList` takes a `force_rerun` predicate and treats those nodes
as uncached while planning, with a test
(`test_a_node_reading_a_forced_output_node_gets_the_new_value`). The
flags are now read by truthiness, and `execution_cached` leaves out only
forced nodes this run will execute.
2026-11-17 20:27:40 -08:00

187 lines
6.6 KiB
Python

"""--force-output-rerun: output nodes re-execute on a cache hit, upstream nodes stay cached.
A long-lived server that deletes output files after each job (a multi-tenant
deployment) otherwise answers a repeated prompt by replaying the first run's
filename for a file that no longer exists.
"""
from contextlib import contextmanager
import json
from pathlib import Path
import socket
import subprocess
import sys
import time
from typing import Iterator
import urllib.error
import urllib.request
import pytest
COMFYUI_ROOT = Path(__file__).parents[2]
def _unused_port() -> int:
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
return sock.getsockname()[1]
def _get_json(url: str) -> dict:
with urllib.request.urlopen(url, timeout=5) as response:
return json.load(response)
def _queue(base_url: str, prompt: dict) -> str:
request = urllib.request.Request(
f"{base_url}/prompt",
data=json.dumps({"prompt": prompt}).encode(),
headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(request, timeout=15) as response:
return json.load(response)["prompt_id"]
def _wait_for_history(base_url: str, prompt_id: str) -> dict:
deadline = time.monotonic() + 60
while time.monotonic() < deadline:
history = _get_json(f"{base_url}/history/{prompt_id}")
if prompt_id in history:
return history[prompt_id]
time.sleep(0.1)
raise AssertionError(f"prompt {prompt_id} did not finish")
def _run(base_url: str, prompt: dict) -> dict:
history = _wait_for_history(base_url, _queue(base_url, prompt))
assert history["status"]["status_str"] == "success", history["status"]
return history
def _cached_nodes(history: dict) -> set:
for event, data in history["status"]["messages"]:
if event == "execution_cached":
return set(data["nodes"])
raise AssertionError("no execution_cached message in history")
@contextmanager
def _running_server(tmp_path: Path, extra_args: list) -> Iterator[str]:
log_path = tmp_path / "server.log"
port = _unused_port()
base_url = f"http://127.0.0.1:{port}"
command = [
sys.executable,
"main.py",
"--listen",
"127.0.0.1",
"--port",
str(port),
"--cpu",
"--disable-assets",
"--output-directory",
str(tmp_path / "output"),
"--temp-directory",
str(tmp_path),
"--extra-model-paths-config",
str(Path(__file__).parent / "extra_model_paths.yaml"),
*extra_args,
]
with log_path.open("w", encoding="utf-8") as log:
process = subprocess.Popen(command, cwd=COMFYUI_ROOT, stdout=log, stderr=subprocess.STDOUT, text=True)
try:
deadline = time.monotonic() + 120
while True:
if process.poll() is not None:
pytest.fail(f"server exited during startup:\n{log_path.read_text(encoding='utf-8')}")
try:
_get_json(f"{base_url}/system_stats")
break
except urllib.error.URLError:
if time.monotonic() > deadline:
pytest.fail(f"server readiness timed out:\n{log_path.read_text(encoding='utf-8')}")
time.sleep(0.25)
yield base_url
finally:
process.terminate()
try:
process.wait(timeout=10)
except subprocess.TimeoutExpired:
process.kill()
process.wait(timeout=10)
PROMPT = {
"source": {
"class_type": "EmptyImage",
"inputs": {"width": 16, "height": 16, "batch_size": 1, "color": 0},
},
"save": {
"class_type": "SaveImage",
"inputs": {"images": ["source", 0], "filename_prefix": "force_rerun"},
},
"paid": {"class_type": "TestPaidOutputNode", "inputs": {"seed": 1}},
}
def _saved_files(history: dict) -> list:
return [image["filename"] for image in history["outputs"]["save"]["images"]]
@pytest.mark.execution
@pytest.mark.parametrize("cache_args", [[], ["--cache-classic"]], ids=["ram-pressure-default", "classic"])
def test_output_node_rewrites_its_file_on_a_cache_hit(tmp_path: Path, cache_args: list) -> None:
# Given a server started with the flag, and a prompt that has already run once
with _running_server(tmp_path, ["--force-output-rerun", *cache_args]) as base_url:
first = _run(base_url, PROMPT)
for name in _saved_files(first): # what a multi-tenant server does after upload
(tmp_path / "output" / name).unlink()
# When the identical prompt runs again
second = _run(base_url, PROMPT)
# Then the save node ran again and wrote a file that exists. (The name can
# repeat: the first file was deleted, so the counter starts over.)
assert _saved_files(second)
for name in _saved_files(second):
assert (tmp_path / "output" / name).is_file()
cached = _cached_nodes(second)
assert "save" not in cached
# and its upstream stayed cached
assert "source" in cached
# and the API output node was not re-run (that would repeat a paid call)
assert "paid" in cached
assert second["outputs"]["paid"] == first["outputs"]["paid"]
@pytest.mark.execution
def test_without_the_flag_a_cache_hit_replays_the_first_filename(tmp_path: Path) -> None:
# Control: the default behaviour this flag exists to change
with _running_server(tmp_path, ["--cache-classic"]) as base_url:
first = _run(base_url, PROMPT)
second = _run(base_url, PROMPT)
assert _saved_files(second) == _saved_files(first)
assert {"source", "save", "paid"} <= _cached_nodes(second)
@pytest.mark.execution
def test_a_node_reading_a_forced_output_node_gets_the_new_value(tmp_path: Path) -> None:
# Given an output node whose result feeds another node, already run once
prompt = {
"producer": {"class_type": "TestOutputPassthrough", "inputs": {"seed": 1}},
"show": {"class_type": "PreviewAny", "inputs": {"source": ["producer", 0]}},
}
with _running_server(tmp_path, ["--force-output-rerun", "--cache-classic"]) as base_url:
first = _run(base_url, prompt)
# When the identical prompt runs again
second = _run(base_url, prompt)
# Then the producer ran again, and the output node reading it (also forced)
# waited for that run instead of reading the previous run's value
produced = second["outputs"]["producer"]["text"]
assert produced != first["outputs"]["producer"]["text"]
assert second["outputs"]["show"]["text"] == produced
assert "producer" not in _cached_nodes(second)