mirror of
https://github.com/Comfy-Org/ComfyUI.git
synced 2026-11-22 06:04:20 +08:00
## Problem A server that keeps one `PromptExecutor` across jobs and deletes output files after each job (Comfy Cloud uploads, then removes them) answers a repeated prompt from the cache: `SaveImage` is not run, the cached `executed` message replays the first run's filename, that file no longer exists, and the job finishes as a success with no output. Cloud's own telemetry counted 2,368 such zero-output successes for 273 users in prod over 7 days at the start of November (not all of them are this cause). ## Change New opt-in flag `--force-output-rerun` (default off, so nothing changes for local users): - On a cache hit, a node with `OUTPUT_NODE = True` is executed again, so it writes its files again. Its inputs still come from the cache, so models, latents and decoded images are not recomputed: the cost is one encode and write per output node. - API nodes (`API_NODE = True`) are excluded. Some are output nodes, and re-running one repeats a paid external call. - Forced nodes are left out of the `execution_cached` message, so clients are not told a node was cached when it ran. ## Tests `tests/execution/test_force_output_rerun.py` starts a real server: - With the flag: run a prompt, delete its saved file, run the identical prompt again. The save node writes a new file that exists, its upstream `EmptyImage` stays cached, and a test output node marked `API_NODE` stays cached with its first output. - Control, without the flag: the second run replays the first filename and every node is cached. Red first: on the commit with only the flag, the flagged test failed because the second run replayed the first filename (`force_rerun_00001_.png`), while the no-flag control passed: https://github.com/Comfy-Org/ComfyUI/actions/runs/44228691563/job/134548112592. A follow-up commit corrects one assertion (the counter can reuse a name after the file is deleted, so the test checks the file exists instead), and the last commit wires in the behaviour. Tracking: https://linear.app/comfyorg/issue/BE-4802 (cloud enables the flag after this ships: https://linear.app/comfyorg/issue/BE-4803). ## Review notes An independent review raised three points: - **Upstream evicted from the RAM cache.** If memory pressure evicted an upstream result but kept the output node's entry, the upstream nodes run again. That already happens without this flag: `ExecutionList.add_node` schedules every uncached input of an output node whether or not the output node itself is cached, so the flag adds only the output node's own write. No change made. - **Output nodes created by an expanding node that is itself cached** are not re-run, because the cached parent does not expand again. This flag covers output nodes in the submitted prompt, which is how the frontend sends subgraphs (flattened). Known limit, noted here rather than widened in this PR. - **The test only used `--cache-classic`.** Fixed: the flagged test now also runs under the default RAM-pressure cache. The Cursor review panel then found that a node reading a forced output node's result could run before the re-run and read the old value. Fixed: `ExecutionList` takes a `force_rerun` predicate and treats those nodes as uncached while planning, with a test (`test_a_node_reading_a_forced_output_node_gets_the_new_value`). The flags are now read by truthiness, and `execution_cached` leaves out only forced nodes this run will execute.