Files
tokenspeed/python
lipanpan03andFC-Li abb3ce0b5a fix(runtime): offload persistent metadata with weight regions (#3620)
## Summary

Releasing weights can leave ordinary CUDA allocator segments resident
because small persistent runtime tensors were allocated outside the
memory saver's regions. A packed cache table can keep an entire 2 MiB
segment alive through a backend's shared-storage view.

Allocate these tensors in the existing CPU-backed persistent-buffer pool
so weights release unmaps their physical storage and weights resume
restores their original addresses and contents:

| Allocation owner | Change |
| --- | --- |
| Sampling and executor inputs | Scope sampler construction and the
executor sentinel. |
| Attention and cache metadata | Scope arena geometry, capture
placeholders, target/draft metadata, and per-forward packed cache
tables. Live forwards and prefill capture share the managed table
factory. |
| Host transfers | Scope immutable geometry and workspace tables at
initialization and when later submissions grow them. |
| Graph capture | Register generator seed/offset state before entering
discardable graph capture; centralize capture barriers whose WORLD
communicator is initialized before managed regions. |

The existing release/drain order, CPU backup policy, CUDA graph path,
NCCL backend, and worker lifecycle remain in place. Tests cover disabled
mode, workspace growth, and content/address recovery through explicit
metadata inputs. A fresh CUDA subprocess regression checks geometry,
real GPU table packing and capture metadata across table growth and
repeated sleep/wake cycles, retaining old views and requiring zero
ordinary-pool bytes after release plus restored addresses and contents.

## Test Plan

- Current review revision: `pre-commit run --all-files` passed.
- Current review revision: local collection of
`test_torch_memory_saver_adapter.py`, `test_host_executor.py`, and
`test_retraction_image_executor.py` is blocked by missing
`tokenspeed_triton` (three collection errors; no tests executed). The
native metadata test requires CUDA VMM hooks and a fresh subprocess; it
has not been run locally. Runtime/GPU CI is still required on this head.
- Historical validation before the main rebase and this review refactor:
the adapter suite passed **24 tests** in a CUDA runtime. A fresh
**32-GPU** serving/training smoke completed two iterations and a final
weights restore/release; all **128 inference offload observations** had
**0 bytes in ordinary allocator segments** (`segment_pool_id=(0, 0)`),
with unchanged inference worker PIDs. This measures the ordinary Torch
pool, not total process GPU memory.
- That historical smoke used non-default NCCL buffer/channel settings
and training-side checkpoint-group cleanup. It does not validate a
configuration with those settings and cleanup removed, or establish an
isolated performance effect for this patch.
- Historical native single-GPU controls covered barrier, graph RNG, and
Host geometry sleep/wake across three cycles. The geometry control did
not execute its payload transfer kernel. An isolated GPU packed-table
baseline script failed in Triton `loadBinary` before reaching the
candidate and is not counted as passed; the complete serving smoke
exercised the packing path.
- GPU decode step latency has **not** been compared before/after region
entry. No latency or throughput neutrality claim is made. A full GPU
smoke has not been repeated after rebasing or this review refactor.

---------

Signed-off-by: lipanpan03 <41904587+lipanpan03@users.noreply.github.com>
Signed-off-by: FC-Li <robert_lexis@163.com>
Co-authored-by: FC-Li <robert_lexis@163.com>
2026-11-19 12:08:17 +08:00
..
2026-05-06 13:27:02 +00:00