mirror of
https://github.com/lightseekorg/tokenspeed.git
synced 2026-11-22 11:34:19 +08:00
## Summary Releasing weights can leave ordinary CUDA allocator segments resident because small persistent runtime tensors were allocated outside the memory saver's regions. A packed cache table can keep an entire 2 MiB segment alive through a backend's shared-storage view. Allocate these tensors in the existing CPU-backed persistent-buffer pool so weights release unmaps their physical storage and weights resume restores their original addresses and contents: | Allocation owner | Change | | --- | --- | | Sampling and executor inputs | Scope sampler construction and the executor sentinel. | | Attention and cache metadata | Scope arena geometry, capture placeholders, target/draft metadata, and per-forward packed cache tables. Live forwards and prefill capture share the managed table factory. | | Host transfers | Scope immutable geometry and workspace tables at initialization and when later submissions grow them. | | Graph capture | Register generator seed/offset state before entering discardable graph capture; centralize capture barriers whose WORLD communicator is initialized before managed regions. | The existing release/drain order, CPU backup policy, CUDA graph path, NCCL backend, and worker lifecycle remain in place. Tests cover disabled mode, workspace growth, and content/address recovery through explicit metadata inputs. A fresh CUDA subprocess regression checks geometry, real GPU table packing and capture metadata across table growth and repeated sleep/wake cycles, retaining old views and requiring zero ordinary-pool bytes after release plus restored addresses and contents. ## Test Plan - Current review revision: `pre-commit run --all-files` passed. - Current review revision: local collection of `test_torch_memory_saver_adapter.py`, `test_host_executor.py`, and `test_retraction_image_executor.py` is blocked by missing `tokenspeed_triton` (three collection errors; no tests executed). The native metadata test requires CUDA VMM hooks and a fresh subprocess; it has not been run locally. Runtime/GPU CI is still required on this head. - Historical validation before the main rebase and this review refactor: the adapter suite passed **24 tests** in a CUDA runtime. A fresh **32-GPU** serving/training smoke completed two iterations and a final weights restore/release; all **128 inference offload observations** had **0 bytes in ordinary allocator segments** (`segment_pool_id=(0, 0)`), with unchanged inference worker PIDs. This measures the ordinary Torch pool, not total process GPU memory. - That historical smoke used non-default NCCL buffer/channel settings and training-side checkpoint-group cleanup. It does not validate a configuration with those settings and cleanup removed, or establish an isolated performance effect for this patch. - Historical native single-GPU controls covered barrier, graph RNG, and Host geometry sleep/wake across three cycles. The geometry control did not execute its payload transfer kernel. An isolated GPU packed-table baseline script failed in Triton `loadBinary` before reaching the candidate and is not counted as passed; the complete serving smoke exercised the packing path. - GPU decode step latency has **not** been compared before/after region entry. No latency or throughput neutrality claim is made. A full GPU smoke has not been repeated after rebasing or this review refactor. --------- Signed-off-by: lipanpan03 <41904587+lipanpan03@users.noreply.github.com> Signed-off-by: FC-Li <robert_lexis@163.com> Co-authored-by: FC-Li <robert_lexis@163.com>