* fix(kernel): order the batch-memcpy probe against the current stream
Adapt the stream dependency and busy-stream regression from #414
(https://github.com/FlashML-org/FreeToken/pull/414). Warm the JIT
before queuing the backlog so compilation cannot mask the race.
* fix(moe): enable prefill hit-d2d on rocm
* feat(moe): fetch locked-layer experts through pinned staging and probe the pin limit
Under hybrid, layers whose banks the pin quota could not take keep GPU slots of their own pools and fetch part of their misses through pinned staging that CPU threads fill (--moe-locked-fetch, --moe-locked-max-fetch). --moe-cpu-layers auto without FREETOKEN_PIN_BUDGET_GB now loads the banks unpinned, pins the staging, then pins layers middle-out until CUDA refuses one and OS-locks the rest.
* fix(moe): resize the locked fetch on cache rebuild and gather staging by device address
* feat(moe): log decode moe stats per layer group with --moe-collect-stats
* perf(moe): default the locked fetch to max(2, min(4, logical cpus / 4)) copy threads
* feat(moe): take the locked fetch share from the ft bench bw profile by default
* fix(moe): fuse the locked fetch into publish and gather kernels and address review findings
* refactor(moe): gather the locked staging with fast_index_copy_multi
* refactor(moe): move the locked fetch setup out of the engine and address review findings
* fix(moe): fail the boot when the locked staging cannot be pinned and keep locked layers out of the hybrid share log
* fix(kernel): correct jit matcher options and pdl arch guards
- with_dtype<Ts...>(sym) and with_device<Codes...>(sym) applied no options, so their dtype and device checks accepted anything
- index and store list kDLROCM, which the ignored check let through
- emit griddepcontrol only for !FREETOKEN_USE_ROCM && __CUDA_ARCH__ >= 900; it failed ptxas below sm_90 and does not exist on HIP
- declare the launch attribute cache only on CUDA
- use kWarpThreads for the index and store launch geometry
Co-authored-by: zihaomu <zihaomu@outlook.com>
* fix(kernel): treat zero as a matched size in tensor matcher
- a zero dimension counted as unset, so empty inputs failed with "Size value is not set" and a zero-row tensor did not catch a mismatched one
* fix(kernel): return early on empty index and store batches
- an empty batch launched a zero-sized grid, which CUDA and HIP reject; the leftover error then failed the next GPU call
Co-authored-by: zihaomu <zihaomu@outlook.com>
---------
Co-authored-by: zihaomu <zihaomu@outlook.com>
* feat: add simple ROCm GPU support for RDNA3 (gfx1100-1103)
- Add hip_compat.h shim mapping CUDA runtime API to HIP equivalents
- Update pinned_tensor.cpp to compile under both nvcc and hipcc
- Add ROCm detection in arch.py (is_rocm, get_rocm_gfx_arch, is_gfx11xx_family)
- Guard NVIDIA arch checks to return None on ROCm
- Skip nvcc version check in _toolchain.py when on ROCm
- Add ROCm build path in setup.py (ROCM_HOME, amdhip64, --offload-arch)
- Add _hip_cflags() in kernel/utils.py for JIT compilation on ROCm
- Add is_rocm() and driver_hip_version() in backend.py
- Add rocm-smi fallback in __main__.py for clangd generation
- Add TODO(ROCm) for NCCL->RCCL, flashinfer/sgl_kernel ROCm builds,
Triton autotune RDNA3 tuning, PDL equivalent, hiprtc JIT cache
- Add AMD ROCm classifier in pyproject.toml
* fix(rocm): fall back for single-bank fast index copy
* test(rocm): cover fast index copy fallback
* test(rocm): use explicit fast copy module import
* fix(rocm): complete RDNA3 runtime path
* fix(rocm): preserve CPU MoE implementation
* build(rocm): support RDNA4 discovery and modular SDKs
* fix(rocm): complete modular SDK runtime support
* test(rocm): harden foundation platform contracts
* fix(rocm): pin jit target and split install guide
* perf(kernel): restore ROCm non-temporal index copy
---------
Co-authored-by: bouclem <bouclement2012@gmail.com>
Co-authored-by: Yuu <206304251+nekomario28@users.noreply.github.com>
Co-authored-by: LZ-QWQ <lz1830837889@163.com>