* feat: add simple ROCm GPU support for RDNA3 (gfx1100-1103) - Add hip_compat.h shim mapping CUDA runtime API to HIP equivalents - Update pinned_tensor.cpp to compile under both nvcc and hipcc - Add ROCm detection in arch.py (is_rocm, get_rocm_gfx_arch, is_gfx11xx_family) - Guard NVIDIA arch checks to return None on ROCm - Skip nvcc version check in _toolchain.py when on ROCm - Add ROCm build path in setup.py (ROCM_HOME, amdhip64, --offload-arch) - Add _hip_cflags() in kernel/utils.py for JIT compilation on ROCm - Add is_rocm() and driver_hip_version() in backend.py - Add rocm-smi fallback in __main__.py for clangd generation - Add TODO(ROCm) for NCCL->RCCL, flashinfer/sgl_kernel ROCm builds, Triton autotune RDNA3 tuning, PDL equivalent, hiprtc JIT cache - Add AMD ROCm classifier in pyproject.toml * fix(rocm): fall back for single-bank fast index copy * test(rocm): cover fast index copy fallback * test(rocm): use explicit fast copy module import * fix(rocm): complete RDNA3 runtime path * fix(rocm): preserve CPU MoE implementation * build(rocm): support RDNA4 discovery and modular SDKs * fix(rocm): complete modular SDK runtime support * test(rocm): harden foundation platform contracts * fix(rocm): pin jit target and split install guide * perf(kernel): restore ROCm non-temporal index copy --------- Co-authored-by: bouclem <bouclement2012@gmail.com> Co-authored-by: Yuu <206304251+nekomario28@users.noreply.github.com> Co-authored-by: LZ-QWQ <lz1830837889@163.com>
1.9 KiB
Install
AMD users: see the AMD ROCm installation guide (WIP).
Requirements
- Linux x86_64, NVIDIA GPU, driver r580+ (CUDA 13)
- Python >= 3.10, with uv recommended (plain
pip+venvworks too)
Method 1: Install from PyPI
uv venv && source .venv/bin/activate
uv pip install "freetoken[accel]"
CUDA kernels are JIT-compiled on first use, need a CUDA 13 toolkit with nvcc on PATH.
Method 2: Install from source
git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"
Method 3: Nightly wheels
Every night main is built into a wheel pair on the rolling
nightly release:
the freetoken runtime (CPython 3.12, Linux x86_64) and the matching
freetoken-kernel-cache with FreeToken's own CUDA kernels prebuilt.
Install both from the URLs on that release page:
uv pip install \
"freetoken[accel] @ https://github.com/FlashML-org/FreeToken/releases/download/nightly/<runtime wheel>" \
"https://github.com/FlashML-org/FreeToken/releases/download/nightly/<kernel-cache wheel>"
Filenames carry a +g<sha> build stamp and change every night, and the nightly
tag moves with them. Pin a wheel URL, never the tag. A local copy of the tag goes
stale: git fetch leaves it alone and git fetch --tags refuses to overwrite it;
refresh it with git fetch --force origin tag nightly. engine-linux_x86_64.json
next to the wheels names the current pair for scripts.
Verify
source .venv/bin/activate
ft --version
ft serve --model ~/path/to/Qwen3.6-35B-A3B
curl http://127.0.0.1:1919/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"Qwen3.6-35B-A3B","messages":[{"role":"user","content":"hi"}]}'
Then head to quickstart.md.