Files
FreeToken/docs/install.md
0d652e73a4 feat(rocm): add RDNA3 and RDNA4 runtime foundation (#132)
* feat: add simple ROCm GPU support for RDNA3 (gfx1100-1103)

- Add hip_compat.h shim mapping CUDA runtime API to HIP equivalents
- Update pinned_tensor.cpp to compile under both nvcc and hipcc
- Add ROCm detection in arch.py (is_rocm, get_rocm_gfx_arch, is_gfx11xx_family)
- Guard NVIDIA arch checks to return None on ROCm
- Skip nvcc version check in _toolchain.py when on ROCm
- Add ROCm build path in setup.py (ROCM_HOME, amdhip64, --offload-arch)
- Add _hip_cflags() in kernel/utils.py for JIT compilation on ROCm
- Add is_rocm() and driver_hip_version() in backend.py
- Add rocm-smi fallback in __main__.py for clangd generation
- Add TODO(ROCm) for NCCL->RCCL, flashinfer/sgl_kernel ROCm builds,
  Triton autotune RDNA3 tuning, PDL equivalent, hiprtc JIT cache
- Add AMD ROCm classifier in pyproject.toml

* fix(rocm): fall back for single-bank fast index copy

* test(rocm): cover fast index copy fallback

* test(rocm): use explicit fast copy module import

* fix(rocm): complete RDNA3 runtime path

* fix(rocm): preserve CPU MoE implementation

* build(rocm): support RDNA4 discovery and modular SDKs

* fix(rocm): complete modular SDK runtime support

* test(rocm): harden foundation platform contracts

* fix(rocm): pin jit target and split install guide

* perf(kernel): restore ROCm non-temporal index copy

---------

Co-authored-by: bouclem <bouclement2012@gmail.com>
Co-authored-by: Yuu <206304251+nekomario28@users.noreply.github.com>
Co-authored-by: LZ-QWQ <lz1830837889@163.com>
2026-09-25 17:49:45 -07:00

1.9 KiB

Install

AMD users: see the AMD ROCm installation guide (WIP).

Requirements

  • Linux x86_64, NVIDIA GPU, driver r580+ (CUDA 13)
  • Python >= 3.10, with uv recommended (plain pip + venv works too)

Method 1: Install from PyPI

uv venv && source .venv/bin/activate
uv pip install "freetoken[accel]"

CUDA kernels are JIT-compiled on first use, need a CUDA 13 toolkit with nvcc on PATH.

Method 2: Install from source

git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"

Method 3: Nightly wheels

Every night main is built into a wheel pair on the rolling nightly release: the freetoken runtime (CPython 3.12, Linux x86_64) and the matching freetoken-kernel-cache with FreeToken's own CUDA kernels prebuilt. Install both from the URLs on that release page:

uv pip install \
  "freetoken[accel] @ https://github.com/FlashML-org/FreeToken/releases/download/nightly/<runtime wheel>" \
  "https://github.com/FlashML-org/FreeToken/releases/download/nightly/<kernel-cache wheel>"

Filenames carry a +g<sha> build stamp and change every night, and the nightly tag moves with them. Pin a wheel URL, never the tag. A local copy of the tag goes stale: git fetch leaves it alone and git fetch --tags refuses to overwrite it; refresh it with git fetch --force origin tag nightly. engine-linux_x86_64.json next to the wheels names the current pair for scripts.

Verify

source .venv/bin/activate
ft --version
ft serve --model ~/path/to/Qwen3.6-35B-A3B
curl http://127.0.0.1:1919/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3.6-35B-A3B","messages":[{"role":"user","content":"hi"}]}'

Then head to quickstart.md.