debug_once / info_once / warning_once log on rank 0, once per distinct message and args. select_kernel, which logged its choice once per layer, and the hand-rolled log-once guards now use them.
* feat(gguf): load the vision tower from a llama.cpp mmproj
Qwen3.5 / Qwen3.6 / Qwen3.8 (qwen3vl_merger) and Gemma-4 26B-A4B / 31B (gemma4v) / 12B (gemma4uv) take image input from a separate mmproj file: --mmproj, else one in the model's own folder. The tower runs in bf16 (quantized mmproj files are dequantized) and its config comes from the GGUF metadata plus per-arch defaults.
* docs(gguf): describe mmproj image input without a model list
* test(gguf): drop the mmproj tests
* fix(gguf): escape the model folder when globbing for mmproj files
* fix(gguf): skip auto-discovered mmproj files the text model cannot use
* fix(gguf): serve text from a GGUF that carries vision tensors itself
* feat(rotary): size the rope table for YaRN and add --hf-overrides
* fix(rotary): guard mrope's table width and copy --hf-overrides values
- MRotaryEmbedding asserts its cos/sin table is rotary_dim wide. A proportional
rope's table spans the head, and the mrope kernel indexes the rotary_dim // 2
section table with the table's width.
- _with_overrides deep-copies every value it stores: transformers fills rope
defaults into the rope_parameters it is handed, which was the caller's dict.
- The rope-table tests the engine test covers fold into it, the two shared-table
tests are one, and the --hf-overrides test moves to tests/utils/test_hf.py,
where it loads the override against the same values written into config.json,
for a model type transformers knows and one it does not.
- docs/cli.md lists --hf-overrides.
* fix(server): give --hf-overrides a fresh dict per parse
argparse hands its default object to every parse that omits the flag, so
default={} put one shared dict in every ServerArgs. The flag defaults to None and
becomes a new dict where the arguments are assembled, as --mm-processor-kwargs
does.
* docs(rotary): an indexer's heads differ in width from the attention's, not wider
* feat: add simple ROCm GPU support for RDNA3 (gfx1100-1103)
- Add hip_compat.h shim mapping CUDA runtime API to HIP equivalents
- Update pinned_tensor.cpp to compile under both nvcc and hipcc
- Add ROCm detection in arch.py (is_rocm, get_rocm_gfx_arch, is_gfx11xx_family)
- Guard NVIDIA arch checks to return None on ROCm
- Skip nvcc version check in _toolchain.py when on ROCm
- Add ROCm build path in setup.py (ROCM_HOME, amdhip64, --offload-arch)
- Add _hip_cflags() in kernel/utils.py for JIT compilation on ROCm
- Add is_rocm() and driver_hip_version() in backend.py
- Add rocm-smi fallback in __main__.py for clangd generation
- Add TODO(ROCm) for NCCL->RCCL, flashinfer/sgl_kernel ROCm builds,
Triton autotune RDNA3 tuning, PDL equivalent, hiprtc JIT cache
- Add AMD ROCm classifier in pyproject.toml
* fix(rocm): fall back for single-bank fast index copy
* test(rocm): cover fast index copy fallback
* test(rocm): use explicit fast copy module import
* fix(rocm): complete RDNA3 runtime path
* fix(rocm): preserve CPU MoE implementation
* build(rocm): support RDNA4 discovery and modular SDKs
* fix(rocm): complete modular SDK runtime support
* test(rocm): harden foundation platform contracts
* fix(rocm): pin jit target and split install guide
* perf(kernel): restore ROCm non-temporal index copy
---------
Co-authored-by: bouclem <bouclement2012@gmail.com>
Co-authored-by: Yuu <206304251+nekomario28@users.noreply.github.com>
Co-authored-by: LZ-QWQ <lz1830837889@163.com>
* refactor(quant): config, scheme and method layers for checkpoint quantization
QuantConfig parses the checkpoint's quantization_config into a QuantScheme per
projection; every Linear*, ParallelLMHead and MoE layer takes quant_config and a
prefix, declares its tensors through a QuantMethod and selects its kernel at
construction. MoE kernels own the expert bank layout, pack and apply, so the
offload cache, the FTW converter and the CPU executor follow one layout.
- --quant-backend layer[.kind]=name selects kernels (--nvfp4-backend forwards to it)
- --moe-strategy replaces --moe-backend; the old spelling stays as an alias
- --moe-cpu-layers auto for Windows/WSL pin budgets; --moe-cache-auto respects the kernel's slot limit
- removes the per-family quantized layer classes, bank providers, format env switches and GLM runtime fp8