Commit Graph
12 Commits
Author SHA1 Message Date
Xiaoze Fan e5856ac471 fix(utils): read stop ids from config.json without generation_config.json (#874) 2026-11-05 19:21:56 -08:00
Xiaoze Fan 33cbb3c54e feat(utils): add once-per-message logger methods (#858)
debug_once / info_once / warning_once log on rank 0, once per distinct message and args. select_kernel, which logged its choice once per layer, and the hand-rolled log-once guards now use them.
2026-11-03 15:31:08 -08:00
Xiaoze Fan 609191d81f feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj (#803)
* feat(gguf): load the vision tower from a llama.cpp mmproj

Qwen3.5 / Qwen3.6 / Qwen3.8 (qwen3vl_merger) and Gemma-4 26B-A4B / 31B (gemma4v) / 12B (gemma4uv) take image input from a separate mmproj file: --mmproj, else one in the model's own folder. The tower runs in bf16 (quantized mmproj files are dequantized) and its config comes from the GGUF metadata plus per-arch defaults.

* docs(gguf): describe mmproj image input without a model list

* test(gguf): drop the mmproj tests

* fix(gguf): escape the model folder when globbing for mmproj files

* fix(gguf): skip auto-discovered mmproj files the text model cannot use

* fix(gguf): serve text from a GGUF that carries vision tensors itself
2026-10-22 14:56:19 -07:00
Xiaoze Fan b6193e86f7 build(deps): upgrade transformers to 5.18 (#698) 2026-10-15 20:35:02 -07:00
YevheniiKotyrlo 235e2016dd feat(rotary): size the rope table for YaRN and add --hf-overrides (#573)
* feat(rotary): size the rope table for YaRN and add --hf-overrides

* fix(rotary): guard mrope's table width and copy --hf-overrides values

- MRotaryEmbedding asserts its cos/sin table is rotary_dim wide. A proportional
  rope's table spans the head, and the mrope kernel indexes the rotary_dim // 2
  section table with the table's width.
- _with_overrides deep-copies every value it stores: transformers fills rope
  defaults into the rope_parameters it is handed, which was the caller's dict.
- The rope-table tests the engine test covers fold into it, the two shared-table
  tests are one, and the --hf-overrides test moves to tests/utils/test_hf.py,
  where it loads the override against the same values written into config.json,
  for a model type transformers knows and one it does not.
- docs/cli.md lists --hf-overrides.

* fix(server): give --hf-overrides a fresh dict per parse

argparse hands its default object to every parse that omits the flag, so
default={} put one shared dict in every ServerArgs. The flag defaults to None and
becomes a new dict where the arguments are assembled, as --mm-processor-kwargs
does.

* docs(rotary): an indexer's heads differ in width from the attention's, not wider
2026-10-01 00:40:27 -07:00
0d652e73a4 feat(rocm): add RDNA3 and RDNA4 runtime foundation (#132)
* feat: add simple ROCm GPU support for RDNA3 (gfx1100-1103)

- Add hip_compat.h shim mapping CUDA runtime API to HIP equivalents
- Update pinned_tensor.cpp to compile under both nvcc and hipcc
- Add ROCm detection in arch.py (is_rocm, get_rocm_gfx_arch, is_gfx11xx_family)
- Guard NVIDIA arch checks to return None on ROCm
- Skip nvcc version check in _toolchain.py when on ROCm
- Add ROCm build path in setup.py (ROCM_HOME, amdhip64, --offload-arch)
- Add _hip_cflags() in kernel/utils.py for JIT compilation on ROCm
- Add is_rocm() and driver_hip_version() in backend.py
- Add rocm-smi fallback in __main__.py for clangd generation
- Add TODO(ROCm) for NCCL->RCCL, flashinfer/sgl_kernel ROCm builds,
  Triton autotune RDNA3 tuning, PDL equivalent, hiprtc JIT cache
- Add AMD ROCm classifier in pyproject.toml

* fix(rocm): fall back for single-bank fast index copy

* test(rocm): cover fast index copy fallback

* test(rocm): use explicit fast copy module import

* fix(rocm): complete RDNA3 runtime path

* fix(rocm): preserve CPU MoE implementation

* build(rocm): support RDNA4 discovery and modular SDKs

* fix(rocm): complete modular SDK runtime support

* test(rocm): harden foundation platform contracts

* fix(rocm): pin jit target and split install guide

* perf(kernel): restore ROCm non-temporal index copy

---------

Co-authored-by: bouclem <bouclement2012@gmail.com>
Co-authored-by: Yuu <206304251+nekomario28@users.noreply.github.com>
Co-authored-by: LZ-QWQ <lz1830837889@163.com>
2026-09-25 17:49:45 -07:00
Xiaoze Fan f7dbab7f15 fix(quant): load ModelOpt exports that ship no input_scale (#462) 2026-09-13 17:50:48 -07:00
Xiaoze Fan 477c8601b2 refactor(quant): config, scheme and method layers for quantization (#418)
* refactor(quant): config, scheme and method layers for checkpoint quantization

QuantConfig parses the checkpoint's quantization_config into a QuantScheme per
projection; every Linear*, ParallelLMHead and MoE layer takes quant_config and a
prefix, declares its tensors through a QuantMethod and selects its kernel at
construction. MoE kernels own the expert bank layout, pack and apply, so the
offload cache, the FTW converter and the CPU executor follow one layout.

- --quant-backend layer[.kind]=name selects kernels (--nvfp4-backend forwards to it)
- --moe-strategy replaces --moe-backend; the old spelling stays as an alias
- --moe-cpu-layers auto for Windows/WSL pin budgets; --moe-cache-auto respects the kernel's slot limit
- removes the per-family quantized layer classes, bank providers, format env switches and GLM runtime fp8
2026-09-08 23:37:23 -07:00
Xiaoze Fan a80b4d308a fix(hf): download the shards the safetensors index names (#336) 2026-09-01 19:43:19 -07:00
Shuo Yang 6223cda1b0 Support muse glimmer (#4)
* feat(muse_glimmer): serve Muse-Glimmer-30B
2026-08-18 12:26:05 -07:00
Xiaoze Fan ea5348b4e3 feat(kvcache): checkpoint decode state at the tool-call opener token 2026-08-13 18:32:02 +00:00
Xiaoze FanandShuo Yang 3af9d90ee5 feat: initial open-source release
Co-authored-by: Shuo Yang <andy_yang@berkeley.edu>
2026-08-11 22:53:25 +00:00