3 Commits
Author SHA1 Message Date
Xiaoze Fan 609191d81f feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj (#803)
* feat(gguf): load the vision tower from a llama.cpp mmproj

Qwen3.5 / Qwen3.6 / Qwen3.8 (qwen3vl_merger) and Gemma-4 26B-A4B / 31B (gemma4v) / 12B (gemma4uv) take image input from a separate mmproj file: --mmproj, else one in the model's own folder. The tower runs in bf16 (quantized mmproj files are dequantized) and its config comes from the GGUF metadata plus per-arch defaults.

* docs(gguf): describe mmproj image input without a model list

* test(gguf): drop the mmproj tests

* fix(gguf): escape the model folder when globbing for mmproj files

* fix(gguf): skip auto-discovered mmproj files the text model cannot use

* fix(gguf): serve text from a GGUF that carries vision tensors itself
2026-10-22 14:56:19 -07:00
Mukkesh Ganesh 23fc922bf8 feat(deepseek_v4): serve image input on the V4-Flash-Vision release (#502)
* feat(deepseek_v4): serve image input on the V4-Flash-Vision release

The block layout depends on where the image lands: lead pads align its first row token to
a 4-token compression stride, so the item hash carries the layout and two alignments never
share a radix key.

The block's tokens all carry one content pad id (the shared pad scheme); the sparse
attention reads its spans from batch.mm_spans, which the scheduler recovers from the flat
block ends its existing plan already reports.

* fix(mm): keep the shared mmprocessor signature and own the dsv4 offset in its apply

* fix(models): build dsv4 image spans in the model and start them at image_start

* fix(models): mount the dsv4 vision tower under visual and keep its weights unquantized

* fix(models): keep dsv4 image blocks whole across chunked prefill
2026-10-08 14:42:39 -07:00
Xiaoze Fan 08d728d5e3 feat(mm): serve image input on the Qwen families (#454)
* feat(mm): serve image input on the Qwen VL families

- frontend: OpenAI image_url, Anthropic image blocks and Responses
  input_image (http(s), base64, gated file://) become MMItems in the
  tokenizer workers; image spans carry content pseudo ids so the radix
  cache keys them by content; mrope positions are precomputed per request
- registry: EncoderSpec per family (Qwen3-VL, Qwen3.6, Qwen3.8-Flash-Next
  register a vision encoder serving image) and ModelSpec.mm_processor;
  EngineConfig.active_encoders decides what a process builds,
  served_modalities what it accepts; --text-model-only, --mm-disable,
  --mm-encoder-weights, --mm-embed-cache-device, --mm-max-pixels
- engine and scheduler: per-request encoder-cache row ownership, chunk
  planning on the batch (jobs, gather rows, scatter rows), items encoded
  right before the LM forward and gathered into batch.mm_embeds
- models: the Qwen VL vision tower on the text tower's TP layers with
  DeepStack, block weights streamed from pinned host banks by default,
  merger outputs parked on the host between blocks; SupportsMultimodal
  (encode, place_encoder_weights) and a shared embed_input_ids
- server: image inputs rejected with a client-facing reason when no
  vision encoder is served; GET /v1/stats reports model.input_modalities

* feat(mm): replace the pixel cap with a token budget and processor kwargs

- --image-min-tokens / --image-max-tokens replace --mm-max-pixels; MMProcessor.get_mm_processor_kwargs turns the MultimodalConfig into the processor call's arguments, the default passes mm.processor_kwargs through and Qwen VL converts the budget to pixel areas in size.shortest_edge / longest_edge
- --mm-processor-kwargs hands a JSON object to the checkpoint's image processor call, applied after the token budget
- MMProcessor is an ABC that owns the model path, the config and the lazily loaded image processor; get_mm_processor returns None when --mm-disable leaves no encoder to serve
- the frontend tokenizer, the tokenizer worker and the offline LLM build one processor each and hand it to TokenizeManager, so the tokenize message carries no per-request pixel cap
2026-09-13 15:39:33 -07:00