Commit Graph
9 Commits
Author SHA1 Message Date
Mukkesh Ganesh 03e5ea0672 feat(server): report per-request prefill and decode timings in api responses (#814)
* feat(server): report per-request prefill and decode timings in api responses

* fix(server): take timings spans from the scheduler and gate them behind --enable-timings-report

* fix(scheduler): count accepted drafts as min(n, emitted) and stamp reqstats with perf_counter
2026-11-03 14:44:25 -08:00
A5bhinav ee43b2a113 fix(server): refuse n != 1 on /v1/completions instead of ignoring it (#842) 2026-10-29 17:15:32 -07:00
cherry77-cloud c8186396f5 fix(sampling): apply presence and frequency penalties (#527)
* fix(sampling): apply presence and frequency penalties

* fix(server): reject non-finite sampling penalties

* fix(server): bound presence and frequency penalties
2026-10-11 12:41:07 -07:00
Xiaoze Fan 08d728d5e3 feat(mm): serve image input on the Qwen families (#454)
* feat(mm): serve image input on the Qwen VL families

- frontend: OpenAI image_url, Anthropic image blocks and Responses
  input_image (http(s), base64, gated file://) become MMItems in the
  tokenizer workers; image spans carry content pseudo ids so the radix
  cache keys them by content; mrope positions are precomputed per request
- registry: EncoderSpec per family (Qwen3-VL, Qwen3.6, Qwen3.8-Flash-Next
  register a vision encoder serving image) and ModelSpec.mm_processor;
  EngineConfig.active_encoders decides what a process builds,
  served_modalities what it accepts; --text-model-only, --mm-disable,
  --mm-encoder-weights, --mm-embed-cache-device, --mm-max-pixels
- engine and scheduler: per-request encoder-cache row ownership, chunk
  planning on the batch (jobs, gather rows, scatter rows), items encoded
  right before the LM forward and gathered into batch.mm_embeds
- models: the Qwen VL vision tower on the text tower's TP layers with
  DeepStack, block weights streamed from pinned host banks by default,
  merger outputs parked on the host between blocks; SupportsMultimodal
  (encode, place_encoder_weights) and a shared embed_input_ids
- server: image inputs rejected with a client-facing reason when no
  vision encoder is served; GET /v1/stats reports model.input_modalities

* feat(mm): replace the pixel cap with a token budget and processor kwargs

- --image-min-tokens / --image-max-tokens replace --mm-max-pixels; MMProcessor.get_mm_processor_kwargs turns the MultimodalConfig into the processor call's arguments, the default passes mm.processor_kwargs through and Qwen VL converts the budget to pixel areas in size.shortest_edge / longest_edge
- --mm-processor-kwargs hands a JSON object to the checkpoint's image processor call, applied after the token budget
- MMProcessor is an ABC that owns the model path, the config and the lazily loaded image processor; get_mm_processor returns None when --mm-disable leaves no encoder to serve
- the frontend tokenizer, the tokenizer worker and the offline LLM build one processor each and hand it to TokenizeManager, so the tokenize message carries no per-request pixel cap
2026-09-13 15:39:33 -07:00
Earl Co 953565667f fix(server): honor configured output default across APIs (#411) 2026-09-11 20:02:23 -07:00
Shuo Yang 6223cda1b0 Support muse glimmer (#4)
* feat(muse_glimmer): serve Muse-Glimmer-30B
2026-08-18 12:26:05 -07:00
Xiaoze Fan d7cd6ed88b [Feature] unify reasoning effort on one shared scale (#5)
* feat(server): accept every reasoning-effort dialect

* refactor(server): derive thinking gears from the checkpoint, not a registry
2026-08-15 23:40:52 -07:00
Xiaoze Fan 4f510f1ba6 feat(server): enable reasoning_effort on /v1/chat/completions (#2)
* feat(server): enable reasoning_effort on /v1/chat/completions

* fix(server): glm thinking-off mislabeling and reasoning_effort mapping edge cases
2026-08-14 00:02:32 -07:00
Xiaoze FanandShuo Yang 3af9d90ee5 feat: initial open-source release
Co-authored-by: Shuo Yang <andy_yang@berkeley.edu>
2026-08-11 22:53:25 +00:00