* feat(spec): hand the pending token to Proposer.propose and return a SpecProposal * feat(engine): Sampler.probs and sparse-q rejection * feat(spec): let a proposer declare slot states riding the GDN slots * feat(models): export layer outputs to the feature sink in the qwen3_5 and qwen4_exp families * feat(attention): record the source slot of each tracked request * feat(kernel): DFlash2 draft kernels * feat(models): DFlash2 draft family * feat(spec): DFlashWindow * feat(spec): DFlashProposer * docs(cli): add dflash to the speculative decoding section * refactor(engine): share the sampling backend choice and drop the extra renorm in Sampler.probs * fix(models): build the DFlash rope table uncached and refuse draft scalings it does not apply * refactor(spec): share the draft path check between the MTP and DFlash proposers * test(e2e): compare rebuilt outputs exactly only without hybrid MoE * refactor(spec): keep proposer slot states on EngineConfig instead of replacing model_config * perf(spec): compute the verify distribution once for sparse-q rejection * perf(models): let the feature sink add the residual on the rows it keeps * perf(models): share the DFlash draft K/V weights with the ring projection * fix(kernel): keep the dflash selector off zero-probability candidates * test(spec): drop the DFlash tests that the draft forward and Monte Carlo tests already cover * fix(spec): default and cap DFlash steps at the draft's block_size - 1 * fix(models): reject DFlash draft configs the draft forward cannot serve * test(e2e): read the MoE strategy from the rebuild test's extra args * docs(spec): say that slot-state owners write their own prefill track snapshot * test(models): move the DFlash draft parity tests under tests/models/dflash * refactor(spec): drop the no-op verify handler of the DFlash window * refactor(models): fold the DFlash candidate top-k into propose * refactor(spec): declare the draft step limits through Proposer.num_steps only * refactor(kernel): make the DFlash ring attention stateless and keep its workspace on the ring * refactor(models): name the DFlash draft submodules like the other model families * refactor(spec): move SpecProposal next to DraftContext in spec/proposer.py * refactor(spec): hand the verify the draft distribution over the whole vocabulary * refactor(spec): call the DFlashRing object dflash_ring and its length ring_len * refactor(models): split the DFlash draft's propose into compute_candidates and sample_path * refactor(spec): keep the draft distribution on its support in SpecProposal * fix(spec): drop the step's proposal once the verify has read it * refactor(spec): name the DFlash feature sinks DFlashPrefillSink and DFlashVerifySink * refactor(models): read the declared Batch.features directly instead of through getattr * test(dflash): check the window rebuild inside the block forward test * test(spec): share the plain run of the verify contract and run the DFlash draft once per family * test(kvcache): drop the window cases from the pool tests and check a layerless state's bytes * test(spec): check the verify probs of each row against a torch top-k and top-p reference * test(e2e): add an accept-len floor and a prefill chunk knob to the AIME gate * refactor(scheduler): let callers take the speculative counts from the status reporter * test(spec): keep the shared plain runs of the verify contract in their own temp dirs * test(models): give the ricursive test's stand-in batch a features field
20 KiB
CLI reference
ft <command> [args]
| Command | Purpose |
|---|---|
ft serve |
Start the API server (OpenAI /v1/*, Anthropic /v1/messages, Responses) |
ft shell |
Chat with a server in the terminal |
ft ctl |
Query and manage a running server over HTTP |
ft launch |
Configure and launch a coding agent against a server |
ft checkpoint |
Convert an HF checkpoint to the FTW fast-load format |
ft bench bw |
Benchmark CPU vs PCIe bandwidth to calibrate hybrid MoE decode |
ft --version prints the installed version (torch-free; nightly wheels carry a
+g<sha> build stamp, tagged releases a bare version). Every command supports
--help.
ft serve
ft serve --model <path-or-hf-id> [options]
--model is the only required flag — dtype, attention backend, MoE backend,
MoE cache size, KV capacity, CUDA-graph sizes and the tool-call/reasoning
parsers all resolve automatically from the checkpoint and the GPU.
Model
| Flag | Default | Meaning |
|---|---|---|
--model-path, --model |
required | Local dir, HF repo id, a .gguf file, or an FTW dir (auto-detected) |
--served-model-name |
basename of --model |
Model id reported by /v1/models |
--hf-overrides |
— | JSON object applied to the checkpoint's config as vLLM's --hf-overrides: a nested config section updates key by key, any other value is replaced whole. A YaRN rope_parameters override serves original_max_position_embeddings * factor positions |
Server & runtime
| Flag | Default | Meaning |
|---|---|---|
--host |
127.0.0.1 | Bind address |
--port |
1919 | Bind port |
--api-key |
$FREETOKEN_API_KEY, else off |
Require the key on every route except /health, as Authorization: Bearer <key> or x-api-key: <key>; other requests get a 401 |
--gpu |
GPU 0 | GPU to run on: a UUID from nvidia-smi -L or an nvidia-smi index; see below |
--max-running-requests |
4 | Max concurrently running requests |
--max-output-tokens |
32768 | Default output budget for requests that omit one |
--max-seq-len-override |
from checkpoint | Max sequence length |
--max-prefill-length |
8192 | Chunked-prefill chunk size in tokens; with offloaded experts see performance/prefill.md |
--cuda-graph-max-bs, --graph |
= max running requests | Max batch size captured as CUDA graphs |
--decode-log-interval |
40 | Scheduler status line every N decode steps |
Choosing a GPU
For example, a machine with an RTX 5090 and an RTX 3060 Ti:
$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 3060 Ti (UUID: GPU-2f3a9b1c-8d7e-4a05-b6c1-0e5f9a3d7b42)
GPU 1: NVIDIA GeForce RTX 5090 (UUID: GPU-9e8d7c6b-5a49-4f13-8207-c1b0a4e6d3f5)
ft serve --model ... --gpu 1 # by nvidia-smi index -- the 5090
ft serve --model ... --gpu GPU-9e8d7c6b # the same card by UUID (a unique prefix is enough)
KV cache & memory
| Flag | Default | Meaning |
|---|---|---|
--memory-ratio |
0.9 | Fraction of free VRAM the engine may use (weights + MoE cache + KV) |
--num-pages / --num-tokens |
auto | KV capacity override in pages / tokens (mutually exclusive; auto sizes from VRAM left after weights and MoE cache) |
--embed-weights |
gpu | Where the input token embedding table lives. host keeps it in pinned host RAM and each lookup reads its rows over PCIe, so its VRAM goes to the expert cache and KV. Refused when the table is tied to the LM head |
--page-size |
1 | KV page size; DSV4 forces 128, the TRTLLM backend needs 16/32/64, SWA models require 1 |
--cache-type |
radix | radix (prefix reuse; SWA/GDN-aware variants picked automatically) or naive |
--attention-backend, --attn |
auto | trtllm/fi/fa/triton/dsv4_sparse/dsa; prefill,decode pair allowed; auto picks per model + GPU |
MoE offload
See models.md for what each strategy does, performance/prefill.md for prefill speed, and performance/decode.md for decode speed.
| Flag | Default | Meaning |
|---|---|---|
--moe-strategy |
auto | fused/offload/cpu/hybrid; auto → offload; fused on unified-memory GPUs (GB10), see models.md. --moe-backend is the deprecated old spelling |
--quant-backend |
auto | Kernel per quantized layer type, layer[.kind]=name entries: linear=marlin,moe=b12x or moe.nvfp4=triton. A layer-level entry applies to every kind whose table lists the name |
--nvfp4-backend |
— | Deprecated: stands in for --quant-backend moe.nvfp4=<marlin|b12x|triton> (flashinfer means b12x); cannot be combined with --quant-backend |
--moe-cache-size / --moe-cache-rate / --moe-cache-mem / --moe-cache-auto |
auto | GPU expert-cache size as slots / fraction of all experts / memory such as 12GiB / sized from free VRAM (mutually exclusive; auto is enabled by default for offload-family strategies) |
--moe-cache-scope |
unified | unified: one LRU pool of slots per expert quant (quant type and bank layout), so all MoE layers share one pool when they share a quant format; per-layer: each layer has its own pool. per-layer does not combine with --moe-strategy cpu or --moe-cpu-layers |
--moe-cache-split |
equal | How the pools share the slots: equal (in proportion to their layer counts), or profile:<path>: every pool keeps top_k slots and the rest go to the most used experts in the profile's per-layer expert counts. With offload, a pool smaller than the expert count lowers --max-running-requests to its slots / top_k |
--kv-reserve-tokens |
8192 | KV token floor reserved before --moe-cache-auto fills experts |
--moe-cpu-threads |
physical cores | CPU worker threads for the cpu/hybrid executor |
--moe-cpu-layers |
all on GPU | With offload/hybrid: which MoE layers decode on CPU (3,7,11, a count, a fraction, or auto). auto is for hosts that cap CUDA pinned memory (Windows/WSL): it pins layers from the middle out until CUDA refuses one and OS-locks the rest (FREETOKEN_PIN_BUDGET_GB and FTW checkpoints lock from a size estimate instead), see models.md; every value needs an expert format the CPU executor serves (bf16, fp8_block, nvfp4, mxfp4, GGUF on x86 CPUs, except gpt-oss GGUF files whose per-expert biases decode on the GPU only) |
--moe-hybrid-max-fetch |
auto | With hybrid: max experts fetched over PCIe per layer per step; rest computed on CPU |
--moe-locked-fetch |
auto | With hybrid and locked layers: share of a locked layer's misses fetched per step into its GPU slots through pinned staging. Auto takes p / (2p + c) from the ft bench bw row of the layer's kind (at most 1 per step without a row); 0 = locked layers decode on CPU only |
--moe-locked-max-fetch |
8 | Most experts a locked layer fetches per step; sizes the staging (that many rows of the widest expert, per bank) |
--moe-collect-stats |
off | Log decode MoE stats with each decode log line: hit rate, and per layer-step the misses, experts fetched and experts computed on CPU, for pinned, locked and CPU-only layers |
--moe-prefill-hit-d2d |
off | Prefill: copy cache-hit experts device-side, stream only misses (CUDA >= 13) |
--disable-moe-prefill-overlap |
overlap on | Disable the two-buffer prefill copy overlap |
Speculative decoding
| Flag | Default | Meaning |
|---|---|---|
--speculative-algorithm, --spec-algo |
off | Draft method. mtp drafts with the checkpoint's multi-token-prediction layer: Qwen3.5 / Qwen3.6 HF or GGUF checkpoints, and Qwen3.8-Flash-Next HF or GGUF. dflash drafts a block of tokens per step with a DFlash 2 draft checkpoint over the target's layer outputs, for the Qwen3.5 / Qwen3.6 / Qwen3.8 family and Qwen3.8-Flash-Next: incoai/Qwen3.8-27B-DFlash2 for Qwen3.8-27B, incoai/Qwen3.6-35B-A3B-DFlash2 for Qwen3.6-35B-A3B, incoai/Qwen3.8-Flash-Next-DFlash2 for Qwen3.8-Flash-Next |
--speculative-num-steps, --spec-num-steps |
the method's default (3 for mtp, the draft's block_size - 1 for dflash) |
Draft tokens per decode step, 1 to 8 for mtp; 1 to the draft's block_size - 1 for dflash |
--speculative-draft-model-path, --spec-draft-model |
the target checkpoint | Where the draft weights load from. A Qwen3.8-Flash-Next target also takes a Strata MTP pack (a GGUF that keeps the checkpoint's mtp.* tensors under their HF names), and a Qwen3.8-Flash-Next GGUF target needs one; a Qwen3.5 / Qwen3.6 GGUF target drafts from its own nextn MTP block (the block llama.cpp keeps after the last decoder layer), or from an mtp-*.gguf sidecar of the same model (unsloth's MTP/ releases) given here; for mtp, a separate checkpoint folder is an HF-target option, read with the target's quantization config, and a GGUF target refuses a folder draft. dflash requires it: the DFlash 2 draft checkpoint (bf16 safetensors with its config.json), whose target_layer_ids, num_target_layers, hidden size and vocabulary must match the target |
Speculative decoding needs --tp 1 and runs without overlap scheduling; targets with sliding-window or DSV4 window-tier KV are refused at startup. mtp refuses an FTW checkpoint as the draft source, since FTW drops the mtp.* tensors, and a Qwen3.5 / Qwen3.6 GGUF converted without its MTP layer (llama.cpp --no-mtp) when no mtp-*.gguf sidecar is given: such a trunk has no draft of its own to read, while a --no-mtp trunk plus a sidecar works (the trunk config's shapes, the sidecar's weights).
dflash keeps the draft's context in a per-request ring of the draft's sliding window (2048 positions for the released drafts) and, on a hybrid GDN target, in a window on the GDN state slots, so a prefix-cache hit restores it with the GDN snapshot; the window adds (window + 64) x hidden activation-dtype bytes to every GDN slot (/v1/cache/status counts them). A target without GDN layers needs --cache-type naive. Quantized drafts and DFlash v1 drafts are refused. With sampling, the drafts are drawn from the draft's own distribution over its top-16 candidates and verified by rejection sampling against the target's distribution, so the output distribution is the target's.
API behaviour
| Flag | Default | Meaning |
|---|---|---|
--sampling-defaults |
model | Fill unspecified sampling params from the checkpoint's generation_config.json (none = framework defaults) |
--tool-call-parser |
auto | Tool-call format; auto-inferred from the model family |
--reasoning-parser |
auto | Splits chain-of-thought into reasoning_content; auto-inferred; off disables |
--enable-cache-report |
off | Report prefix-cache hits in each response's usage block |
--enable-timings-report |
off | Attach a llama.cpp-shaped per-request timings object to responses |
--anthropic-inline-system |
auto | Placement of late Anthropic system instructions: auto, preserve, or fold |
With --enable-timings-report every chat completion, completion and Responses response
also carries a llama.cpp-shaped timings object next to usage (cache_n, prompt_n,
prompt_ms, prompt_per_token_ms, prompt_per_second, predicted_n, predicted_ms,
predicted_per_token_ms, predicted_per_second, plus queue_ms as a FreeToken
extension and draft_n / draft_n_accepted under speculative decoding), so frontends
like llama-swap and Open Web UI show per-request prefill/decode rates. All spans are
measured in the scheduler: queue_ms covers admission to the first prefill chunk,
prompt_ms the first chunk to the first sampled token (chunk gaps included),
predicted_ms the first to the last committed token; prompt_n counts only the
evaluated tokens and the decode rates divide by predicted_n - 1, as in llama.cpp. On
streams the object rides the final chunk; a multi-prompt /v1/completions request
reports the first prompt's spans, as llama.cpp does. /v1/messages never carries it
(llama.cpp emits no timings there either).
Image input
Experimental. Needs a checkpoint whose family registers a vision encoder (models.md lists them and how each one
maps the flags below); a request carrying images is rejected otherwise. Images are accepted on all three protocols (OpenAI image_url,
Anthropic image blocks, Responses input_image) as an http(s) URL or base64. Images inside a tool
result (an Anthropic tool_result block from Claude Code's Read, a Responses function_call_output
from Codex's view_image) are moved to the user turn that follows the tool message, as vLLM does,
because chat templates render tool messages as plain text.
GET /v1/stats reports what the server accepts as model.input_modalities (["text"] or ["text", "image"]),
so a client can gate its attachment controls without reading the checkpoint config.
| Flag | Default | Meaning |
|---|---|---|
--text-model-only |
off | Serve a multimodal checkpoint text-only: no encoder tower is built (its VRAM goes to the KV/expert pools) and every multimodal input is rejected. Same as --mm-disable with every encoder kind |
--mm-disable |
none | Encoder towers to leave unbuilt (vision, audio); every input they would serve is rejected |
--mmproj |
found next to the model | GGUF models only: the llama.cpp vision tower file (mmproj-*.gguf), see Image input. Without it, FreeToken takes a *mmproj*.gguf that fits the model (output width, tower type, image tokens) from the model file's own folder, BF16 before F32, F16 and quantized files. With none, or with files of several models it cannot tell apart, the model serves text only; a repo that keeps each quant in a subfolder and the mmproj at the top needs --mmproj. The log names the file it loads |
--mm-encoder-weights |
host | Where the encoder tower's block weights live. host streams them from pinned host banks two blocks at a time behind the compute, so the GPU holds two blocks instead of the whole tower; small images pay the copy time, large ones hide it behind the compute. gpu keeps them resident. An encoder without a block stack stays resident either way |
--image-min-tokens, --image-max-tokens |
processor defaults | Per-image token budget: the image processor resizes every image to take between these many tokens, converted to the family's own units by its processor. A family with fixed budgets honors the maximum only and refuses one below its smallest budget at start-up |
--mm-processor-kwargs |
none | JSON object of extra keyword arguments for the checkpoint's image processor call, for knobs the token budget does not cover; applied after the budget, so an explicit key wins |
--mm-embed-cache-device |
cpu | Where encoded image embeddings live between prefill chunks. cpu keeps them out of the VRAM budget; cuda skips the copy back |
--allowed-media-domains |
any | Comma-separated hostname allowlist for image URLs; requests for other domains are rejected with a 400. Empty allows any domain |
--allowed-local-media-path |
off | Directory file:// image refs may be read from; unset rejects local files |
ft shell
ft shell # attach to a running server
ft shell --model ~/models/Qwen3.6-35B-A3B # serve + chat in one process
- Attach mode talks to
--server URL(defaulthttp://127.0.0.1:1919) --api-key KEY(default$FREETOKEN_API_KEY) attaches to a server started with--api-key; with--modelthe shell uses that server's own key/helpinside the shell lists the commands (/think,/cache,/reset).
ft ctl
ft ctl [--base-url http://127.0.0.1:1919] [--api-key KEY] [--timeout 10] [--json] <subcommand>
--api-key defaults to $FREETOKEN_API_KEY.
| Subcommand | Endpoint | Purpose |
|---|---|---|
health |
GET /health |
Server status, model, load progress |
stats |
GET /v1/stats |
Throughput, latency, VRAM, pool occupancy, accepted input modalities |
generate [prompt] [--max-tokens N] [--ignore-eos] |
POST /generate |
Raw completion smoke test (no chat template) |
cache |
GET /v1/cache/status |
Cache pool table |
cache --moe N | --kv N | --mamba N | --swa N [--wait 300] |
POST /v1/cache/rebuild |
Live pool resizing without a restart (k/m suffixes; --kv/--swa in tokens) |
requests [--since N] [--limit N] |
GET /v1/requests |
Recent request ring |
ft launch
ft launch {claude,codex,dsh,hermes,openclaw,opencode} [options] [-- <agent args>]
Discovers the served model via /v1/models, writes the agent's provider
config, installs the agent CLI if missing, then launches it. Cloud API keys
(ANTHROPIC_API_KEY, OPENAI_API_KEY, …) are cleared from the child
environment so the agent cannot silently fall back to a paid endpoint.
When /v1/stats reports image among model.input_modalities, the written
config declares the model image-capable, which Codex, OpenCode, OpenClaw and
dsh require before their image tools and attachments send anything; Claude
Code and Hermes need no declaration.
| Flag | Meaning |
|---|---|
--server URL |
Server to point the agent at (default http://127.0.0.1:1919) |
--api-key KEY |
Key of a server started with --api-key (default $FREETOKEN_API_KEY). The agent gets it in its environment; Hermes and OpenClaw keep it in their config files |
--dry-run |
Print the planned config changes and command, touch nothing |
-y, --yes |
Approve install/config prompts |
--config |
Configure without launching |
--install-only |
Just install the agent CLI (needs no server) |
--force-reinstall |
Re-run the agent installer |
-- <args> |
Forwarded verbatim to the agent |
ft checkpoint
ft checkpoint --model <hf_dir> --out <ftw_dir> [--dtype bfloat16] [--moe-backend offload] [--quant-backend moe.nvfp4=b12x] [--shard-gib 8] [--gpu <uuid-or-index>]
Converts an HF safetensors checkpoint to FTW, FreeToken's self-contained
fast-load format; point ft serve --model at the output dir. MoE experts are
always packed into expert banks, which serve the fused and the offload
strategies alike (DeepSeek-V4 experts serve offload only);
--moe-backend is kept for compatibility. See the FTW caveats in
models.md; FTW files from older builds can be repaired with
scripts/ftw_hotfix.py instead of reconverting.
ft bench bw
ft bench bw --model /path/to/model.gguf # every MoE layer kind of a checkpoint (GGUF or HF)
ft bench bw # canonical geometry per expert format
ft bench bw --gpu 1 # a specific GPU (UUID or nvidia-smi index, as for ft serve)
Measures host-RAM vs PCIe bandwidth with the real cpu/offload MoE kernels and writes one
row per MoE layer kind, which ft serve --moe-strategy hybrid --moe-hybrid-max-fetch -1 reads
to split each layer's misses.
- One profile per GPU, at
~/.cache/freetoken/benchbw/<gpu-uuid>.json; a profile from other hardware or another profile version is ignored. - A row's key is the layer's CPU format and options, expert shape, top_k and CPU worker
threads; a layer without an exact row keeps a fetch cap of 1. Bench with the same
--cpu-threadsas the server's--moe-cpu-threads. - Each run replaces the rows it benched and keeps the others.
- What to measure:
--model(checkpoint paths or presets),--dtype,--formats,--isa. --threshold(default 2.0) sets each row'srecommended: hybrid when CPU bandwidth beats PCIe by that factor.
On NVLink-C2C hosts, see models.md
for FREETOKEN_H2D_BLOCKS_PER_BANK if PCIe-gather sits well below the linear H2D ceiling.