Model-Optimizer/modelopt_recipes/ptq.md

PTQ Recipes & Schemes

This doc walks through the PTQ quantization schemes in two parts: the model-agnostic recipes under general/ptq/ (the recommended starting point for any model), and then the model-specific recipes — per-model_type folders under model_type/ plus the checkpoint-mirror models/<org>/<checkpoint>/ tier — comparing each to its general baseline and explaining why it deviates.

---

General recipes

The general recipes are model-agnostic. Each file name combines a formats + scope (what gets quantized, and to what format) with a KV-cache mode, optionally with an algorithm (calibration variant):

<formats-scope>-<kv-mode>[-<algorithm>].yaml
            nvfp4_experts_only - kv_fp8_cast

Pick one model-body scheme + one KV-cache scheme; the shipped files are the supported combinations.

---

The shipped recipes

All 26 general/ptq/ recipes (click to expand)

Recipe	Model body	KV cache	Calibration
fp8_default-kv_fp8	FP8 W8A8, all linears	FP8 (calibrated)	max
fp8_default-kv_fp8_cast	FP8 W8A8, all linears	FP8 (constant amax)	max
nvfp4_default-kv_fp8	NVFP4 W4A4, all linears	FP8 (calibrated)	max
nvfp4_default-kv_fp8_cast	NVFP4 W4A4, all linears	FP8 (constant amax)	max
nvfp4_act_headroom-kv_fp8_cast	NVFP4 W4A4, all linears	FP8 (constant amax)	nvfp4_act_headroom
nvfp4_default-kv_nvfp4_cast	NVFP4 W4A4, all linears	NVFP4 (constant amax)	max
nvfp4_default-kv_none-local_hessian	NVFP4 W4A4 (static W), all linears	none	local Hessian + FP8 sweep
nvfp4_default-kv_none-gptq	NVFP4 W4A4 (static W), all linears	none	GPTQ (layerwise)
nvfp4_mlp_only-kv_fp8	NVFP4 W4A4, MLP + MoE experts	FP8 (calibrated)	max
nvfp4_mlp_only-novit-kv_fp8	NVFP4 W4A4, MLP + MoE experts (VL vision tower excluded)	FP8 (calibrated)	max
nvfp4_mlp_only-kv_fp8_cast	NVFP4 W4A4, MLP + MoE experts	FP8 (constant amax)	max
nvfp4_mlp_only_mse-kv_fp8_cast	NVFP4 W4A4, MLP + MoE experts	FP8 (constant amax)	MSE + FP8 sweep
nvfp4_experts_only-kv_fp8	NVFP4 W4A4, MoE experts only	FP8 (calibrated)	max
nvfp4_experts_only-kv_fp8_cast	NVFP4 W4A4, MoE experts only	FP8 (constant amax)	max
nvfp4_experts_only-kv_fp8_layerwise	NVFP4 W4A4, MoE experts only	FP8 (calibrated)	max, layerwise
nvfp4_experts_only-kv_fp8_layerwise_offload	NVFP4 W4A4, MoE experts only	FP8 (calibrated)	max, layerwise (non-mutating, for disk offload)
nvfp4_experts_only-kv_fp8_layerwise_export	NVFP4 W4A4, MoE experts only	FP8 (calibrated)	max, layerwise (exports each layer as it is calibrated)
nvfp4_experts_only_mse-kv_fp8_cast	NVFP4 W4A4, MoE experts only	FP8 (constant amax)	MSE + FP8 sweep
nvfp4_experts_only_input_scale1-kv_fp8_cast	NVFP4 W4A4, MoE experts only, expert input_scale pinned to 1.0	FP8 (constant amax)	max (weights); expert activations uncalibrated
nvfp4_omlp_only-kv_fp8	NVFP4 W4A4, o_proj + MLP/MoE	FP8 (calibrated)	max
nvfp4_omlp_only-kv_fp8_cast	NVFP4 W4A4, o_proj + MLP/MoE	FP8 (constant amax)	max
nvfp4_weight_only-kv_fp16	NVFP4 W4A16, weights only	none (BF16/FP16)	max
nvfp4_weight_only-kv_fp8_cast	NVFP4 W4A16, weights only	FP8 (constant amax)	max
int4_blockwise_weight_only	INT4 W4A16, block 128, weights only	none	max
nvfp4_mlp_weight_only	NVFP4 W4A16 (block 32), MLP + MoE weights only	none	max
mxfp4_mlp_weight_only	MXFP4 W4A16, MLP + MoE weights only	none	none (no calibration)

---

Model-body schemes

The body scheme is the main lever: it trades accuracy against memory/throughput by choosing which parts of the model drop to low precision and whether activations are quantized too (W4A4/W8A8 vs weight-only W4A16).

Full-model schemes (quantize everything)

• fp8_default — per-tensor FP8 E4M3 W8A8 on every linear (attention q/k/v/o + MLP/MoE) — one scale per weight/activation tensor. The safest aggressive option: FP8 has a wide dynamic range, so accuracy loss is usually negligible. Needs Hopper+ for FP8 kernels. Good default when the target hardware is FP8-class and you want the broadest speedup.
• nvfp4_default — NVFP4 (E2M1, block-16, FP8 block scales) W4A4 on every linear. The most aggressive scheme — 4-bit weights and activations everywhere — for maximum memory/throughput on Blackwell+. Highest risk of accuracy loss; if it regresses, fall back to one of the scoped schemes below rather than abandoning NVFP4.

Scoped schemes (quantize part of the model)

• nvfp4_experts_only — NVFP4 W4A4 on MoE routed experts only (*.experts.*, *block_sparse_moe*). Dense layers, shared experts, and attention stay BF16. The most recommended NVFP4 recipe for MoE models: it's the narrowest, most accuracy-preserving NVFP4 scope, so it recovers the most accuracy — while still compressing well, because the routed experts are usually the largest share of the model's total weights.
• nvfp4_mlp_only — NVFP4 W4A4 on all MLP/FFN compute: dense MLP layers, MoE routed experts, and block_sparse_moe blocks. Attention stays BF16. Recommended for dense models: most FLOPs/params live in the MLP, so this captures most of the win while leaving the sensitive attention path untouched for accuracy.
• nvfp4_omlp_only — NVFP4 W4A4 on MLP/MoE plus the attention output projection (o_proj), but not q/k/v. A middle ground between mlp_only and default: adds the o_proj GEMM (often safe) without quantizing the more sensitive q/k/v projections.

Scope vs. compression. These schemes keep the accuracy-sensitive attention path (or the whole dense path) at the model's original precision — BF16 for most checkpoints — and quantize only the FFN/expert weights, which dominate params and compute. That's good for accuracy, but the left-out layers stay uncompressed. How much that matters is model-dependent: on MoE models the routed experts are the vast majority of the weights, so leaving attention in BF16 costs almost nothing on disk; on some dense models the attention projections are large enough that they noticeably bound the checkpoint size. If the attention weights are large and you want to compress them, we recommend adding an FP8 rule for the attention projections (keep NVFP4 on the MLP/experts) rather than leaving them BF16 — FP8 keeps that sensitive path at a safer precision than NVFP4 while still halving those weights vs. BF16.

Weight-only schemes (W4A16 — activations stay BF16)

Quantize weights only; activations run in BF16. This shrinks the model (memory-bound decode win) with much lower accuracy risk than W4A4, and needs no calibration forward pass.

These are usually recommended for low-concurrency deployments — edge and on-device/client use cases — where the workload is memory-bandwidth-bound and shrinking the weights is the main win. For high-concurrency data-center serving, prefer a scheme that also quantizes activations (the W4A4/W8A8 body schemes above): at large batch sizes the GEMMs become compute-bound, so low-bit activations and tensor-core math are what deliver the throughput.

• nvfp4_weight_only — NVFP4 weights, BF16 activations. Memory savings of 4-bit weights without the activation-quantization risk.
• int4_blockwise_weight_only — INT4 weights, block size 128, BF16 activations. Classic W4A16 weight compression; works without NVFP4-class hardware.
• nvfp4_mlp_weight_only — NVFP4 (block size 32) weights on MLP/MoE layers only, BF16 activations.
• mxfp4_mlp_weight_only — MXFP4 weights on MLP/MoE layers only, BF16 activations. Needs no calibration forward pass; the QAT starting point for the GPT-OSS family (see examples/gpt-oss).

---

KV-cache schemes

The kv_* suffix controls how the attention KV cache is quantized — independent of the body scheme. Quantizing the KV cache reduces memory at long context.

• kv_fp8_cast — FP8 KV with a constant amax (cast mode): skips KV calibration entirely. Cheaper to produce and the safe default for KV. For most models it is as accurate as the calibrated kv_fp8 below, so prefer it unless you have a specific reason to calibrate KV scales. Hopper+.

• kv_nvfp4_cast — NVFP4 KV cache with constant amax. More aggressive KV compression (4-bit); combines with any body scheme. Blackwell+.

• kv_fp8 — FP8 E4M3 KV cache with calibrated per-tensor amax. The KV scales are measured during the calibration pass. Hopper+.

• kv_fp16 — the KV cache is not quantized; it stays at the model's activation dtype (BF16/FP16). Combine with any body scheme when the deployment stack does not consume an FP8 KV cache, or when long-context memory is not the binding constraint. Some published checkpoints ship this way, so a body scheme gains a -kv_fp16 variant when one of them needs it.

kv_fp8_cast vs kv_fp8: both produce an FP8 KV cache. _cast uses a fixed scale and skips the KV calibration step (faster, no extra data dependence); plain kv_fp8 calibrates the scale from data. The cast version usually matches calibrated accuracy, so start with kv_fp8_cast.

---

Calibration variants

How the quantization scales are searched. The default (no suffix) is max.

• max (default) — amax/max calibration. Fast, one calibration pass; the baseline choice.

• mse (e.g. nvfp4_mlp_only_mse, nvfp4_experts_only_mse) — MSE search for static NVFP4 weight scales, with an FP8-scale sweep over the e4m3 scale values. The MSE search applies to the weights; activations are still max (amax) calibrated as in the default recipes. Costs more calibration time but recovers accuracy NVFP4 W4A4 can lose under plain max. Reach for it when a max recipe regresses.

• nvfp4_act_headroom (nvfp4_act_headroom-kv_fp8_cast) — leaves headroom on the NVFP4 activation global scale. Plain max anchors that scale to the largest per-block amax seen during calibration, so any larger activation at inference saturates. This variant instead sets amax = max(rho * anchor, upper) from the per-block amaxes at anchor_percentile (default 1) and upper_percentile (default 99.99), with headroom factor rho (default 16384): calibrated blocks sit low in the FP8 block-scale range, leaving the rest for unseen outliers. Costs one calibration pass and exports a standard NVFP4 checkpoint; coverage matches nvfp4_default-kv_fp8_cast, and the nested weight_scale_algorithm still accepts mse / local_hessian. Affects NVFP4 input quantizers only — a no-op for FP8 and weight-only recipes.

  Reach for it when a W4A4 recipe regresses and the activations, not the weights, are to blame: an A16 ablation of the same scope clears it while mse does not; the symptom is behavioral (verbose or runaway generations, hitting the generation cap) rather than a flat score drop; inference contexts run longer than the calibration set; or a few rare blocks dominate the activation error. MoE experts-only scopes are the common case — one observer covers every expert in a layer, and down_proj inputs clip first. On a GLM-5.3-Flash experts-only W4A4 study (SciCode, temperature 1.0) it cut the median generation-length regression from +38% to +19% and the mean from +19% to +4% with no capped generations: the best strict-W4A4 result there, but still short of the p50/p75 gate. A strong first lever, not a guaranteed fix — and sweep rho, since headroom above the calibrated range costs resolution inside it.

• local_hessian (nvfp4_default-kv_none-local_hessian) — searches static NVFP4 weight scales by minimizing output reconstruction error using a local Hessian approximation, with an FP8-scale sweep. It can recover accuracy when plain max calibration regresses, but currently supports only single-rank calibration.

• input_scale1 (nvfp4_experts_only_input_scale1-kv_fp8_cast) — pins the expert activation per-tensor amax to a constant 2688.0 (= E2M1_MAX × E4M3_MAX = 6 × 448) via constant_amax, so the exported NVFP4 input_scale is exactly 1.0 and those quantizers skip activation calibration entirely (no forward statistics collected). Weights are still max-calibrated (computed directly from the weight tensors, no data needed), and the per-block E4M3 activation scales remain dynamic. Because nothing in the recipe needs a calibration forward pass — expert activations are pinned, weight amax comes from the weights, and the KV cast uses a constant amax — it may work out of the box for very large LLMs (hundreds of billions of parameters and up), where running calibration PTQ is difficult on a resource-limited setup. Also reach for it when the deployment stack expects a unit expert input_scale (e.g. NVFP4 expert kernels that assume input_scale == 1.0) or to take expert activation calibration out of the picture.

• gptq (nvfp4_default-kv_none-gptq) — GPTQ layerwise calibration of the weight scales; writes layerwise checkpoints. GPTQ is best established for INT4 weight-only quantization; its effectiveness on NVFP4 weight quantization varies model by model — it tends to help most when the other recipes show a larger accuracy loss. Applying GPTQ to MoE models is still an open research topic and needs extra recipe tuning.

• layerwise (nvfp4_experts_only-kv_fp8_layerwise) — max calibration done one decoder layer at a time to lower peak memory; same numerics as the non-layerwise variant.

• layerwise export (nvfp4_experts_only-kv_fp8_layerwise_export) — the same calibration, additionally writing each decoder layer to the export checkpoint as soon as it is calibrated. A run interrupted part-way resumes without redoing finished layers, and no separate export pass is needed. Same numerics again.

These can also be stacked when a single method isn't enough — e.g. mse + gptq combines an MSE-searched weight scale with GPTQ's layerwise update.

---

Choosing a general recipe

1. Match the format to your hardware/target. FP8 (fp8_default) on Hopper+; NVFP4 (nvfp4_*) on Blackwell+; weight-only (*_weight_only) when you want compression with minimal risk or lack NVFP4-class kernels.
2. Start from the most accurate scope, then quantize more toward your memory/performance target. For low-concurrency deployments (edge, on-device/client), start from a weight-only recipe (nvfp4_weight_only / int4_blockwise_weight_only) — shrinking weights is the main win there. For higher-concurrency serving, begin with the narrowest activation-quantized scope — nvfp4_experts_only for MoE, nvfp4_mlp_only for dense — then widen (mlp_only → omlp_only → default) only as far as your memory/throughput target requires, checking accuracy as you go.
3. Recover accuracy via calibration before backing off the scope. If a wider-scope recipe regresses, switch its max to the mse variant before retreating to a narrower scope. If mse doesn't clear it but an A16 ablation does, the activations are the problem: try nvfp4_act_headroom next.
4. Pick KV by deployment. kv_fp8_cast is the safe default (usually as accurate as calibrated kv_fp8); use kv_nvfp4_cast for maximum KV compression.

Beyond PTQ: these recipes' quantize sections are also reused as the quantization config for QAT/QAD training flows. Dedicated scale-learning (LSQ / Dual-LSQ) QAD recipes live separately under general/qad/.

---

Model-specific recipes

The general recipes above are model-agnostic: they select layers by wildcard (*mlp*, *self_attn*, *[kv]_bmm_quantizer) and lean on the shared default_disabled_quantizers exclusions, so the same file works on any architecture whose module names follow the usual conventions. A recipe only earns a body under model_type/<model_type>/ or models/<org>/<checkpoint>/ when a model has to deviate from that baseline. The deviations come in four kinds:

ℹ️ model_type/ was previously named huggingface/; old huggingface/<model_type>/... --recipe paths still resolve for backward compatibility, but use model_type/ going forward.

Kind	What changes vs. the general recipe	Examples
Architecture-aware quant_cfg	Per-sub-module format choices a single wildcard scheme can't express	minimax_m3_vl, qwen3_vl, qwen3_5, qwen3_5_moe, qwen3_6_moe, vit, nemotron_llama
Algorithm override	Same numerics & scope, but the calibration algorithm is tweaked because the default breaks or regresses	gemma, gemma4, mpt
Extra exclusions	Adds disabled-quantizer patterns so non-language branches stay full precision	nemotron_vl, diffusion_gemma
Checkpoint mirror	A mixed-precision map reproducing one published checkpoint exactly	models/nvidia/NVIDIA-Nemotron-3-*, models/mistralai/Mistral-Medium-3.5-128B

The numerics and standard exclusions are still inherited from configs/ wherever possible — the model folder captures only the delta. Each <task>/ folder may carry a README.md spelling out that delta.

Not every models/<org>/<checkpoint>/ folder holds a deviation. When a general (or model_type/<model_type>) recipe already produces a released checkpoint's scheme, the folder holds a thin alias that imports that recipe wholesale and overrides only metadata — no duplicated quant_cfg — so the recipe is reachable from the checkpoint's own hub path. Checkpoint mirrors covers the folders that genuinely deviate and aliases the ones that don't; see models/README.md for the alias format and when to write one.

Architecture-aware quant_cfg — minimax_m3_vl, qwen3_vl, qwen3_5, qwen3_5_moe, qwen3_6_moe, vit, nemotron_llama

minimax_m3_vl/ptq/mxfp8_nvfp4_experts applies MXFP8 to the language-model linear layers and MSE-calibrated NVFP4 to routed experts, with expert input_scale fixed to 1.0. The vision branch, routers, lm_head, and KV cache remain unquantized.

qwen3_vl/ptq and qwen3_5/ptq provide FP8 recipes for the visual branch of validated Qwen3-VL and dense Qwen3.5 checkpoints. fp8_vision-kv_none enables only Vision Encoder nn.Linear weights and inputs, including the primary merger and any deepstack mergers. fp8_vision_lm-kv_fp8_cast combines that visual configuration with the standard W8A8 FP8 model and FP8 KV-cache-cast units. Both keep patch embedding and vision-attention BMM operands in high precision. The shared visual snippet lives under qwen3_vl; thin wrappers remain discoverable under each exact Hugging Face model_type.

model_type/qwen3_5/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast (and its MoE twin, which shares the same quant_cfg snippet) is a mixed scheme no single general body covers: NVFP4 W4A16 on MLP / expert projection weights and lm_head, FP8 on self-attention and the large linear-attention projections (in_proj_qkv, in_proj_z, out_proj), plus FP8 KV cast. It also disables architecture-specific submodules that aren't in the reference recipe (linear_attn.in_proj_a/b, conv1d, and any visual/mtp siblings).

Why special: these are hybrid linear-attention + softmax-attention models. A general scheme would apply one format per wildcard class; this architecture needs FP8 for the big linear-attention projections but NVFP4 for MLP weights, and needs the linear-attention conv/gate submodules left alone. The dense and MoE families share the identical wildcard rules, so one snippet drives both.

On Qwen3.5 / Qwen3.6 this W4A16 recipe usually does not regress accuracy versus the official checkpoint, and it is designed for best performance in low-concurrency use cases (weight-only on the MLP keeps the memory-bound decode path fast without quantizing activations). Both the dense and MoE folders also ship an MSE twin, w4a16_nvfp4_mse-fp8_attn-kv_fp8_cast — the identical layout with the NVFP4 weight scales chosen by an MSE FP8-scale sweep instead of max calibration (the mse variant applied to this architecture-specific scheme) — for when the max-calibrated recipe regresses.

vit/ptq/fp8 covers ViT image classifiers (the FP8 + Torch-TRT example): FP8 W8A8 on every linear, like fp8_default, but it additionally enables the attention BMM quantizers — q/k/v BMM plus the softmax-P quantizer — in FP8 and disables the output quantizers. Why special: the general LLM recipes never quantize the attention BMM inputs; for ViT the whole attention block runs in FP8 so Torch-TRT can compile it end-to-end.

nemotron_llama/ptq/{nvfp4,fp8}_output_quant_proj covers the Llama-Nemotron embedding/reranking encoders (e.g. llama-nemotron-embed-1b-v2): same numerics as the general nvfp4/fp8 presets, plus output quantizers on the projection Linears (*_proj.output_quantizer) so TensorRT engines carry inter-layer activations in the low-precision format instead of FP16 — roughly half the engine activation memory on these models. The sequence-classification score head stays unquantized, like lm_head. In the NVFP4 recipe, its [1, hidden] weight cannot be packed by the NVFP4 exporter. Why special: the general recipes never enable output quantizers, and the pattern must stay scoped to GEMM outputs — a DynamicQuantize on non-GEMM outputs (embedding lookup, pooling) fails to compile in TensorRT.

A lighter case: models/stepfun-ai/Step-3.5-Flash/ptq/nvfp4-mlp-only is close to general/ptq/nvfp4_mlp_only (NVFP4 on MoE/MLP weights+inputs, FP8 KV) but pinned to one released checkpoint and carrying instance-specific disables (share_expert, moe.gate, the conv1d branches).

step3p7/ptq/{nvfp4_experts_only-kv_fp8_cast,nvfp4_mlp_only-kv_fp8} are the Step-3.7 equivalents, and the reason they exist is module naming: Step calls the MoE block moe and the dense sibling share_expert, so the general recipes' *.experts.*, *block_sparse_moe* and *mlp* patterns match nothing on the routed experts — the general recipe would quantize nothing and export a checkpoint with quant_algo: null. These select *moe* instead and disable the router (moe.gate) and share_expert on top. Use them, not the general recipes, for Step-3.7 checkpoints; Step-3.5 has its own recipe above.

• qwen3_6_moe/ptq/w4a4_nvfp4-fp8_attn-kv_fp8_cast_mcore is the Megatron-Core W4A4 counterpart of qwen3_5_moe's W4A16 recipe, used as the QAD student in the Qwen3.6-35B-A3B tutorial. MoE routed + shared experts and lm_head → NVFP4 W4A4 (block 16); softmax attention and the GatedDeltaNet in_proj/out_proj → FP8 W8A8; KV cache → FP8 cast; MTP block, routers, conv1d, the vision tower and embeddings stay BF16. W4A4 rather than W4A16 because a BF16 activation keeps vLLM on the Marlin dequant fallback, which measured slower than BF16. The _mcore suffix is load-bearing: selectors are Megatron-Core leaf names (mlp.experts.linear_fc1, self_attention.linear_qkv), so under hf_ptq.py nothing matches and mtq.quantize raises.

Algorithm overrides — gemma, gemma4, mpt

These quantize the same layers as the general recipes; only the quantize.algorithm block differs, to work around model-specific numerics:

• gemma/ptq/w4a8_awq-kv_fp8_cast (INT4 block weights + FP8 inputs + FP8 KV cast), its multimodal sibling gemma4/ptq/w4a8_awq-kv_fp8_cast (the Gemma 4 gemma4 model type, whose vision branch stays BF16 via the standard exclusions), and mpt/ptq/w4a8_awq-kv_fp8_cast use awq_lite with alpha_step: 1 instead of the default AWQ search. The default search overflows the TRT-LLM kernels on these models; the coarser sweep avoids it without measurably hurting accuracy.
• gemma/ptq/int8_sq-kv_fp8_cast (INT8 per-channel weights + INT8 inputs + FP8 KV cast) sets SmoothQuant alpha: 0.5 instead of the default 1.0 — Gemma 7B regresses at 1.0, and 0.5 recovers it.

Why special: identical scope/numerics to a general scheme, but a general recipe's default algorithm would overflow or regress here.

Extra exclusions — nemotron_vl, diffusion_gemma

Each of these is numerically identical to a general recipe. What makes them special is a model-local disabled_quantizers.yaml unit that extends the standard exclusions so a model-specific branch stays in full precision:

• nemotron_vl (vision-language, incl. Nemotron-Parse) — general nvfp4_default-kv_fp8_cast numerics, adding *vision*, *image*, *radio*, *visual*, *encoder*, *model_encoder* so only the language decoder is quantized.
• diffusion_gemma (block-diffusion encoder-decoder text LLM on a Gemma4 MoE backbone) — general nvfp4_experts_only-kv_fp8_cast numerics, adding *self_conditioning*: the self-conditioning network is text-only and never exercised by standard PTQ calibration data, so its quantizers collect no amax and export crashes; the exclusion keeps it in BF16.

Why special: a general recipe would happily quantize the vision/audio encoders (or the never-calibrated self-conditioning branch), regressing those modalities or crashing export. The extra patterns keep them in full precision; everything else matches the general recipe.

Checkpoint mirrors — models/<org>/<checkpoint>

The models/ tier reproduces a single published (or planned) checkpoint's quant config verbatim:

• models/moonshotai/Kimi-K3/ptq/nvfp4_experts-fp8_pb_attention mirrors nvidia/Kimi-K3-NVFP4: the source MXFP4 routed experts are cast to NVFP4, with activation input_scale=1.0, while KDA and MLA projection weights use 128x128 block FP8. Attention activations are dynamic; shared and latent experts, routers, convolutions, norms, the vision tower, lm_head, and KV cache remain BF16. Because the 2.8T source uses packed MXFP4 expert tensors, use the calibration-free streaming converter in examples/kimi/ rather than the in-memory hf_ptq.py flow.

• models/deepseek-ai/DeepSeek-V4-Pro-0813/ptq/nvfp4_experts_only mirrors nvidia/DeepSeek-V4-Pro-0813-NVFP4: the source MXFP4 routed experts are cast to NVFP4 (weights and activations, block 16), while shared experts, attention, router gates, lm_head and the MTP/DSpark speculative-decoding block stay in their source format. Because the 1.65T source ships its experts packed as MXFP4 in DeepSeek's native layout, calibration runs through examples/deepseek/deepseek_v4/ptq.py and the cast through quantize_to_nvfp4.py --cast_mxfp4_to_nvfp4, rather than the in-memory hf_ptq.py flow.

• models/mistralai/Mistral-Medium-3.5-128B/ptq/nvfp4-max-calib mirrors nvidia/Mistral-Medium-3.5-128B-NVFP4: decoder MLP layers 4–86 use NVFP4 W4A4, edge MLP layers 0–3 and 87 use FP8 W8A8, and all attention projections and the KV cache use FP8. It uses max calibration.

• models/MiniMaxAI/MiniMax-M2.7/ptq/nvfp4_experts_only-kv_fp8_cast reproduces nvidia/MiniMax-M2.7-NVFP4 with max-calibrated NVFP4 W4A4 experts (block size 16) and FP8 KV-cache cast mode; attention projections, router, and LM head stay BF16.

• models/Qwen/Qwen3.8-27B/ptq/nvfp4_w4a4_mlp_fp8_attn_local_hessian reproduces NVFP4 W4A4 mixed precision quantization recipe used for nvidia/Qwen3.8-27B-NVFP4: MLP projections and lm_head use NVFP4 W4A4, self-attention and linear-attention projections use FP8 W8A8. Static NVFP4 weight scales are calibrated with local-Hessian algorithm.

• models/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/ptq/nvfp4-mse mirrors nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 exactly — a hybrid Mamba-MoE with a hand-mapped, per-component precision scheme:

  • MoE routed experts → NVFP4 W4A4, group_size 16, static weight scales
  • shared experts and Mamba in/out_proj → FP8 per-tensor
  • KV cache → FP8
  • attention q/k/v, MTP head, lm_head, latent-MoE, Mamba conv1d → BF16

  nvfp4-mse.yaml uses MSE calibration with an FP8-scale sweep (matches the release); nvfp4-max-calib.yaml is the identical layer map under plain max calibration, kept for comparison.

• models/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16/ptq/nvfp4-4o6 follows the same Super-style component map (routed experts NVFP4 W4A4 block-16; shared experts + Mamba in/out_proj + KV cache FP8; everything else BF16), but the routed-expert weights use Four-over-Six (4/6) NVFP4: an MSE search picks each weight's amax multiplier from [1.0, 1.5] (M=6 vs. M=4). Activations stay dynamic NVFP4 (not MSE-calibrated).

• models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/ptq/w4a16_nvfp4_4o6 applies Four-over-Six NVFP4 W4A16 to routed experts, shared experts, and the language model head; Mamba in/out_proj weights and inputs plus the KV cache use FP8, while attention remains BF16.

• models/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16/ptq/nvfp4_w4a16 mirrors the GGUF Q4_K_M bit allocation of the Nemotron-H hybrid, mapped onto NVFP4/FP8 per layer: Q4_K/Q5_0 linears → NVFP4 W4A4 (attention q/k/v/o kept uniform so export can fuse them), the Q6_K MLP down_proj layers → FP8 W8A8, embeddings → NVFP4 W4A16, lm_head → FP8 W8A16, and the F32 tensors (conv1d, norms) → BF16.

• models/Qwen/Qwen3.8-2.4T-A95B/ptq/nvfp4_experts_mse-fp8_self_attn-fp8_linear_attn-kv_fp8_cast mirrors nvidia/Qwen3.8-2.4T-A95B-NVFP4: a qwen3_5_moe_text MoE with hybrid attention — gated-delta (linear-attention) layers interleaved with full-attention layers. Routed experts → NVFP4 (MSE-searched static weight scales, dynamic input scales); both self-attention and the full gated-delta linear-attention path (conv1d, in_proj_qkv/z/a/b, out_proj) → FP8 W8A8; KV cache → FP8 cast; everything else, including the MTP block, stays BF16. The gated-delta norms stay BF16, but the conv1d is FP8 like the projections: nn.Conv1d is a registered quant module, so the recipe's broad *linear_attn* wildcards reach linear_attn.conv1d too — matching the published checkpoint, whose hf_quant_config.json lists linear_attn.conv1d as FP8 on every gated-delta layer (the full-attention layers have no conv1d). The source ships as native block-FP8 (weight_block_size [128, 128]); the loader dequantizes it to BF16 before quantizers are inserted, so the scales are calibrated against BF16 weights, not the shipped FP8.

• models/zai-org/GLM-5.3-Flash/ptq/nvfp4_experts_dense_mlp-kv_fp8_cast is the NVFP4 config for zai-org/GLM-5.3-Flash, a glm5_next VLM MoE with hybrid attention — KDA (linear-attention) layers interleaved with NoPE sparse-MLA layers. Routed experts and the dense MLP → NVFP4 W4A4; KV cache → FP8 cast; everything else stays BF16 (shared experts, router gate, both attention families, the vision tower, embeddings and lm_head). mlp_layer_types marks only layers 0-2 dense, so the dense-MLP scope adds just 9 modules (mlp.gate_proj / up_proj / down_proj) on top of the routed experts. The vision tower reuses those same leaf names, so a single *visual* disable is appended last to keep model.visual.* in BF16. It pins layerwise.enable=false, which this VLM requires because its decoder layers nest under model.language_model.layers. The MTP layer is not built by the HF class at num_hidden_layers: 45, so it is neither quantized nor exported. (For plain experts-only NVFP4 on this model, use the general general/ptq/nvfp4_experts_only-kv_fp8_cast — the model-specific delta here is the dense-MLP scope plus the vision-tower exclusion.)

Why special: unlike any general recipe, each is pinned to one checkpoint and captures a model-specific deviation a portable general recipe can't express. Most mix FP8 and NVFP4 across different component types — or individual layers — and hardcode the precise published layout (for Super, matched on both HF and Megatron-Core module names) rather than a portable wildcard scheme. GLM-5.3-Flash is the exception: its deviation is a model-specific scope — a wildcard scheme plus a load-bearing vision-tower exclusion and the VLM-required layerwise.enable=false — rather than a per-component precision map.

Checkpoint aliases — models/<org>/<checkpoint>

Some published checkpoints use a scheme a portable recipe already produces, so their entry is a thin alias that imports that recipe wholesale and overrides only metadata (see models/README.md for the format):

• models/moonshotai/Kimi-K2.6/ptq/nvfp4_experts_only_mse-kv_fp8_cast aliases the general general/ptq/nvfp4_experts_only_mse-kv_fp8_cast — expert-only NVFP4 with MSE-swept static weight scales and dynamic inputs, plus an FP8 KV cache in cast mode — as published in nvidia/Kimi-K2.6-NVFP4.
• models/Qwen/Qwen3.5-397B-A17B/ptq/nvfp4_experts_mse-fp8_rest-kv_fp8 aliases the qwen3_5_moe architecture recipe model_type/qwen3_5_moe/ptq/nvfp4_experts_mse-fp8_rest-kv_fp8 — NVFP4 (MSE static weights) on the routed experts, ModelOpt-default FP8 elsewhere, and an FP8 KV cache — as published in nvidia/Qwen3.5-397B-A17B-NVFP4-V2.

---

For the full catalog and how to pick a starting recipe for a given model, see README.md.