mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-19 05:49:19 +08:00
Jenny Chen
f70991f36e
### What does this PR do? Type of change: Documentation Add detailed quantization aware training (QAT and QAD) guide in our docs, featuring examples on how to run in HF, Megatron-Bridge, and Megatron-LM ### Usage ```python # Add a code snippet demonstrating how to use this ``` ### Testing <!-- Mention how have you tested your change if applicable. --> ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ / ❌ / N/A <!--- If ❌, explain why. --> - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ / ❌ / N/A <!--- Mandatory --> - Did you write any new necessary tests?: ✅ / ❌ / N/A <!--- Mandatory for new features or examples. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ / ❌ / N/A <!--- Very short summary of changes only for new features, backward breaking changes, deprecations, or fixes for critical bugs present in previous releases. --> - Did you get Claude approval on this PR?: ✅ / ❌ / N/A <!--- Run `/claude review`. NVIDIA org members can self-trigger for complex changes; orthogonal to CodeRabbit. --> ### Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added a comprehensive guide for quantization-aware training (QAT) and distillation (QAD), including workflows, framework guidance, setup, training, and export steps. * Updated the Guides navigation to include the new QAT/QAD guide. * Expanded quantization guidance with QAT/QAD workflows, scale handling, and NVFP4 references. * Improved save and restore guidance with clearer references and an explanatory note. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Jennifer Chen <jennifchen@nvidia.com>
68 lines
3.9 KiB
ReStructuredText
68 lines
3.9 KiB
ReStructuredText
Basic Concepts
|
|
==============
|
|
|
|
A quantization format consists of the precision format, the block format, and the calibration
|
|
algorithm.
|
|
The detailed list of available quantization formats can be found in :ref:`quantization-formats`.
|
|
Below we provide an overview of the important topics:
|
|
|
|
Precision format
|
|
****************
|
|
The precision format defines the bit-width of the quantized values. Generally, there are integer
|
|
formats (sign bit + mantissa bits) and floating-point formats (sign bit + exponent bits + mantissa
|
|
bits). `Fp8 Formats for Deep Learning <https://arxiv.org/pdf/2209.05433>`_ provides a detailed
|
|
explanation of the floating-point formats.
|
|
|
|
Scaling factor
|
|
**************
|
|
The scaling factor is a floating-point value that is used to scale and unscale the values before and
|
|
after the quantized operation, respectively. The scaling factor is used to map the range of the
|
|
original values to the range of the quantized values. The scaling factor is shared across the
|
|
quantized values in the same block. The scaling factor is calculated during the calibration process.
|
|
|
|
Block format
|
|
************
|
|
The block format defines the way the tensor is divided into blocks for sharing the scaling factors.
|
|
The most common block format is per-tensor quantization, where the whole tensor is quantized as a
|
|
single block with one global scaling factor. Other block formats include per-channel quantization,
|
|
where each channel is quantized separately, and the fine-grained per-block quantization, where the
|
|
tensor is divided into fix-size blocks along the channel dimension. For low-bit quantization (e.g.
|
|
4-bit), per-block quantization is typically needed to preserve the accuracy.
|
|
|
|
Weight and activation may share different precision and block formats. For example, in GPTQ and AWQ,
|
|
the weight is quantized to 4-bit while activation stays in high precision. Weight-only quantization
|
|
is helpful for bandwidth-constrained scenarios, while weight and activation quantization can reduce
|
|
both bandwidth and computation cost.
|
|
|
|
Calibration algorithm
|
|
*********************
|
|
|
|
The calibration algorithm calculate scaling factors and potentially adjust weights to maximize
|
|
accuracy post quantization. The simplest calibration algorithm is "max calibration", in which the
|
|
scaling factor is calculated from the global maximum of the tensor and the weights are unchanged and
|
|
rounded to the nearest quantized value. An example of a more advanced calibration algorithm is
|
|
`Entropy Calibration <https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/python-api/infer/Int8/EntropyCalibrator.html>`_,
|
|
`SmoothQuant <https://arxiv.org/abs/2211.10438>`_, `AWQ <https://arxiv.org/abs/2306.00978>`_, and
|
|
`SVDQuant <https://arxiv.org/pdf/2411.05007>`_.
|
|
|
|
Quantization-aware training (QAT) and Quantization-aware distillation (QAD)
|
|
***************************************************************************
|
|
QAT and QAD both start from PTQ and update the underlying unquantized weights while modeling the
|
|
effect of quantization in the forward and backward passes. QAT uses the normal supervised loss,
|
|
such as cross-entropy against labeled data. QAD instead uses a frozen BF16 teacher to guide the
|
|
quantized student with a distillation loss. Commonly used QAT techniques like `Straight-Through
|
|
Estimator (STE) <https://arxiv.org/abs/1308.3432>`_ or STE with clipping have fixed scaling factors
|
|
and tune the weights during training to minimize the loss. ModelOpt implements STE with clipping for
|
|
QAT and QAD. See :ref:`quantization-aware-training` for more details.
|
|
|
|
|
|
More Readings
|
|
*************
|
|
|
|
* Math behind quantization: `Integer Quantization <https://arxiv.org/pdf/2004.09602.pdf>`_
|
|
|
|
* Explicit quantization graph representation with QDQ node:
|
|
`work-with-qat-networks <https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#work-with-qat-networks>`_
|
|
|
|
* `Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery <https://arxiv.org/abs/2601.20088>`_
|