mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-26 06:54:04 +08:00
vishalpandya1990
4a848c4f52
## What does this PR do? Documentation **Overview:** - Update support matrix, changelog, deployment page, example readmes as per recent feature and model support on Windows side. ## Testing - No testing, its just documentation change ## Before your PR is "*Ready for review*" <!-- If you haven't finished some of the above items you can still open `Draft` PR. --> - **Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)** and your commits are signed. - **Is this change backward compatible?**: Yes/No <!--- If No, explain why. --> - **Did you write any new necessary tests?**: Yes/No - **Did you add or update any necessary documentation?**: Yes/No - **Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?**: Yes/No <!--- Only for new features, API changes, critical bug fixes or bw breaking changes. --> ## Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added ONNX Mixed Precision Weight-only quantization (INT4/INT8) support. * Introduced diffusion-model quantization on Windows. * Added new accuracy benchmarks (Perplexity and KL-Divergence). * Expanded deployment with multiple ONNX Runtime Execution Providers (CUDA, DirectML, TensorRT-RTX). * **Bug Fixes** * Fixed ONNX 1.19 compatibility issue with CuPy during INT4 AWQ quantization. * **Documentation** * Updated installation guides with system requirements and multiple backend options. * Reorganized deployment documentation with comprehensive execution provider guidance. * Expanded example workflows with improved setup instructions and support matrices. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: vipandya <vipandya@nvidia.com>
104 lines
5.3 KiB
ReStructuredText
104 lines
5.3 KiB
ReStructuredText
|
|
====
|
|
FAQs
|
|
====
|
|
|
|
.. _FAQ_ModelOpt_Windows:
|
|
|
|
ModelOpt-Windows
|
|
================
|
|
|
|
**ONNX PTQ**
|
|
|
|
1. Why is awq-scale search taking too long or stuck at 0% during ONNX INT4 quantization?
|
|
----------------------------------------------------------------------------------------
|
|
|
|
Awq-scale search should complete in minutes with NVIDIA GPU acceleration. If stalled:
|
|
|
|
- **GPU acceleration may be disabled.** If CUDA 12.x is not available, quantization will fall back to slower ``numpy`` implementation instead of ``cupy-cuda12x``. Make sure that ``cupy`` package is compatible with installed CUDA toolkit.
|
|
- **Low GPU memory.** Quantization needs 20-24GB VRAM; low memory forces slower shared memory usage.
|
|
- **Using CPU for quantization.** Install ORT-GPU (supports CUDA EP) or ORT-DML (supports DML EP) for better speed.
|
|
|
|
2. Why is "CUDA EP not found" error showing during ONNX quantization?
|
|
---------------------------------------------------------------------
|
|
|
|
ORT used in GenAI may conflict with ModelOpt-Windows ORT:
|
|
|
|
- Uninstall ORT, run ``pip cache purge``, and reinstall *nvidia-modelopt[onnx]*.
|
|
- Use separate virtual environments for GenAI and quantization (e.g., with *venv* or *conda*).
|
|
|
|
3. Why does ORT-session creation fail for CUDA-EP despite having CUDA toolkit and cuDNN?
|
|
----------------------------------------------------------------------------------------
|
|
|
|
This usually results from mismatched CUDA and cuDNN versions or missing paths. Ensure:
|
|
|
|
- Compatible CUDA toolkit and cuDNN versions (check `CUDA EP requirements <https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements/>`_).
|
|
- Add cuDNN's *bin* and *lib* paths to *PATH* and restart the command prompt.
|
|
|
|
4. Why quantized model's size increases on re-runs?
|
|
---------------------------------------------------
|
|
|
|
Make sure that the output directory is clean before each quantization run otherwise, existing quantized model file may get appended in each run leading to increase in model's size and possibly corrupting it.
|
|
|
|
5. Running INT4 quantized ONNX model on DirectML backend gives following error. What can be the issue?
|
|
------------------------------------------------------------------------------------------------------
|
|
|
|
`Error Unrecognized attribute: block_size for operator DequantizeLinear`
|
|
|
|
ModelOpt-Windows uses ONNX's `DequantizeLinear <https://onnx.ai/onnx/operators/onnx__DequantizeLinear.html>`_ (DQ) nodes. The int4 data-type support in DeQuantizeLinear node came in opset-21. And, *block_size* attribute was added in DequantizeLinear node in Opset-21. Make sure that quantized model's opset version is 21 or higher. Refer :ref:`Apply_ONNX_PTQ` for details.
|
|
|
|
6. Running INT4 quantized ONNX model on DirectML backend gives following kind of error. What can be the issue?
|
|
--------------------------------------------------------------------------------------------------------------
|
|
|
|
`Error: Type 'tensor(int4)' of input parameter (onnx::MatMul_6508_i4) of operator (DequantizeLinear) in node (onnx::MatMul_6508_DequantizeLinear) is invalid.`
|
|
|
|
One possible reason for above error is that INT4 quantized ONNX model's opset version (default or onnx domain) is less than 21. Ensure the INT4 quantized model's opset version is 21 or higher since INT4 data-type support in DequantizeLinear ONNX node came in opset-21.
|
|
|
|
7. Running 8-bit quantized ONNX model with ORT-DML gives onnxruntime error about using 8-bit data-type (e.g. INT8/FP8). What can be the issue?
|
|
-----------------------------------------------------------------------------------------------------------------------------------------------
|
|
|
|
Currently, DirectML backend (ORT-DML) doesn't support 8-bit precision. So, it expectedly complains about 8-bit data-type. Try using ORT-CUDA or other 8-bit compatible backend.
|
|
|
|
8. How to resolve onnxruntime error about invalid use of FP8 type in QuantizeLinear / DequantizeLinear node?
|
|
-------------------------------------------------------------------------------------------------------------
|
|
|
|
The FP8 type support in QuantizeLinear / DeQuantizeLinear node came with Opset-19. So, ensure that opset of ONNX model is 19+.
|
|
|
|
.. _nas_faqs:
|
|
|
|
NAS/Pruning
|
|
===========
|
|
|
|
|
|
Parallel search
|
|
---------------
|
|
|
|
If your ``score_func`` in :meth:`mtn.search() <modelopt.torch.nas.algorithms.search>`
|
|
supports parallel evaluation, you can make use of it by passing in ``DistributedDataParallel``
|
|
module to search.
|
|
|
|
|
|
Monkey-patched functions
|
|
------------------------
|
|
|
|
During the conversion process (:meth:`mtn.convert() <modelopt.torch.nas.conversion.convert>`), we
|
|
use a `monkey patch <https://en.wikipedia.org/wiki/Monkey_patch>`_ to augment the ``forward()``,
|
|
``eval()``, and ``train()`` methods of ``nn.Module``. This renders the ModelOpt conversion process
|
|
incompatible with other monkey patches to those methods.
|
|
|
|
.. code-block:: python
|
|
|
|
# Internally in mtn.convert, we do:
|
|
model.forward = types.MethodType(nas_forward_func, model)
|
|
model.train = types.MethodType(nas_train_func, model)
|
|
|
|
Known Issues
|
|
============
|
|
|
|
1. Potential memory leak for ``FSDP`` with ``use_orig_params=True``
|
|
-------------------------------------------------------------------
|
|
|
|
When using ``FSDP`` with ``use_orig_params=True``, there is a potential memory leak during training
|
|
when using ``FSDP`` in conjunction with modelopt-converted models. Please use
|
|
``use_orig_params=False`` to avoid this issue.
|