See https://github.com/NVIDIA/Model-Optimizer/pull/781
This is the MR that only includes the refactoring of the llm export,
please ignore the change on quantize.py from the diffusion example.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added `--hf-ckpt-dir` CLI option to save checkpoints in HuggingFace
format
* Enabled support for exporting Diffusers-based pipelines
* Unified export system now handles both transformer and diffusion model
architectures
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Jingyu Xin <jingyux@nvidia.com>
- [x] Product Rename: TensorRT Model Optimizer to Model Optimizer
(OMNIML-3033)
- [x] Mention in Latest News section with date on the date of merging
this PR (12/08)
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
## What does this PR do?
**Type of change:**
new feature
**Overview:** ?
Attention mechanism has changed from diffusers 0.35.
Many model attentions are now subclass of a new Mixin class:
AttentionModuleMixin, which is not a sub class of Attention
To fix it, patch the mixin class by forcing to use native attention
impl so the existing function monkey patch still work.
## Testing
manual quant of Wan, Flux
---------
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
## What does this PR do?
**Type of change:** ?
Bug fix
**Overview:** ?
1. We are passing incorrect extra args to pipeline inference.
2. Need a default empty list for the list argument component-dtype
## Testing
Passed SDXL pipeline without component dtype
---------
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
## What does this PR do?
**Type of change:**
new feature
**Overview:**
Support Wan2.2 t2v diffusers quantization
1. fix torch2.9 support
2. add Wan2.2 t2v diffusers pipeline quantization
Main difference of the Wan2.2 pipeline comparing to exisiting
pipelines is that there are 2 backbone models for denoising. For the
quantization therefore we need to quantize both of them.
However, it turns out our base library does not well support
quantization of multiple models in the same time. Therefore, the
change here just stick to quantize a single model each time, and then
run the quantization multiple times.
So, we need to allow users to pick which backbone to quantize,
therefore adding a new argment for it
3. add a workaround for the exporting ONNX issue when we upgrade
diffusers to >= 0.35.0. The issue lies is the exporting of the
torch.nn.RMSNorm. Some pipelines in the diffusers > 0.35.0 use the
torch version RMSNorm while before that they use the diffusers' own
version of RMSNorm. It turns out they are directly replacable so the
workaround is to simply replace the torch RMSNorm usages with
diffusers RMSNorm. But we need to fix it properly soon by porting our
ONNX export to be based on torch dynamo instead of torchscript. Issue
reported from external user:
https://github.com/NVIDIA/TensorRT-Model-Optimizer/issues/262
4. allow use of a prompts file, which is simply a text file with a list
of prompts, one prompt each line
5. allow each component of a pipeline to have different dtype accuracy.
added a new list stype command line arg --component-dtype for this.
example: --component-dtype vae:Float
6. print the summary of the quantized model so users can capture issues
from
log
## Usage
python quantize.py \
--model wan2.2-t2v-14b \
--format fp8 \
--batch-size 4 \
--calib-size 64 \
--n-steps 20 \
--backbone transformer \
--model-dtype BFloat16 \
--component-dtype vae:Float \
--trt-high-precision-dtype BFloat16 \
--quantized-torch-ckpt-save-path ./wan_transformer.pt \
--onnx-dir wan-transformer-onnx \
--prompts-file wan-prompts.txt
## Testing
Tested SDXL_BASE, LTX_VIDEO_DEV, WAN22_T2V
## Before your PR is "*Ready for review*"
<!-- If you haven't finished some of the above items you can still open
`Draft` PR. -->
- **Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CONTRIBUTING.md)**
and your commits are signed.
- **Is this change backward compatible?**: Yes/No <!--- If No, explain
why. -->
- **Did you write any new necessary tests?**: Yes/No
- **Did you add or update any necessary documentation?**: Yes/No
- **Did you update
[Changelog](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CHANGELOG.rst)?**:
Yes/No <!--- Only for new features, API changes, critical bug fixes or
bw breaking changes. -->
## Additional Information
https://github.com/NVIDIA/TensorRT-Model-Optimizer/issues/262
---------
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
jingyu-ml