## What does this PR do?
bug fix
**Overview:**
[NVBug 5659126] dummy inputs are kwargs maps, and rename the
generate_... function to more explicitly reflect the return value type
## Testing
`python diffusion_trt.py --model flux-dev --override-model-path
/models/FLUX.1-dev --torch --benchmark --skip-image
`
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
- [x] Product Rename: TensorRT Model Optimizer to Model Optimizer
(OMNIML-3033)
- [x] Mention in Latest News section with date on the date of merging
this PR (12/08)
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
## What does this PR do?
**Type of change:**
New feature
**Overview:**
- Implemented the NVFP4QuantExporter
- Deprecated fp4qdq_to_2dq
- Updated tests
## Usage
```python
python torch_quant_to_onnx.py --quantize_mode=nvfp4 \
--onnx_save_path=vit_base_patch16_224.nvfp4.onnx \
--calibration_data_size 64 \
--batch_size 128
```
## Testing
<!-- Mention how have you tested your change if applicable. -->
```
python evaluate.py --onnx_path=vit_base_patch16_224.nvfp4.onnx \
--model_name=vit_base_patch16_224 \
--results_path=./results.txt \
--batch_size 128
```
Results:
```
The top1 accuracy of the model is 84.39%
The top5 accuracy of the model is 97.312%
Inference latency of the model is 7.22412 ms
```
## Before your PR is "*Ready for review*"
<!-- If you haven't finished some of the above items you can still open
`Draft` PR. -->
- **Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CONTRIBUTING.md)**
and your commits are signed.
- **Is this change backward compatible?**: No
- Deprecated fp4qdq_to_2dq
- **Did you write any new necessary tests?**: No
- **Did you add or update any necessary documentation?**: No
- **Did you update
[Changelog](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CHANGELOG.rst)?**:
No <!--- Only for new features, API changes, critical bug fixes or bw
breaking changes. -->
Signed-off-by: ajrasane <131806219+ajrasane@users.noreply.github.com>
## What does this PR do?
**Type of change:** ?
Bug fix
**Overview:** ?
For the trt_diffusions script
## Testing
python diffusion_trt.py --model flux-dev --benchmark --skip-image
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
## What does this PR do?
**Type of change:**
new feature
**Overview:** ?
Attention mechanism has changed from diffusers 0.35.
Many model attentions are now subclass of a new Mixin class:
AttentionModuleMixin, which is not a sub class of Attention
To fix it, patch the mixin class by forcing to use native attention
impl so the existing function monkey patch still work.
## Testing
manual quant of Wan, Flux
---------
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
## What does this PR do?
**Type of change:**
Bug fix
**Overview:**
1. The diffusion_trt.py needs the dynamic_shapes when running trtexec
for engine building. A previous change altered the format of
dynamic_shapes, fix it here.
2. the dynamic_shapes logic gets cleaned up. The existing logic is very
confusing
3. recover min-batch_size config for some pipelines. Previously some
pipelines set the min batch_size to be > 1, which was odd, so a previous
change sets them to be 1, but it turns out the oddity has a reason, the
trt engine building fails with the altered batch_size min/opt, thus
recover them.
## Testing
pytest tests/examples/diffusers
---------
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
## What does this PR do?
**Type of change:** ?
Bug fix
**Overview:** ?
1. We are passing incorrect extra args to pipeline inference.
2. Need a default empty list for the list argument component-dtype
## Testing
Passed SDXL pipeline without component dtype
---------
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
## What does this PR do?
**Type of change:**
new feature
**Overview:**
Support Wan2.2 t2v diffusers quantization
1. fix torch2.9 support
2. add Wan2.2 t2v diffusers pipeline quantization
Main difference of the Wan2.2 pipeline comparing to exisiting
pipelines is that there are 2 backbone models for denoising. For the
quantization therefore we need to quantize both of them.
However, it turns out our base library does not well support
quantization of multiple models in the same time. Therefore, the
change here just stick to quantize a single model each time, and then
run the quantization multiple times.
So, we need to allow users to pick which backbone to quantize,
therefore adding a new argment for it
3. add a workaround for the exporting ONNX issue when we upgrade
diffusers to >= 0.35.0. The issue lies is the exporting of the
torch.nn.RMSNorm. Some pipelines in the diffusers > 0.35.0 use the
torch version RMSNorm while before that they use the diffusers' own
version of RMSNorm. It turns out they are directly replacable so the
workaround is to simply replace the torch RMSNorm usages with
diffusers RMSNorm. But we need to fix it properly soon by porting our
ONNX export to be based on torch dynamo instead of torchscript. Issue
reported from external user:
https://github.com/NVIDIA/TensorRT-Model-Optimizer/issues/262
4. allow use of a prompts file, which is simply a text file with a list
of prompts, one prompt each line
5. allow each component of a pipeline to have different dtype accuracy.
added a new list stype command line arg --component-dtype for this.
example: --component-dtype vae:Float
6. print the summary of the quantized model so users can capture issues
from
log
## Usage
python quantize.py \
--model wan2.2-t2v-14b \
--format fp8 \
--batch-size 4 \
--calib-size 64 \
--n-steps 20 \
--backbone transformer \
--model-dtype BFloat16 \
--component-dtype vae:Float \
--trt-high-precision-dtype BFloat16 \
--quantized-torch-ckpt-save-path ./wan_transformer.pt \
--onnx-dir wan-transformer-onnx \
--prompts-file wan-prompts.txt
## Testing
Tested SDXL_BASE, LTX_VIDEO_DEV, WAN22_T2V
## Before your PR is "*Ready for review*"
<!-- If you haven't finished some of the above items you can still open
`Draft` PR. -->
- **Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CONTRIBUTING.md)**
and your commits are signed.
- **Is this change backward compatible?**: Yes/No <!--- If No, explain
why. -->
- **Did you write any new necessary tests?**: Yes/No
- **Did you add or update any necessary documentation?**: Yes/No
- **Did you update
[Changelog](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CHANGELOG.rst)?**:
Yes/No <!--- Only for new features, API changes, critical bug fixes or
bw breaking changes. -->
## Additional Information
https://github.com/NVIDIA/TensorRT-Model-Optimizer/issues/262
---------
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
## What does this PR do?
**Type of change:**
Minor code change
**Overview:**
- Delete backbone after Device Model creation
- Add assertion for torch compile
- Update dummy input generation function
## Testing
```
python diffusion_trt.py --model flux-dev --benchmark --skip-image
python diffusion_trt.py --model flux-dev --benchmark --skip-image --restore-from ./flux_dev_fp8_autodeploy_fake.pt
python diffusion_trt.py --model flux-dev --benchmark --skip-image --restore-from ./flux_dev_fp4_autodeploy_fake.pt
python diffusion_trt.py --model flux-dev --benchmark --skip-image --torch
python diffusion_trt.py --model flux-dev --benchmark --skip-image --restore-from ./flux_dev_fp8_autodeploy_fake.pt --torch
python diffusion_trt.py --model flux-dev --benchmark --skip-image --restore-from ./flux_dev_fp4_autodeploy_fake.pt --torch
python diffusion_trt.py --model flux-dev --benchmark --skip-image --torch --torch-compile
```
## Before your PR is "*Ready for review*"
<!-- If you haven't finished some of the above items you can still open
`Draft` PR. -->
- **Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CONTRIBUTING.md)**
and your commits are signed.
- **Is this change backward compatible?**: Yes
- **Did you write any new necessary tests?**: No
- **Did you add or update any necessary documentation?**: No
- **Did you update
[Changelog](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CHANGELOG.rst)?**:
No <!--- Only for new features, API changes, critical bug fixes or bw
breaking changes. -->
---------
Signed-off-by: ajrasane <131806219+ajrasane@users.noreply.github.com>
## What does this PR do?
**Type of change:**
Example update
**Overview:**
- Optimize the benchmarking function in the diffusers example
```python
python diffusion_trt.py --model flux-dev --benchmark --model-dtype BFloat16 --skip-image --torch
```
## Testing
```
Backbone-only inference latency (BFloat16):
Average: 139.48 ms
P50: 139.36 ms
P95: 141.13 ms
P99: 141.35 ms
```
## Before your PR is "*Ready for review*"
<!-- If you haven't finished some of the above items you can still open
`Draft` PR. -->
- **Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CONTRIBUTING.md)**
and your commits are signed.
- **Is this change backward compatible?**: Yes
- **Did you write any new necessary tests?**: No
- **Did you add or update any necessary documentation?**: No
- **Did you update
[Changelog](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CHANGELOG.rst)?**:
No
---------
Signed-off-by: ajrasane <131806219+ajrasane@users.noreply.github.com>
## What does this PR do?
**Type of change:**
Minor code change
**Overview:**
- Select the high precision dtype directly based on model type - FP16
for Stable Diffusion models, BF16 for Flux
## Testing
```python
python diffusion_trt.py --model flux-dev --benchmark
```
## Before your PR is "*Ready for review*"
<!-- If you haven't finished some of the above items you can still open
`Draft` PR. -->
- **Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CONTRIBUTING.md)**
and your commits are signed.
- **Is this change backward compatible?**: No (No option to specify
dtype while loading pipeline)
- **Did you write any new necessary tests?**: No
- **Did you add or update any necessary documentation?**: Yes
- **Did you update
[Changelog](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CHANGELOG.rst)?**:
Yes
---------
Signed-off-by: ajrasane <131806219+ajrasane@users.noreply.github.com>
Shengliang Xu