ynankani-nv
4a70040ddd
### What does this PR do? Type of change: ? New benchmark addition <!-- Details about the change. --> This PR adds the following to examples/windows/: 1) FVD (Fréchet Video Distance) evaluation tool — a standalone script for computing FVD between two sets of videos using a pre-trained I3D model (Kinetics-400, 1024-dim pooled features). 2) Directory reorganization — moved torch_onnx/ and qad_example/ under a new diffusers/ folder for clearer grouping of diffusion model examples. 3) Minor fixes: Updated WikiText-2 dataset link in perplexity_metrics/README.md (old wikitext → Salesforce/wikitext) ### Usage ```python python compute_fvd.py \ --ref-dir /path/to/reference/videos \ --gen-dir /path/to/generated/videos \ --output results.json \ --device cuda ``` ### Testing Evaluated FVD tool on LTX-2.3 outputs comparing QAD vs PTQ checkpoints (both NVFp4) against a BF16 baseline across 11 VBench dimensions. ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A <!--- Mandatory --> - Did you write any new necessary tests?: N/A <!--- Mandatory for new features or examples. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> ### Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added FVD (Fréchet Video Distance) evaluation: command‑line tool, I3D‑based feature extraction, PCA option, and quantized comparisons (PTQ vs QAD) with per‑category and overall results showing QAD's lower average FVD. * **Documentation** * Expanded benchmarks TOC with “Additional Metrics”, added FVD evaluation guide, usage examples, troubleshooting, and updated perplexity dataset reference. * **Chores** * Added Python requirements for the FVD benchmark. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: ynankani <ynankani@nvidia.com> Signed-off-by: Yash Nankani <ynankani@nvidia.com>
10 KiB
Model Optimizer - Windows: Benchmark Reference
This document provides a summary of the performance and accuracy measurements of Model Optimizer - Windows for several popular models. The benchmark results in the following tables serve as reference points and should not be viewed as the maximum performance achievable by Model Optimizer - Windows.
1 Performance And Accuracy Comparison: ONNX INT4 vs ONNX FP16 Models
1.1 Performance Comparison
All performance metrics are tested using the onnxruntime-genai perf benchmark with the DirectML backend.
- Configuration: Windows OS, GPU RTX 4090, NVIDIA Model Optimizer v0.19.0.
- Batch Size: 1
Memory savings and inference speedup are compared to the ONNX FP16 baseline.
| Model | Input Prompt Length | Output tokens length | GPU Memory Saving | Generation Phase Inference Speedup |
| Llama3.1-8B-Instruct | 128 | 256 | 2.44x | 2.68x |
| Phi3.5-mini-Instruct | 128 | 256 | 2.53x | 2.51x |
| Mistral-7B-Instruct-v0.3 | 128 | 256 | 2.88x | 3.41x |
| Llama3.2-3B-Instruct | 128 | 256 | 1.96x | 2.19x |
| Gemma-2b-it | 128 | 256 | 1.64x | 1.94x |
1.2 Accuracy Comparison
1.2.1 MMLU
For accuracy evaluation, the Massive Multitask Language Understanding (MMLU) benchmark has been utilized. Please refer to the detailed instructions for running the MMLU accuracy benchmark.
The table below shows the MMLU 5-shot score for some models.
- FP16 ONNX model: Generated using GenAI Model Builder with DML EP
- INT4 AWQ model: Generated by quantizing FP16 ONNX model using ModelOpt-Windows
- Configuration: Windows OS, GPU RTX4090, nvidia-modelopt v0.19.0, onnxruntime-genai-directml 0.4, transformers 4.44
| Model | ONNX FP16 | ONNX INT4 |
|---|---|---|
| Llama3.1-8B-Instruct | 68.45 | 66.1 |
| Phi3.5-mini-Instruct | 68.9 | 65.7 |
| Mistral-7B-Instruct-v0.3 | 61.76 | 60.73 |
| Llama3.2-3B-Instruct | 60.8 | 57.71 |
| Gemma-2b-it | 37.01 | 37.2 |
1.2.2 Perplexity (PPL)
Perplexity measures how well a probability model predicts a sample. Lower perplexity values indicate better model quality. The following table shows perplexity values at input sequence length 1024 with chunk size of 512.
Learn more about Perplexity: Perplexity - Wikipedia | Hugging Face - Perplexity of Fixed-Length Models
- FP16-MB: Baseline FP16 genai model (Model Builder)
- Mixed AWQ-MO: Important linear layers in INT8, rest in INT4 (AWQ), with ModelOpt.
- Mixed RTN-MO: Important linear layers in INT8, rest in INT4 (RTN), with ModelOpt.
- Pure INT4 AWQ-MO: All linear layers INT4 (AWQ) with ModelOpt.
- Pure INT4 RTN-MO: All linear layers INT4 (RTN) with ModelOpt.
- Pure INT8 RTN-MO: All linear layers INT8 (RTN) with ModelOpt.
- Pure INT8 AWQ-MO: All linear layers INT8 (AWQ) with ModelOpt.
- Configuration: Windows OS, GPU RTX 5090, nvidia-modelopt v0.39.0, onnxruntime-genai-cuda 0.9.2, onnxruntime-gpu 1.23.0, torch 2.8.0+cu128, transformers 4.49.0
| Model | FP16-MB | Mixed AWQ-MO | Mixed RTN-MO | Pure INT4 AWQ-MO | Pure INT4 RTN-MO | Pure INT8 RTN-MO | Pure INT8 AWQ-MO |
|---|---|---|---|---|---|---|---|
| DeepSeek R1 Distill Qwen 1.5B | 39.447 | 41.699 | 44.332 | 44.213 | 46.304 | 39.802 | 39.713 |
| Llama 3.2 1B Instruct | 12.631 | 13.852 | 14.176 | 14.549 | 16.900 | 12.664 | 12.637 |
| Phi-3.5 Mini Instruct | 6.046 | 6.500 | 6.599 | 6.711 | 7.070 | - | - |
| Phi-4 Mini Instruct | 9.039 | 9.673 | 9.712 | 10.015 | 10.911 | - | - |
| Qwen 2.5 1.5B Instruct | 9.216 | 10.084 | 10.338 | 10.495 | 10.933 | 9.227 | 9.232 |
For detailed instructions on evaluating perplexity, please refer to the Perplexity Evaluation Guide.
1.2.3 KL-divergence
KL-divergence (Kullback-Leibler divergence) quantifies the distributional difference between the quantized model and the baseline model. Lower KL-divergence values indicate that the quantized model's output distribution is closer to the original model.
Learn more about KL-divergence: KL Divergence - Wikipedia | Understanding KL Divergence
Supported backends: PyTorch and Onnxruntim-cuda, onnxruntime-trt-rtx-ep are both supported for evaluation.
- Baseline model: Hugging Face FP16 model
- Quantized models: Models where quantization is simulated (a.k.a. fake quantization), typically using the PyTorch-CUDA backend for evaluation. Fake quantization means quantized weights and dequantized simultaneously to simulate quantization. The inference backend column in the table below indicates whether the reported results are from PyTorch simulation or ONNX-runtime-based inference.
- Configuration: Windows OS, GPU RTX 5090, nvidia-modelopt v0.39.0, onnxruntime-genai-cuda 0.9.2, onnxruntime-gpu 1.23.0, torch 2.8.0+cu128, transformers 4.49.0
| Model | Quantization Method | Quantization Granularity | KL-divergence | Inference Backend |
|---|---|---|---|---|
| Qwen2.5-1.5B-Instruct | Base FP16 (Baseline) | - | 0.000 | PyTorch (FP16) |
| Qwen2.5-1.5B-Instruct | int4+int8 Blockwise-max_algo-mixed_quant (simulated) | INT4: per-block (block-size=128), INT8: per-channel (row-wise) | 0.336 | PyTorch (fake quantization) |
| Qwen2.5-1.5B-Instruct | int4+int8 max_algo-mixed_quant (simulated, per-channel) | INT4: per-block (block-size=128), INT8: per-channel (row-wise) | 0.337 | PyTorch (fake quantization) |
| Llama-3.2-3B-Instruct | Base FP16 (Baseline) | - | 0.000 | PyTorch (FP16) |
| Llama-3.2-3B-Instruct | int4+int8 Blockwise-awq-lite_algo-mixed_quant (simulated) | INT4: per-block (block-size=128), INT8: per-channel (row-wise) | 0.228 | PyTorch (fake quantization) |
| Llama-3.2-3B-Instruct | int4+int8 per-channel-awq-lite_algo-mixed_quant (simulated) | INT4: per-block (block-size=128), INT8: per-channel (row-wise) | 0.230 | PyTorch (fake quantization) |
| Llama-3.2-3B-Instruct | int4+int8 Blockwise-max_algo-mixed_quant (simulated) | INT4: per-block (block-size=128), INT8: per-channel (row-wise) | 0.238 | PyTorch (fake quantization) |
| Llama-3.2-3B-Instruct | int4+int8 per-channel-max_algo-mixed_quant (simulated) | INT4: per-block (block-size=128), INT8: per-channel (row-wise) | 0.238 | PyTorch (fake quantization) |
| Llama-3.2-3B-Instruct | int4 Blockwise-max_algo only (simulated) | INT4: per-block (block-size=128) | 0.334 | PyTorch (fake quantization) |
All KL-divergence results above are obtained via PyTorch fake quantization simulation unless otherwise noted. Inference with ONNX-runtime can also be evaluated .
For detailed instructions on computing KL-divergence, please refer to the KL-divergence Evaluation Guide.
1.2.4 FVD (Fréchet Video Distance)
FVD measures the distributional similarity between two sets of videos using features extracted from a pre-trained I3D model (Kinetics-400). Lower FVD values indicate that the generated videos are closer to the reference. FVD is the standard metric for evaluating video generation quality.
Learn more about FVD: FVD: A New Metric for Video Generation (arXiv)
- Reference baseline: BF16 model outputs
- Quantized models: PTQ (Post-Training Quantization) and QAD (Quantization-Aware Distillation)
- Model: LTX-2.3 video generation
- Evaluation dimensions: VBench benchmark categories
- Feature extractor: I3D (1024-dim pooled features from
rgb_imagenet.pt)
| Category | FVD: PTQ vs BF16 ↓ | FVD: QAD vs BF16 ↓ |
|---|---|---|
| Temporal Flickering | 31.92 | 21.97 |
| Subject Dynamic Motion | 23.44 | 16.28 |
| Multiple Objects | 35.35 | 22.47 |
| Human Action | 30.08 | 21.82 |
| Object Class | 51.51 | 26.86 |
| Color | 36.52 | 25.09 |
| Spatial Relationship | 25.07 | 18.41 |
| Scene Background | 64.92 | 35.69 |
| Appearance Style | 31.08 | 20.82 |
| Temporal Style | 23.61 | 15.85 |
| Overall Consistency | 25.03 | 18.85 |
| Average | 34.41 | 22.19 |
QAD consistently outperforms PTQ across all VBench dimensions, achieving 35% lower average FVD (22.19 vs 34.41).
For detailed instructions on computing FVD, please refer to the FVD Evaluation Guide.