mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-11-10 09:34:01 +08:00
sugunav14
02c5f292f0
## What does this PR do? **Type of change:** New feature <!-- Use one of the following: Bug fix, new feature, new example, new tests, documentation. --> **Overview:** Adds support for GPTQ algorithm. This PR implements a modified version of the official GPTQ algorithm; the key difference is that updated activations from each layer are not used for hessian computation ## Usage <!-- You can potentially add a usage example below. --> Modify "algorithm" field in quant_cfg to "gptq_lite". Note: Does not currently work with AWQ ```python # Add a code snippet demonstrating how to use this ``` ## Testing <!-- Mention how have you tested your change if applicable. --> - [x] Added unit tests to test helper functions + e2e flow - [x] Perplexity and GPQA results | Model | Qformat | Perplexity wikitext2 | GPQA | |-------------|------------------------|------------------------|------| | Qwen3-8B | INT4 weight only (modelopt + amax/7) no GPTQ | 10.75 | n/a | | Qwen3-8B | INT4 weight only (modelopt + amax/7) | **10.56** | 0.388 | | Qwen3-8B | INT4 weight only + FP-Quant hessians + amax/7.5 | 10.25 | **0.449** | | Qwen3-8B | INT4 weight only (FP-Quant) | 10.24 | 0.46 | | Qwen3-8B | NVFP4 static weight only | 10.25 | n/a | | Qwen3-8B | NVFP4 static weight only no GPTQ | 10.25 | n/a | | Qwen3-0.6B | NVFP4 static weight only | **22.75** | n/a | | Qwen3-0.6B | NVFP4 dynamic weight only | 23.50 | n/a | | Qwen3-0.6B | NVFP4 static weight only with FP-Quant hessians | 22.0 | n/a | | Qwen3-0.6B | NVFP4 static weight only no GPTQ | 24.25 | n/a | Conclusions from results - Perplexity remains the same or shows improvement with Modelopt implementation. The magnitude of improvement is lesser in modelopt when compared to FP-Quant - GPQA shows no improvement with modelopt, but shows improvement with FP-Quant ## Before your PR is "*Ready for review*" <!-- If you haven't finished some of the above items you can still open `Draft` PR. --> - **Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CONTRIBUTING.md)** and your commits are signed. - **Is this change backward compatible?**: Yes <!--- If No, explain why. --> - **Did you write any new necessary tests?**: Yes - **Did you add or update any necessary documentation?**: No - **Did you update [Changelog](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CHANGELOG.rst)?**: No <!--- Only for new features, API changes, critical bug fixes or bw breaking changes. --> ## Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * GPTQ Lite quantization mode now available for efficient model calibration * GPU memory usage monitoring utility added * Quantization configuration extended to support complex nested structures and lists * **Tests** * Comprehensive test coverage added for GPTQ quantization workflows <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Suguna Velury <178320438+sugunav14@users.noreply.github.com>