mirror of https://github.com/NVIDIA/Model-Optimizer.git synced 2026-11-07 08:54:02 +08:00

Files

T

Keval MorabiaandRinZ27 5dc17dfd15 [Security] Enable torch.load(weights_only=True) for secure checkpoint loading + trust_remote_code fix (#1181 )

### What does this PR do?

- Add secure checkpoint loading support using
`torch.serialization.add_safe_globals([cls])`. This also removes 1
existing pickle usage.
- Remove hard-coded `trust_remote_code=True`
- Replaces https://github.com/NVIDIA/Model-Optimizer/pull/1056 by
@RinZ27

### Testing
<!-- Mention how have you tested your change if applicable. -->

CICD tests ran

### Before your PR is "*Ready for review*"

Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).

Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).

- Is this change backward compatible?: ✅ <!--- If ❌, explain why. -->
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ <!---
Mandatory -->
- Did you write any new necessary tests?: ✅ <!--- Mandatory for new
features or examples. -->
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
✅ <!--- Only for new features, API changes, critical bug fixes or
backward incompatible changes. -->

### Additional Information

NVBug: 5999336

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added safe checkpoint save/load helpers and a --trust_remote_code CLI
flag in examples to control remote-code loading.

* **Bug Fixes**
* Checkpoint loading now defaults to safer, weights-only semantics to
reduce arbitrary-code exposure.

* **Documentation**
* CHANGELOG updated with security guidance and opt-in procedure for
unsafe checkpoint loading.

* **Tests**
  * New unit tests validating the safe-load behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: RinZ27 <222222878+RinZ27@users.noreply.github.com>
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Co-authored-by: RinZ27 <222222878+RinZ27@users.noreply.github.com>

2026-04-08 00:36:28 +05:30

scripts

Update for 0.25.0 release

2025-03-03 22:54:22 +05:30

api_client.py

Update for 0.25.0 release

2025-03-03 22:54:22 +05:30

api_server.py

[fix][5875912] Fix autoquant-autodeploy example (#878 )

2026-02-11 20:33:08 +00:00

README.md

Product Rename: TensorRT Model Optimizer to Model Optimizer (#583 )

2025-12-07 12:33:21 +05:30

run_auto_quantize.py

[Security] Enable torch.load(weights_only=True) for secure checkpoint loading + trust_remote_code fix (#1181 )

2026-04-08 00:36:28 +05:30

README.md

Deploy AutoQuant Models with AutoDeploy

This guide demonstrates how to deploy mixed-precision models using ModelOpt's AutoQuant and TRT-LLM's AutoDeploy.

ModelOpt's AutoQuant is a post-training quantization (PTQ) algorithm that optimizes model quantization by selecting the best quantization format for each layer while adhering to user-defined compression constraints. This approach allows users to balance model accuracy and performance effectively.

TRT-LLM's AutoDeploy is designed to simplify and accelerate the deployment of PyTorch models, including off-the-shelf models like those from Hugging Face, to optimized inference environments with TRT-LLM. It automates graph transformations to integrate inference optimizations such as tensor parallelism, KV-caching and quantization. AutoDeploy supports optimized in-framework deployment, minimizing the amount of manual modification needed.

Prerequisites

AutoDeploy is available in TensorRT-LLM docker images. Please refer to our Installation Guide for more details.

1. Quantize and Deploy Model

Run the following command to quantize your model and launch an OpenAI-compatible endpoint:

./scripts/run_auto_quant_and_deploy.sh \
    --hf_ckpt <path_to_HF_model> \
    --save_quantized_ckpt <path_to_save_quantized_checkpoint> \
    --quant fp8,nvfp4 \
    --effective_bits 4.5

Parameters:

--hf_ckpt: Path to the unquantized Hugging Face checkpoint
--save_quantized_ckpt: Output path for the quantized checkpoint
--quant: Quantization formats to use (e.g., fp8,nvfp4)
--effective_bits: Target overall precision (higher values preserve accuracy for sensitive layers)
--calib_batch_size: (Optional, default=8) Calibration batch size. Reduce if encountering OOM issues

Note

:

NVFP4 is only available on Blackwell GPUs. For Hopper GPUs:

Remove nvfp4 from the --quant parameter

Increase --effective_bits above 8.0 for FP8-only AutoQuant

For tensor parallelism, add --world_size <gpu_num>

Additional generation and sampling configurations can be found in api_server.py

2. Test the Deployment

Send test prompts to the server:

python api_client.py --prompt "What is AI?" "What is golf?"

This will return generated responses for both prompts from your deployed model.