mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-19 05:49:19 +08:00
Keval Morabia
53a2ddebab
- [x] Product Rename: TensorRT Model Optimizer to Model Optimizer (OMNIML-3033) - [x] Mention in Latest News section with date on the date of merging this PR (12/08) Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Deploy quantized models from Nvidia's Hugging Face model hub with TensorRT-LLM, vLLM, and SGLang
The provided bash scripts show an example to deploy and run the quantized LLama 3.1 8B Instruct FP8 model from Nvidia's Hugging Face model hub on TensorRT-LLM, vLLM and SGLang respectively.
Before running the bash scripts, please make sure you have setup the environment properly:
- Make sure you are authenticated with a Hugging Face account to interact with the Hub, e.g., use
huggingface-cli loginto save the access token on your machine. - Install TensorRT-LLM properly by following instructions here.
- Install vLLM properly by following instructions here.
- Install SGLang properly by following instructions here
Then, to deploy and run on TensorRT-LLM:
python run_llama_fp8_trtllm.py
To deploy and run on vLLM:
python run_llama_fp8_vllm.py
To deploy and run on SGLang:
python run_llama_fp8_sglang.py
If you want to run post-training quantization with Model Optimizer for your selected models, check here.