Fix the llm_eval README commands that no longer run as written (#2358)

### What does this PR do? Type of change: Documentation (plus one small example-script fix) An audit of `examples/llm_eval/README.md` against the current scripts (nvbug 6701343) found several documented commands that no longer run as written: - **T5 / seq2seq.** `--model hf-seq2seq` is not a registered lm-eval backend in any version this example supports — the string does not appear in the 0.4.12 or 0.4.13 wheels, so the command fails at model lookup. `HFLM` detects encoder-decoder models from `config.json`, so the example now uses `--model hf` and mentions `backend=seq2seq` as the override for checkpoints lm-eval cannot classify. No ModelOpt-side change was needed: encoder-decoder calibration already works (verified below). - **auto_quantize format list.** `FP8_DEFAULT_CFG|NVFP4_DEFAULT_CFG` was shown as a literal value in both README locations, but each comma-separated entry is resolved with `getattr(mtq, ...)` and that name does not exist. Now shows a valid list, spells out the choices, and names the placeholder consistently with the surrounding block. - **`vllm serve`.** A missing line continuation meant `--port` ran as a separate shell command. - **MMLU setup.** Dropped a stray `cd ..` left over from the 0.11 examples release. It leaves `examples/llm_eval`, where both `mmlu.py` and its default `--data_dir data/mmlu` live; `hf_ptq/scripts/huggingface_example.sh` correctly stays put throughout its MMLU flow, so the README was the only thing out of step. - **`run_simple_eval.sh`.** Documented the optional fifth argument (`--examples`), which `huggingface_example.sh` already passes as `$SIMPLE_EVAL_LIMIT`. Two changes beyond the docs: - **`quantization_utils.py`:** under `auto_quantize`, a `quant_cfg` string was iterated character by character, so a single format failed with the baffling `AttributeError: module 'modelopt.torch.quantization' has no attribute 'F'`. Normalized `str -> list` at the point the list is consumed, which covers both `mmlu.py` and `lm_eval_hf.py` rather than one caller. This also honors the existing `str | list[str]` annotation. - **`requirements.txt`:** added the missing `openai`. `modeling.py` imports it unconditionally and `lm_eval[api]` supplies only `tiktoken`, so every documented `mmlu.py` command died with `ModuleNotFoundError` on a clean install of the stated requirements. Note on the filed report: its item 3 claimed `mmlu.py` fails to split the comma-separated config list. That does not reproduce — `mmlu.py` uses `fire`, which already parses `A,B,NONE` into a tuple, and the unmodified script completes `auto_quantize` fine. Applying the suggested `quant_cfg.split(",")` would have *broken* the documented command with `AttributeError: 'tuple' object has no attribute 'split'`. The `quantization_utils.py` change above addresses the real adjacent defect instead. Pushback recorded on the bug. ### Usage No new API or flag. The corrected commands: ```bash # T5 / encoder-decoder (was: --model hf-seq2seq, which does not exist) python lm_eval_hf.py --model hf --model_args pretrained=t5-small \ --quant_cfg FP8_DEFAULT_CFG --tasks <comma separated tasks> --batch_size 4 # auto_quantize search list (was: W4A8_AWQ_BETA_CFG,FP8_DEFAULT_CFG|NVFP4_DEFAULT_CFG,NONE) python mmlu.py --model_name causal --model_path <model> \ --quant_cfg W4A8_AWQ_BETA_CFG,FP8_DEFAULT_CFG,NONE --auto_quantize_bits 4.8 --batch_size 4 # simple evals, optional 5th arg bash run_simple_eval.sh <model> <evals> <max_tokens> <port> [num examples per eval] ``` ### Testing Ran on 2x RTX 6000 Ada with a tiny Qwen3 and a locally synthesized MMLU tree (no download): - **`mmlu.py --auto_quantize_bits` with the documented comma-separated list** — completes quantization on both the unpatched and patched script, confirming the reported item 3 is a false positive. Probed `fire` directly: bare, quoted and `--flag=value` forms all yield `('W4A8_AWQ_BETA_CFG', 'FP8_DEFAULT_CFG', 'NONE')`. - **`mmlu.py --auto_quantize_bits` with a single format** — proved the new guard fires by reverting it: without the change the run dies with `AttributeError: module 'modelopt.torch.quantization' has no attribute 'F'`; with it, the run reaches a legitimate domain assertion (`effective_bits 4.8` cannot be below FP8's 8 bits). - **Encoder-decoder calibration** — quantized a T5 with `FP8_DEFAULT_CFG` through `quantize_model` and confirmed encoder, decoder and cross-attention (`EncDecAttention`) layers all calibrate with real amax values. This is what settled keeping the T5 example rather than deleting it. - **`vllm serve` snippet** — parsed the fixed block with `bash`; `--quantization`, `--port` and `--tensor-parallel-size` now all belong to one command. - **`run_simple_eval.sh`** — confirmed the 4-arg form is unchanged and the 5-arg form emits `--examples 16`. - **Lint** — `ruff-check`, `ruff-format`, `markdownlint-cli2`, `typos`, `bandit`, `mypy`, `requirements-txt-fixer`, `mixed-line-ending` all pass. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ — added `openai` to `examples/llm_eval/requirements.txt`; it is Apache 2.0 (permissive), so no codeowners exception is needed. It is not a new runtime dependency of the library, and `run_simple_eval.sh` already `pip install`s it. - Did you write any new necessary tests?: N/A — docs plus a two-line defensive normalization in an example util. `mmlu.py` cannot be imported without `openai`/`rwkv`/`tiktoken`, so a hermetic unit test would need more stub scaffolding than the line it guards; verified by direct execution instead, as above. - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A — examples-only documentation cleanup, not a feature, breaking change, deprecation, or a critical bug from a previous release. - Did you get Claude approval on this PR?: ❌ — not yet run. ### Additional Information Fixes nvbug 6701343 / OMNIML-5806. Item 3 of the filed report is a false positive; pushback and evidence are recorded in a comment on the bug.  ## Summary by CodeRabbit - **New Features** - Auto-quantization now supports comma-separated format configurations. - Added an optional example-limit setting for Simple Evals. - Added OpenAI support for LLM evaluation examples. - **Documentation** - Clarified encoder-decoder model usage with `lm_eval`. - Added instructions for running MMLU from the evaluation examples directory. - Corrected the vLLM command formatting.  Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-02 03:14:52 +08:00 · 2026-09-09 00:44:23 +05:30
parent 5c123ce183
commit 58eafdf172
3 changed files with 29 additions and 17 deletions
@@ -79,30 +79,33 @@ For data-parallel evaluation, launch with `accelerate launch --multi_gpu --num_p
 Multi-GPU evaluation without data-parallelism:

 ```sh
-# MODELOPT_QUANT_CFG_TO_SEARCH: Choose the formats to search separated by commas from [W4A8_AWQ_BETA_CFG,FP8_DEFAULT_CFG|NVFP4_DEFAULT_CFG,NONE]
+# MODELOPT_QUANT_CFG_TO_SEARCH: Comma-separated list of the formats auto_quantize searches over.
+# Pick each one from [INT8_SMOOTHQUANT_CFG|FP8_DEFAULT_CFG|NVFP4_DEFAULT_CFG|INT4_AWQ_CFG|W4A8_AWQ_BETA_CFG|MXFP8_DEFAULT_CFG|NONE],
+# where NONE lets auto_quantize leave a layer unquantized.
 # EFFECTIVE_BITS: Effective bits constraint for auto_quantize

-# Examples settings for optimally quantized model with W4A8 & FP8 with effective bits to 4.8:
-# MODELOPT_QUANT_CFG_TO_SEARCH=W4A8_AWQ_BETA_CFG,FP8_DEFAULT_CFG|NVFP4_DEFAULT_CFG,NONE
+# Example settings for an optimally quantized model with W4A8 & FP8 with effective bits of 4.8:
+# MODELOPT_QUANT_CFG_TO_SEARCH=W4A8_AWQ_BETA_CFG,FP8_DEFAULT_CFG,NONE
 # EFFECTIVE_BITS=4.8

 python lm_eval_hf.py --model hf \
    --tasks <comma separated tasks> \
    --model_args pretrained=<HF model folder or model card>,parallelize=True \
-    --quant_cfg <AUTOQUANTIZE_SEARCH_FORMATS> \
+    --quant_cfg <MODELOPT_QUANT_CFG_TO_SEARCH> \
    --auto_quantize_bits <EFFECTIVE_BITS> \
    --batch_size 4
 ```

 For data-parallel evaluation, launch with `accelerate launch --multi_gpu --num_processes <num_copies_of_your_model>` (as shown earlier).

- If evaluating T5 models:
-
-  - use `--model hf-seq2seq` instead.
+- If evaluating encoder-decoder models such as T5, keep `--model hf`: lm-eval detects the
+  encoder-decoder architecture from `config.json`. There is no `hf-seq2seq` backend in the
+  supported lm-eval versions (>= 0.4.12); add `backend=seq2seq` to `--model_args` only for
+  checkpoints lm-eval cannot classify on its own.

 ```sh
 # MODELOPT_QUANT_CFG: Choose from [INT8_SMOOTHQUANT_CFG|FP8_DEFAULT_CFG|NVFP4_DEFAULT_CFG|INT4_AWQ_CFG|W4A8_AWQ_BETA_CFG|MXFP8_DEFAULT_CFG]
-python lm_eval_hf.py --model hf-seq2seq --model_args pretrained=t5-small --quant_cfg=<MODELOPT_QUANT_CFG> --tasks <comma separated tasks> --batch_size 4
+python lm_eval_hf.py --model hf --model_args pretrained=t5-small --quant_cfg <MODELOPT_QUANT_CFG> --tasks <comma separated tasks> --batch_size 4
 ```

 If `trust_remote_code` needs to be true, please append the command with the `--trust_remote_code` flag.
@@ -158,9 +161,10 @@ mkdir -p data
 wget --connect-timeout=20 --read-timeout=60 --tries=3 -c \
    https://huggingface.co/datasets/cais/mmlu/resolve/c30699e8356da336a370243923dbaf21066bb9fe/data.tar -O data/mmlu.tar
 tar -xf data/mmlu.tar -C data && mv data/data data/mmlu
-cd ..
 ```

+Run the commands below from `examples/llm_eval`; `mmlu.py` resolves its default `--data_dir data/mmlu` relative to the current directory.
+
 ### Baseline

 ```bash
@@ -171,17 +175,19 @@ python mmlu.py --model_name causal --model_path <HF model folder or model card>

 ```bash
 # MODELOPT_QUANT_CFG: Choose from [INT8_SMOOTHQUANT_CFG|FP8_DEFAULT_CFG|NVFP4_DEFAULT_CFG|INT4_AWQ_CFG|W4A8_AWQ_BETA_CFG|MXFP8_DEFAULT_CFG]
-python mmlu.py --model_name causal --model_path <HF model folder or model card> --quant_cfg MODELOPT_QUANT_CFG
+python mmlu.py --model_name causal --model_path <HF model folder or model card> --quant_cfg <MODELOPT_QUANT_CFG>
 ```

 ### auto_quantize (simulated)

 ```bash
-# MODELOPT_QUANT_CFG_TO_SEARCH: Choose the formats to search separated by commas from [W4A8_AWQ_BETA_CFG,FP8_DEFAULT_CFG|NVFP4_DEFAULT_CFG,NONE]
+# MODELOPT_QUANT_CFG_TO_SEARCH: Comma-separated list of the formats auto_quantize searches over.
+# Pick each one from [INT8_SMOOTHQUANT_CFG|FP8_DEFAULT_CFG|NVFP4_DEFAULT_CFG|INT4_AWQ_CFG|W4A8_AWQ_BETA_CFG|MXFP8_DEFAULT_CFG|NONE],
+# where NONE lets auto_quantize leave a layer unquantized.
 # EFFECTIVE_BITS: Effective bits constraint for auto_quantize

-# Examples settings for optimally quantized model with W4A8 & FP8 with effective bits to 4.8:
-# MODELOPT_QUANT_CFG_TO_SEARCH=W4A8_AWQ_BETA_CFG,FP8_DEFAULT_CFG|NVFP4_DEFAULT_CFG,NONE
+# Example settings for an optimally quantized model with W4A8 & FP8 with effective bits of 4.8:
+# MODELOPT_QUANT_CFG_TO_SEARCH=W4A8_AWQ_BETA_CFG,FP8_DEFAULT_CFG,NONE
 # EFFECTIVE_BITS=4.8

 python mmlu.py --model_name causal --model_path <HF model folder or model card> --quant_cfg $MODELOPT_QUANT_CFG_TO_SEARCH --auto_quantize_bits $EFFECTIVE_BITS --batch_size 4
@@ -210,9 +216,11 @@ bash run_livecodebench.sh <custom defined model name> <prompt batch size in para
 Similarly, we support running simple evals against a local running OpenAI API compatible server. Once the local server is up, the following command can be used to run the Simple Evals:

 ```bash
-bash run_simple_eval.sh <custom defined model name> <comma separated eval names> <max output tokens> <local model server port>
+bash run_simple_eval.sh <custom defined model name> <comma separated eval names> <max output tokens> <local model server port> [num examples per eval]
 ```

+The optional fifth argument caps the number of examples per eval (`--examples`); omit it to run the full eval.
+
 ## Customize quantization method for evaluation

 An example of customized quantization config is shown in `quantization_utils.py`. It allows users to test accuracy of a custom method without the need of modifying the whole deployment framework, e.g., TensorRT-LLM, vLLM, SGLang, etc. Users can disable quantization of specific layers to debug the cause of accuracy drop, or explore a promising new quantization method.
@@ -258,9 +266,9 @@ This is useful for evaluating quantized models deployed with vLLM or any model s
   ```bash
   # Example using vLLM's built-in server
   vllm serve nvidia/Llama-3.1-8B-Instruct-FP8 \
-    --quantization modelopt
-    --port 8000 \
-    --tensor-parallel-size <tp_size> # Adjust as needed
+       --quantization modelopt \
+       --port 8000 \
+       --tensor-parallel-size <tp_size> # Adjust as needed
   ```

   To generate the quantized model such as `nvidia/Llama-3.1-8B-Instruct-FP8`, please refer to instructions [here](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/hf_ptq#deploy-fp8-quantized-model-using-vllm-and-sglang). Note currently modelopt quantized model support in vLLM is limited, we are working on expanding the model and quant formats support.
@@ -88,6 +88,9 @@ def _quantize_model_with_dataset(
        net = lm

    if auto_quantize_bits is not None:
+        # A bare string would otherwise be iterated character by character below.
+        if isinstance(quant_cfg, str):
+            quant_cfg = quant_cfg.split(",")
        quant_cfg_for_search = [
            getattr(mtq, quant_fmt) for quant_fmt in quant_cfg if quant_fmt != "NONE"
        ]
@@ -1,5 +1,6 @@
 fire>=0.5.0
 lm_eval[api,ifeval]>=0.4.12,<0.5
+openai
 peft>=0.5.0
 rwkv>=0.7.3
 torchvision