mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-27 07:05:14 +08:00
Chenhan D. YuandClaude Opus 4.6
355c6b7883
## Summary - **megatron_lm_ptq.yaml**: Qwen3-8B PTQ to single GPU for L40 clusters (TP=1, all tasks) - **quantize.sh**: Auto-find largest PP dividing model's `num_hidden_layers` for export step. Qwen3-8B has 36 layers which isn't divisible by 8, causing `AssertionError` on 8-GPU nodes - **compute_hidden_states_trtllm.py**: Use `messages` with `conversations` fallback, matching the HF version. Fixes `KeyError: 'conversations'` when data uses OpenAI `messages` format ## Test plan - [x] Qwen3-8B PTQ runs on single L40 GPU - [x] Export PP auto-selects valid divisor (36 layers → PP=6 on 8 GPUs, PP=4 on 4 GPUs, PP=1 on 1 GPU) - [x] EAGLE3 offline pipeline reads data with `messages` field 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Dataset input handling now supports multiple field formats for enhanced compatibility. * **Bug Fixes** * Optimized GPU resource allocation during model quantization with improved pipeline parallelism computation. * Updated quantization configuration for more efficient resource utilization. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Chenhan Yu <chenhany@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ModelOpt Launcher
Submit ModelOpt quantization, training, and evaluation jobs to Slurm clusters or run them locally with Docker.
Quick Start
# Install
curl -LsSf https://astral.sh/uv/install.sh | sh
git submodule update --init --recursive
# Run locally with 1 GPU
cd Model-Optimizer/tools/launcher
uv run launch.py --yaml examples/Qwen/Qwen3-8B/megatron_lm_ptq_local.yaml hf_local=/mnt/hf-local --yes
# Run on a Slurm cluster (4 GPUs)
export SLURM_HOST=login-node.example.com
export SLURM_ACCOUNT=my_account
export SLURM_HF_LOCAL=/mnt/hf-local
export SLURM_JOB_DIR=/shared/experiments
uv run launch.py --yaml examples/Qwen/Qwen3-8B/megatron_lm_ptq.yaml --yes
Local vs cluster:
megatron_lm_ptq.yamldefaults to TP=4 on 4 GPUs. Usemegatron_lm_ptq_local.yamlfor single-GPU local Docker runs.
Directory Structure
tools/launcher/
├── launch.py # Main entrypoint
├── core.py # Core logic (dataclasses, executors, run loop)
├── slurm_config.py # SlurmConfig dataclass and factory
├── common/ # Scripts and typed tasks
│ ├── megatron_lm/quantize/
│ │ ├── quantize.sh # PTQ quantization + MMLU evaluation
│ │ └── task.py # MegatronLMQuantizeTask (typed config)
│ ├── tensorrt_llm/query.sh # TRT-LLM server + query
│ ├── vllm/query.sh # vLLM server + query
│ ├── eagle3/ # EAGLE3 speculative decoding scripts
│ └── specdec_bench/ # Speculative decoding benchmark
├── examples/ # Example configs
│ └── Qwen/Qwen3-8B/
│ ├── megatron_lm_ptq.yaml # PTQ (4 GPUs, Slurm)
│ ├── megatron_lm_ptq_local.yaml # PTQ (1 GPU, local Docker)
│ └── hf_offline_eagle3.yaml # EAGLE3 offline pipeline
├── tests/ # 64 unit tests
├── modules/ # Dependencies
│ ├── Megatron-LM/ # Git submodule
│ └── Model-Optimizer -> ../.. # Symlink (auto-created)
└── docs/ # Documentation
├── configuration.md # YAML formats, overrides, hf_local
├── architecture.md # Design, factory system, typed tasks
├── testing.md # Running tests, CI
├── claude_code.md # Claude Code workflows
└── contributing.md # Adding models, bug reporting
Documentation
| Guide | Description |
|---|---|
| Configuration | YAML formats, CLI overrides, flags, hf_local |
| Architecture | Shared core, factory system, typed tasks, mount mechanism |
| Testing | Running tests locally and in CI |
| Claude Code | Submit, monitor, diagnose workflows |
| Contributing | Adding models, typed tasks, bug reporting |