mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-02 03:14:52 +08:00
h-guo18
cfc823d127
### What does this PR do? Type of change: new tests / tooling Adds pre-commit validation for speculative-decoding recipes (the existing `check-modelopt-recipes` hook only ran on PTQ) and for launcher YAML references into the recipe library. - `tools/precommit/check_modelopt_recipes.py`: accept `speculative_eagle` / `speculative_dflash` / `speculative_medusa` in addition to `ptq`, so per-model spec-dec recipes (e.g. `modelopt_recipes/models/Qwen3-8B/dflash.yaml`) get full Pydantic validation via `load_recipe()` at commit time. - `tools/precommit/check_launcher_yaml.py` (new): scans every `tools/launcher/examples/**/*.yaml` for `--config <path>` and `data.chat_template=<path>` references, verifies the resolved files exist, and runs `load_recipe()` on any path under `modelopt_recipes/`. Skips `<<global_vars.x>>` interpolation. `pass_filenames: false` so recipe-side edits also re-validate all launcher references. ### Usage ```bash pre-commit run check-modelopt-recipes --all-files pre-commit run check-launcher-yaml --all-files ``` ### Testing Smoke-tested both hooks manually: | Scenario | Result | |---|---| | spec-dec recipe with `dflash_block_size: not_an_int` | exit 1, Pydantic int_parsing error | | launcher YAML with non-existent `--config` path | exit 1, source file + resolved path reported | | launcher YAML with non-existent `data.chat_template` path | exit 1 | | Current repo state (all valid) | exit 0 | ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: ✅ (hooks themselves are the tests) - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A - Did you get Claude approval on this PR?: ❌ pending ### Additional Information Motivated by the per-model recipe migration in #TBD — without these hooks, broken `--config` paths and recipe schema typos surface only at CI or runtime. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Added an automated pre-commit check that validates launcher example YAMLs, reporting parse errors and missing or invalid references. * Expanded recipe validation to cover additional recipe types beyond PTQ, improving detection of invalid recipe formats and metadata. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: h-guo18 <67671475+h-guo18@users.noreply.github.com>
191 lines
7.0 KiB
YAML
191 lines
7.0 KiB
YAML
# NOTE: Make sure to update version in dev requirements (pyproject.toml) as well!
|
|
repos:
|
|
- repo: https://github.com/pre-commit/pre-commit-hooks
|
|
rev: v6.0.0
|
|
hooks:
|
|
- id: check-added-large-files
|
|
args: [--maxkb=500, --enforce-all]
|
|
exclude: >
|
|
(?x)^(
|
|
uv.lock|
|
|
examples/diffusers/quantization/assets/.*.png|
|
|
examples/diffusers/cache_diffusion/assets/.*.png|
|
|
)$
|
|
- id: check-json
|
|
exclude: ^.vscode/.*.json # vscode files can take comments
|
|
- id: check-merge-conflict
|
|
- id: check-symlinks
|
|
- id: check-toml
|
|
- id: mixed-line-ending
|
|
args: [--fix=lf]
|
|
- id: requirements-txt-fixer
|
|
|
|
- repo: https://github.com/astral-sh/ruff-pre-commit
|
|
rev: v0.12.11
|
|
hooks:
|
|
- id: ruff-check
|
|
args: [--fix, --exit-non-zero-on-fix]
|
|
exclude: ^examples/specdec_bench/specdec_bench/datasets/speed\.py$
|
|
- id: ruff-format
|
|
exclude: ^examples/specdec_bench/specdec_bench/datasets/speed\.py$
|
|
|
|
- repo: https://github.com/pre-commit/mirrors-mypy
|
|
rev: v1.17.1
|
|
hooks:
|
|
- id: mypy
|
|
|
|
- repo: https://github.com/pre-commit/mirrors-clang-format
|
|
rev: v21.1.0
|
|
hooks:
|
|
- id: clang-format
|
|
types_or: [c++, c, c#, cuda, java, javascript, objective-c, proto] # no json!
|
|
args: ["--style={ColumnLimit: 100}"]
|
|
|
|
- repo: https://github.com/pre-commit/pygrep-hooks
|
|
rev: v1.10.0
|
|
hooks:
|
|
- id: rst-backticks
|
|
- id: rst-directive-colons
|
|
- id: rst-inline-touching-normal
|
|
|
|
- repo: https://github.com/jumanjihouse/pre-commit-hook-yamlfmt
|
|
rev: 0.2.3
|
|
hooks:
|
|
- id: yamlfmt
|
|
args: [--mapping=2, --sequence=4, --offset=2, --implicit_start, --implicit_end, --preserve-quotes]
|
|
exclude: ^.github/workflows/
|
|
|
|
- repo: local
|
|
hooks:
|
|
- id: normalize-yaml-ext
|
|
name: normalize .yml to .yaml in required places, right now only yaml files in modelopt_recipes
|
|
entry: python tools/precommit/normalize_yaml_ext.py
|
|
language: system
|
|
files: ^modelopt_recipes/.*\.yml$
|
|
|
|
- id: check-modelopt-recipes
|
|
name: validate modelopt recipes
|
|
entry: python tools/precommit/check_modelopt_recipes.py
|
|
language: system
|
|
files: ^modelopt_recipes/
|
|
# configs/ contains reusable snippets (not full recipes) — skip recipe validation
|
|
exclude: ^modelopt_recipes/configs/
|
|
|
|
- id: sync-claude-skills
|
|
name: sync .claude/skills/ symlinks from .agents/skills/
|
|
entry: bash tools/precommit/sync_claude_skills.sh
|
|
language: system
|
|
files: ^\.agents/skills/
|
|
pass_filenames: false
|
|
|
|
- id: check-launcher-yaml
|
|
name: validate launcher YAML references to recipes and templates
|
|
entry: python tools/precommit/check_launcher_yaml.py
|
|
language: system
|
|
files: ^(tools/launcher/examples/.*\.yaml|tools/precommit/check_launcher_yaml\.py)$
|
|
|
|
# Instructions to change license file if ever needed:
|
|
# https://github.com/Lucas-C/pre-commit-hooks#removing-old-license-and-replacing-it-with-a-new-one
|
|
- repo: https://github.com/Lucas-C/pre-commit-hooks
|
|
rev: v1.5.5
|
|
hooks:
|
|
# Default hook for Apache 2.0 in python and shell files
|
|
- id: insert-license
|
|
alias: insert-license-py
|
|
args:
|
|
- --license-filepath
|
|
- ./LICENSE_HEADER
|
|
- --comment-style
|
|
- "#"
|
|
- --allow-past-years
|
|
types_or: [python, shell]
|
|
# NOTE: Exclude files that have copyright or license headers from another company or individual
|
|
# since we want to keep those above the license header added by this hook.
|
|
# Instead, we should manually add the license header to those files *after* the original header.
|
|
exclude: >
|
|
(?x)^(
|
|
modelopt/torch/quantization/utils/calib_utils.py|
|
|
modelopt/onnx/quantization/operators.py|
|
|
modelopt/onnx/quantization/ort_patching.py|
|
|
modelopt/torch/_deploy/utils/onnx_utils.py|
|
|
modelopt/torch/export/transformer_engine.py|
|
|
modelopt/torch/puzzletron/anymodel/models/gpt_oss/gpt_oss_pruned_to_mxfp4.py|
|
|
modelopt/torch/quantization/export_onnx.py|
|
|
modelopt/torch/quantization/plugins/attention.py|
|
|
modelopt/torch/sparsity/attention_sparsity/methods/vsa_utils.py|
|
|
modelopt/torch/speculative/eagle/utils.py|
|
|
modelopt/torch/speculative/plugins/hf_medusa.py|
|
|
modelopt/torch/utils/plugins/megatron_mmlu.py|
|
|
examples/deepseek/deepseek_v3/quantize_to_nvfp4.py|
|
|
examples/deepseek/deepseek_v3/ptq.py|
|
|
examples/diffusers/quantization/onnx_utils/export.py|
|
|
examples/llm_eval/lm_eval_hf.py|
|
|
examples/llm_eval/mmlu.py|
|
|
examples/llm_eval/modeling.py|
|
|
examples/llm_qat/train.py|
|
|
examples/llm_sparsity/weight_sparsity/finetune.py|
|
|
examples/specdec_bench/specdec_bench/models/specbench_medusa.py|
|
|
examples/speculative_decoding/main.py|
|
|
examples/speculative_decoding/medusa_utils.py|
|
|
examples/speculative_decoding/scripts/server_generate.py|
|
|
experimental/dms/models/qwen3/configuration_qwen3_dms.py|
|
|
experimental/dms/models/qwen3/modeling_qwen3_dms.py|
|
|
)$
|
|
|
|
# Default hook for Apache 2.0 in c/c++/cuda files
|
|
- id: insert-license
|
|
alias: insert-license-c
|
|
args:
|
|
- --license-filepath
|
|
- ./LICENSE_HEADER
|
|
- --comment-style
|
|
- "/*| *| */"
|
|
- --allow-past-years
|
|
types_or: [c++, cuda, c]
|
|
|
|
- repo: https://github.com/PyCQA/bandit
|
|
rev: 1.7.9
|
|
hooks:
|
|
- id: bandit
|
|
args: ["-c", "pyproject.toml", "-q"]
|
|
additional_dependencies: ["bandit[toml]"]
|
|
|
|
- repo: local
|
|
hooks:
|
|
- id: generate-arguments-md
|
|
name: Regenerate examples/llm_qat/ARGUMENTS.md
|
|
entry: bash -c 'python examples/llm_qat/arguments.py --generate_docs examples/llm_qat/ARGUMENTS.md'
|
|
language: system
|
|
files: >-
|
|
(?x)^(
|
|
examples/llm_qat/arguments\.py|
|
|
modelopt/torch/distill/plugins/huggingface\.py|
|
|
modelopt/torch/opt/plugins/transformers\.py|
|
|
modelopt/torch/quantization/plugins/transformers_trainer\.py
|
|
)$
|
|
pass_filenames: false
|
|
|
|
- repo: https://github.com/DavidAnson/markdownlint-cli2
|
|
rev: v0.18.1
|
|
hooks:
|
|
- id: markdownlint-cli2
|
|
args: ["--fix"]
|
|
|
|
##### Manual hooks (Expect many false positives)
|
|
# These hooks are only run with `pre-commit run --all-files --hook-stage manual <hook_id>`
|
|
|
|
# Spell checker
|
|
- repo: https://github.com/crate-ci/typos
|
|
rev: v1.35.8
|
|
hooks:
|
|
- id: typos
|
|
stages: [manual]
|
|
|
|
# Link checker
|
|
- repo: https://github.com/lycheeverse/lychee.git
|
|
rev: v0.15.1
|
|
hooks:
|
|
- id: lychee
|
|
args: ["--no-progress", "--exclude-loopback"]
|
|
stages: [manual]
|