mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-02 03:14:52 +08:00
Izzy Putterman
2802302bf9
## What does this PR do? **Type of change:** ? <!-- Use one of the following: Bug fix, new feature, new example, new tests, documentation. --> **Overview:** Addition of SpecBench Dataset Addition of NVIDID SPEED-Bench dataset, preproc scripts, and custom metrics aggregator Addition of example of converting SpecBench Medusa to this FW Addition of Initial TRTLLM AutoDeploy Specdec support Updates to all frameworks for better perf (overlap/async scheduling etc) ## Usage <!-- You can potentially add a usage example below. --> ```python # Add a code snippet demonstrating how to use this ``` ## Testing <!-- Mention how have you tested your change if applicable. --> ## Before your PR is "*Ready for review*" <!-- If you haven't finished some of the above items you can still open `Draft` PR. --> - **Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)** and your commits are signed. - **Is this change backward compatible?**: Yes/No <!--- If No, explain why. --> - **Did you write any new necessary tests?**: Yes/No - **Did you add or update any necessary documentation?**: Yes/No - **Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?**: Yes/No <!--- Only for new features, API changes, critical bug fixes or bw breaking changes. --> ## Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Added SPEED-Bench dataset support with configurable throughput and qualitative configurations * Introduced SpecBench metrics with acceptance rate analysis and visualizations * Added progress bar during benchmark execution * New model implementations for auto-deployment and Medusa-style speculative decoding * Data preparation utility for benchmark datasets * Enhanced metrics with per-category analysis and performance charts * **Documentation** * Updated README with SPEED-Bench workflow and examples * New porting guide for integrating custom benchmark runners * **Refactor** * Streamlined model and runner interfaces for improved flexibility * Consolidated dataset implementations and removed deprecated base classes * **Chores** * Added required dependencies for data handling and visualizations <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Izzy Putterman <iputterman@nvidia.com>
147 lines
5.2 KiB
YAML
147 lines
5.2 KiB
YAML
# NOTE: Make sure to update version in dev requirements (setup.py) as well!
|
|
repos:
|
|
- repo: https://github.com/pre-commit/pre-commit-hooks
|
|
rev: v6.0.0
|
|
hooks:
|
|
- id: check-added-large-files
|
|
args: [--maxkb=500, --enforce-all]
|
|
exclude: >
|
|
(?x)^(
|
|
examples/diffusers/quantization/assets/.*.png|
|
|
examples/diffusers/cache_diffusion/assets/.*.png|
|
|
)$
|
|
- id: check-json
|
|
exclude: ^.vscode/.*.json # vscode files can take comments
|
|
- id: check-merge-conflict
|
|
- id: check-symlinks
|
|
- id: check-toml
|
|
- id: mixed-line-ending
|
|
args: [--fix=lf]
|
|
- id: requirements-txt-fixer
|
|
|
|
- repo: https://github.com/astral-sh/ruff-pre-commit
|
|
rev: v0.12.11
|
|
hooks:
|
|
- id: ruff-check
|
|
args: [--fix, --exit-non-zero-on-fix]
|
|
exclude: ^examples/specdec_bench/specdec_bench/datasets/speed\.py$
|
|
- id: ruff-format
|
|
exclude: ^examples/specdec_bench/specdec_bench/datasets/speed\.py$
|
|
|
|
- repo: https://github.com/pre-commit/mirrors-mypy
|
|
rev: v1.17.1
|
|
hooks:
|
|
- id: mypy
|
|
|
|
- repo: https://github.com/pre-commit/mirrors-clang-format
|
|
rev: v21.1.0
|
|
hooks:
|
|
- id: clang-format
|
|
types_or: [c++, c, c#, cuda, java, javascript, objective-c, proto] # no json!
|
|
args: ["--style={ColumnLimit: 100}"]
|
|
|
|
- repo: https://github.com/pre-commit/pygrep-hooks
|
|
rev: v1.10.0
|
|
hooks:
|
|
- id: rst-backticks
|
|
- id: rst-directive-colons
|
|
- id: rst-inline-touching-normal
|
|
|
|
- repo: https://github.com/jumanjihouse/pre-commit-hook-yamlfmt
|
|
rev: 0.2.3
|
|
hooks:
|
|
- id: yamlfmt
|
|
args: [--mapping=2, --sequence=4, --offset=2, --implicit_start, --implicit_end, --preserve-quotes]
|
|
exclude: ^.github/workflows/
|
|
|
|
# Instructions to change license file if ever needed:
|
|
# https://github.com/Lucas-C/pre-commit-hooks#removing-old-license-and-replacing-it-with-a-new-one
|
|
- repo: https://github.com/Lucas-C/pre-commit-hooks
|
|
rev: v1.5.5
|
|
hooks:
|
|
# Default hook for Apache 2.0 in python and shell files
|
|
- id: insert-license
|
|
alias: insert-license-py
|
|
args:
|
|
- --license-filepath
|
|
- ./LICENSE_HEADER
|
|
- --comment-style
|
|
- "#"
|
|
- --allow-past-years
|
|
types_or: [python, shell]
|
|
# NOTE: Exclude files that have copyright or license headers from another company or individual
|
|
# since we want to keep those above the license header added by this hook.
|
|
# Instead, we should manually add the license header to those files *after* the original header.
|
|
exclude: >
|
|
(?x)^(
|
|
modelopt/onnx/quantization/operators.py|
|
|
modelopt/onnx/quantization/ort_patching.py|
|
|
modelopt/torch/_deploy/utils/onnx_utils.py|
|
|
modelopt/torch/export/transformer_engine.py|
|
|
modelopt/torch/quantization/export_onnx.py|
|
|
modelopt/torch/quantization/plugins/attention.py|
|
|
modelopt/torch/speculative/eagle/utils.py|
|
|
modelopt/torch/speculative/plugins/transformers.py|
|
|
modelopt/torch/utils/plugins/megatron_mmlu.py|
|
|
examples/chained_optimizations/bert_prune_distill_quantize.py|
|
|
examples/deepseek/quantize_to_nvfp4.py|
|
|
examples/deepseek/ptq.py|
|
|
examples/diffusers/cache_diffusion/pipeline/models/sdxl.py|
|
|
examples/diffusers/quantization/onnx_utils/export.py|
|
|
examples/llm_eval/gen_model_answer.py|
|
|
examples/llm_eval/humaneval.py|
|
|
examples/llm_eval/lm_eval_hf.py|
|
|
examples/llm_eval/mmlu.py|
|
|
examples/llm_eval/modeling.py|
|
|
examples/llm_qat/main.py|
|
|
examples/llm_sparsity/weight_sparsity/finetune.py|
|
|
examples/specdec_bench/specdec_bench/models/specbench_medusa.py|
|
|
examples/speculative_decoding/main.py|
|
|
examples/speculative_decoding/medusa_utils.py|
|
|
examples/speculative_decoding/server_generate.py|
|
|
experimental/dms/models/qwen3/configuration_qwen3_dms.py|
|
|
experimental/dms/models/qwen3/modeling_qwen3_dms.py|
|
|
)$
|
|
|
|
# Default hook for Apache 2.0 in c/c++/cuda files
|
|
- id: insert-license
|
|
alias: insert-license-c
|
|
args:
|
|
- --license-filepath
|
|
- ./LICENSE_HEADER
|
|
- --comment-style
|
|
- "/*| *| */"
|
|
- --allow-past-years
|
|
types_or: [c++, cuda, c]
|
|
|
|
- repo: https://github.com/PyCQA/bandit
|
|
rev: 1.7.9
|
|
hooks:
|
|
- id: bandit
|
|
args: ["-c", "pyproject.toml", "-q"]
|
|
additional_dependencies: ["bandit[toml]"]
|
|
|
|
- repo: https://github.com/DavidAnson/markdownlint-cli2
|
|
rev: v0.18.1
|
|
hooks:
|
|
- id: markdownlint-cli2
|
|
args: ["--fix"]
|
|
|
|
##### Manual hooks (Expect many false positives)
|
|
# These hooks are only run with `pre-commit run --all-files --hook-stage manual <hook_id>`
|
|
|
|
# Spell checker
|
|
- repo: https://github.com/crate-ci/typos
|
|
rev: v1.35.8
|
|
hooks:
|
|
- id: typos
|
|
stages: [manual]
|
|
|
|
# Link checker
|
|
- repo: https://github.com/lycheeverse/lychee.git
|
|
rev: v0.15.1
|
|
hooks:
|
|
- id: lychee
|
|
args: ["--no-progress", "--exclude-loopback"]
|
|
stages: [manual]
|