mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-20 05:59:32 +08:00
pull-request/2240
80
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 gcunhaseandmodelopt-fix-agent-bot
|
521f030cd4 |
sensitivity: address CodeRabbit findings (picker collision, docs, CLI tests, utils move)
picker.py -- singleton-name collision fix
=========================================
Regression: when ``blocks=`` was set and an unmatched node's name happened to
equal a configured group name, ``_assign_groups``'s
``groups.setdefault(key, []).append(node_name)`` silently merged the unmatched
node into that group instead of keeping it as an isolated singleton. On a
graph like ``scores={"g_n1": 0.01, "g": 10.0}, blocks={"g": [r"^g_"]}``, the
standalone node ``g`` (score 10.0) would land inside the ``g`` group's member
list alongside ``g_n1``, inflating the group's aggregated score and altering
which groups the coverage / threshold picker selected.
Fix: prefix unmatched-node singleton keys with a sentinel (``"\0singleton:"``)
that cannot collide with any user-supplied group name. Group ranking still
receives correct (name, members) pairs; the sentinel is internal to the
``groups`` dict and does not leak into the return value (the picker returns
member node names, not group keys).
Added ``test_unmatched_node_name_collision_with_group_name_stays_isolated``
in ``test_picker.py`` as a regression witness.
utils.py -- host ``validate_file_size``
=======================================
``modelopt.onnx.quantization.sensitivity.__main__`` imported ``validate_file_size``
from ``modelopt.onnx.quantization.__main__``, pulling the main quantize CLI's
full import graph (``autotune.utils``, ``quantize``, ...) into the sensitivity
CLI just to reuse a 15-line size check. If the main CLI later grows a heavier
import, the sensitivity CLI breaks silently.
Moved ``validate_file_size`` to ``modelopt/onnx/utils.py`` alongside the other
low-level ONNX helpers. Both CLIs now import from ``modelopt.onnx.utils``. No
behavior change.
docs -- coverage-mode wording + sample output
=============================================
The ``_onnx_quantization.rst`` guide and ``examples/onnx_ptq/README.md`` still
described coverage mode as "excludes the largest node set whose cumulative
sensitivity score stays at or below ``coverage * total_mass``" -- the same
phrasing the ``suggest_exclusion`` docstring was corrected away from in
an earlier round. Synced to the rank-prefix wording ("walks targets in
descending score order and accumulates them until the next one would push
cumulative sensitivity above ``coverage * total_mass``").
The rendered-ranking example in ``_onnx_quantization.rst`` also could not
be produced by ``_render_ranked_table()``: it mixed ``~0`` string values
(``_render_ranked_table`` uses ``{value:.3f}``, never ``~0``) with a
visible ``Gemm 0 <-- lowest impact`` row (zero-score rows are hidden by
default) and a ``1 target(s) hidden`` footer. Regenerated the sample to
match actual CLI output: drop the ``~0`` rows, move the lowest-impact
marker to the last visible non-zero row (MatMul 0.015), and update the
hidden-count footer to 4.
test_cli.py -- coverage for previously untested CLI helpers
===========================================================
New ``tests/unit/onnx/quantization/sensitivity/test_cli.py`` covers the CLI
functions that landed in earlier rounds but had no unit tests:
* ``_default_output_json``: absolute-path and relative-path derivation.
* ``_validate_calibration_dir``: empty dir raises, under-limit dir passes,
per-file cap trips, aggregate cap trips.
* ``_render_ranked_table``: no scores + no failed, no scores + all failed,
normal ranking with hidden zeros, ``show_zero_scores=True``, all-zero
footer, failed footer.
* ``main()``: synthetic-calibration happy path, default output-JSON path,
oversize ONNX rejected before ``score()`` is called, calibration-directory
path validated before ``score()`` runs.
17 tests total, all pure Python (``score`` monkeypatched, no GPU / no real
ONNX quantization). Wall-clock < 1 s.
test_quantize_api.py -- hoist local imports
===========================================
Moved ``onnx_graphsurgeon as gs`` and ``modelopt.onnx.utils.save_onnx``
from inside ``test_quantize_honors_nodes_to_quantize_allowlist`` to
module scope. Neither is optional or circular.
Signed-off-by: gcunhase <4861122+gcunhase@users.noreply.github.com>
Co-Authored-By: modelopt-fix-agent-bot (Opus 4.7) <noreply@anthropic.com>
|
||
|
gcunhaseandmodelopt-fix-agent-bot
|
5e2b0947dc |
sensitivity: address CodeRabbit review batch (docs, correctness, CLI hardening, tests)
Fixes the 11 CodeRabbit findings that survived triage on PR #2240. Two Security/Major items (public-API artifact size guards on :func:`score`; removing target names from logs) were consciously deferred per author call. Documentation ============= * :pr:`comment:3883483129` -- ``docs/.../_onnx_quantization.rst``: ``[expr for i, ex in enumerate(ds) if i < 500]`` doesn't stop the streaming iterator -- Python evaluates every element of ``ds`` against the ``i < 500`` filter, so users copy-pasting the doc drain the full 50k-image ImageNet-1k validation split each time. Replaced with ``itertools.islice(ds, 500)``. * :pr:`comment:3883483137` -- picker examples referenced ``scores`` but the preceding block only defined ``result``; copying the block raised ``NameError``. Added ``scores = result["scores"]`` bridging line. * :pr:`comment:3883483146` -- CLI ``--metric`` help said "activations", but :func:`score` runs both graphs through ORT and compares **graph outputs**. Same imprecision propagated into the ``Metric`` enum docstring and the ``metrics.py`` module docstring. Standardised on "graph outputs" across all three. * :pr:`comment:3883483177` -- ``op_types_scope`` docstring still said unsupported targets are scored ``0.0``; the earlier ``failed``-field rework moved them to a distinct list. Doc now documents the new behavior and the "0.0 means free / ``failed`` means we don't know" distinction. Correctness =========== * :pr:`comment:3883483142` -- ``_render_ranked_table`` printed "no quantizable targets found" whenever ``scores`` was empty, hiding the case where every probe failed. Split into three branches: empty + no failed = "no targets"; empty + failed = "no scores produced; N target(s) failed"; non-empty + failed adds a trailing "N target(s) failed to probe" line. * :pr:`comment:3883483161` -- ``cos_dist`` returned ``1.0`` when both reference and quantized outputs were all-zero (``0 / (0 + _EPS) = 0`` -> distance 1). A probe whose outputs are both zero should score as identical, not maximally sensitive. Guarded with ``np.where(norm > 0, ..., 1.0)`` so ``distance == 0.0``. Added ``test_both_zero_vectors_return_zero_distance`` as a regression witness in ``test_metrics.py``. * :pr:`comment:3883483169` -- :func:`suggest_exclusion` coverage-mode docstring said "largest target set whose cumulative score stays at or below ``coverage * total_mass``", but the implementation is a rank-prefix walker that stops at the first non-fitting target (e.g. ``{a: 4, b: 3, c: 2}`` at ``coverage=0.7`` returns ``[a]`` even though ``{a, c}`` fits with more members). Reworded to describe the actual algorithm and its non-largest-set trade-off. * :pr:`comment:3883483175` -- :func:`summarize_exclusion` skipped unknown ``excluded`` names in ``excluded_mass`` but counted them in ``num_excluded`` / ``num_remaining_quantized``, so ``scores={"a":5,"b":5}`` + ``excluded=["a","unknown"]`` yielded ``num_remaining_quantized == 0`` even though ``b`` remained quantized. Duplicate excluded names skewed the counts too. Filter ``excluded`` to unique names in ``scores`` before counting so all four fields stay consistent. Updated ``test_missing_node_names_default_zero`` (previously pinned the buggy behaviour) and added ``test_duplicate_excluded_names_counted_once``. * :pr:`comment:3883483188` -- ``test_failed_probe_is_recorded`` ``monkeypatch.setattr(score_mod, "quantize", ...)`` targeted the package-level re-exported function object (which ``score_mod`` was bound to via ``from modelopt.onnx.quantization.sensitivity import score``), not the ``modelopt.onnx.quantization.sensitivity.score`` module namespace that :func:`score` actually resolves ``quantize`` from. The fake was never called. Fixed by importing the implementation submodule directly and patching that module's ``quantize`` binding. Also added ``test_failed_probe_records_exceptions`` covering the sibling code path where ``quantize()`` raises rather than silently no-ops. Package structure ================= * :pr:`comment:3883483140` -- ``sensitivity/__init__.py`` converted to the ``from .module import *`` re-export pattern used by ``modelopt/torch/quantization/qtensor/`` and other subpackages, keeping the file-scope ``# ruff: noqa: F405`` and the explicit ``__all__`` public-contract list (same shape as ``modelopt/torch/quantization/utils/__init__.py``). ``picker.py`` gained an ``__all__`` list so its wildcard re-export ships exactly ``suggest_exclusion`` + ``summarize_exclusion``. CLI hardening ============= * :pr:`comment:3883483152` -- directory-mode ``--calibration_data_path`` bypassed size validation entirely, and ``_load_calibration_from_path`` then loaded every ``.npz`` shard under it and concatenated the arrays without per-file or aggregate limits. Added ``_validate_calibration_dir()`` that enforces a per- file cap of ``_CALIB_MAX_SIZE_BYTES`` (4 GiB, same as the file-mode guard) and an aggregate cap of ``_CALIB_DIR_MAX_TOTAL_BYTES`` (16 GiB). Empty directories now raise ``FileNotFoundError``. Logging (partial) ================= * :pr:`comment:3883483182` -- scrubbed the raw ``onnx_path`` from the scan-start log line and moved raw exception text to a ``logger.debug(..., exc_info=True)`` on the ``quantize()`` failure path. The per-probe warning still surfaces the exception **class** and the target name (target names are load-bearing for debugging; callers who want them redacted can filter at the log-handler layer). pyproject.toml ============== Registered the ``slow`` and ``slow_gpu`` markers in ``[tool.pytest.ini_options] markers = [...]`` so ``--strict-markers`` (project default via ``addopts``) doesn't fail collection on ``@pytest.mark.slow`` / ``@pytest.mark.slow_gpu`` decorators. These were already used on the CoAtNet integration tests but the marker declarations had never been included in a shipping commit. Deferred (author call, documented in the review threads) ======================================================== * :pr:`comment:3883483180` -- public-API artifact size guards on :func:`score`. Requires expanding the API contract (raise on legitimately large ONNX inputs OR plumb a trust knob through). Not in scope for this PR. * Target-name scrubbing from log messages -- keeping them is a deliberate debuggability choice. Signed-off-by: gcunhase <4861122+gcunhase@users.noreply.github.com> Co-Authored-By: modelopt-fix-agent-bot (Opus 4.7) <noreply@anthropic.com> |
||
|
gcunhaseandmodelopt-fix-agent-bot
|
b53cce4b2d |
docs/picker/tests: post-review simplifications across the sensitivity module
Review-driven cleanup after the block-picker + ruff-format commits.
Zero API changes, zero behavior changes; documentation, docstring, and
test-scaffolding trims only.
CHANGELOG.rst
=============
Move the ``modelopt.onnx.quantization.sensitivity`` entry from the
Megatron subsection to Quantization (where it belongs) and mention the
optional block-level aggregation (``blocks=`` / ``block_agg=``) so the
line covers the block-picker feature that landed alongside the primitive.
docs/source/guides/_onnx_quantization.rst
=========================================
* Opening (128-138): compressed 11 lines of marketing-toned framing
("runs into the same friction ... hand-crafted exclusion policies
until they find one that works", "Works across CNN, Transformer, and
hybrid architectures alike") down to a 4-line spec of what
``sensitivity.score`` does.
* Supported-options bullets (142-161): trimmed the parenthetical
restatements from ``metric`` / ``calibration_method`` / ``calibration_data`` /
``op_types_scope`` -- the details are in the ``score`` docstring.
* Synthetic-calibration note: dropped the "random Q times K^T produces
near-uniform softmax that hides real-input MHA quantization pathology"
overexplanation; the "directional-only; attention-heavy is the
highest-risk case" punchline is enough.
* Per-node granularity paragraph: deleted (the ``granularity`` bullet
above already covers it).
* Meta-narration: dropped "In the rest of this documentation, we'll
cover per-node granularity for simplicity, but the same logic goes
for per-op-type granularity."
* Coverage / threshold bullets (274-286): 13-line bulleted restatement
of ``suggest_exclusion``'s docstring cut to 6 lines pointing the
reader at the docstring for full argument reference.
* Block-picker intro (333-350): dropped "and softmax numerics degrade
catastrophically" hedge + ``block_agg`` docstring parenthetical.
* Near-tie note: 7-line bulleted restatement of the picker docstring
compressed to 3 lines with the actual guidance in one sentence.
* Empirically-recovers phrasing: replaced "closing the ViT-tiny parity
gap to implicit quantization" (jargon) with the direct number
comparison "~75% top-1 versus ~60% for the best per-node picking".
* When per-block picking doesn't help: dropped "typically" and
"Reach for ``blocks`` first" chatty wording.
* CoAtNet timm-export prep: dropped the "Use the analogous timm handle
for any other model family" filler sentence.
modelopt/onnx/quantization/sensitivity/picker.py
================================================
* Module docstring: 8 lines -> 1 sentence ("Turn a sensitivity score
dictionary into an exclusion list, with optional block-level
aggregation.").
* ``suggest_exclusion`` docstring: 55 lines -> 32 lines. Consolidated
the coverage/threshold trade-off restatement (previously in the mode
summary + the ``coverage`` arg + a free-standing paragraph); the
standalone paragraph moved to the RST guide. ``block_agg`` collapsed
from a 12-line bullet forest to a 4-line paragraph -- the natural
pairing table (``sum`` with ``coverage``, ``max`` with ``threshold``)
is preserved.
* ``_aggregate_group_scores``: dropped ``if members else 0.0`` defensive
ternary. ``groups`` is built by ``setdefault().append()`` so every
key is guaranteed at least one member.
* ``_warn_near_tie``: renamed local variables from ``last_included_kl``
/ ``first_excluded_kl`` to ``last_included_score`` /
``first_excluded_score`` -- this module supports MSE and cos too, so
the ``_kl`` suffix was metric-specific and misleading. Rewrote the
awkward "helps guiding the user into adjusting coverage or threshold"
docstring line as "The warning prompts widening ``coverage`` or
narrowing ``threshold``".
* ``_pick_from_scores``: dropped ``# Threshold mode`` / ``# Coverage
mode`` block comments -- the ``if threshold is not None:`` branch is
self-labeling.
* ``summarize_exclusion``: dropped ``float(...)`` cast on
``scores.get(...)`` -- ``scores`` is already
``Mapping[str, float]``. Collapsed the 12-line bulleted Returns block
to a single paragraph.
modelopt/onnx/quantization/sensitivity/score.py
===============================================
* Module docstring: 7 lines -> 4 lines ("Core sensitivity primitive:
rank quantizable targets by per-target Q/DQ drift.").
* ``_default_op_types_scope`` docstring: 11 lines -> 5 lines. Kept the
load-bearing rationale (copy ops excluded because TRT never produces
INT8 kernels for them) but dropped the narrative "clutters the
output with 'don't do this anyway' entries".
* ``op_types_scope`` argument in ``score()``'s docstring: removed the
CLI-specific "hides those from the pretty-printed table by default"
aside -- that behavior is documented in ``__main__.py``.
modelopt/onnx/quantization/sensitivity/__init__.py
==================================================
Module docstring: 8 lines -> 1 sentence ("ONNX quantization sensitivity:
rank quantizable targets by per-target Q/DQ drift.").
modelopt/onnx/quantization/sensitivity/__main__.py
==================================================
* Removed the defensive
``assert result["calibration_source"] in {c.value for c in CalibrationSource}``
along with its "score() already emits the enum's string value"
comment -- ``score()`` sets that field from the enum's own ``.value``
so the assert was checking that ``score()`` doesn't lie about its
own contract. ``CalibrationSource`` is no longer imported here
(still ships in the public API via ``__init__.py``).
* ``show_zero_scores`` docstring: 4-line "graph plumbing" explanation
trimmed to a one-line statement of what the flag does.
* ``--show_zero_scores`` CLI help: same treatment.
* ``_load_calibration`` docstring: dropped "matches what the main
quantize CLI does" background parenthetical.
modelopt/onnx/quantization/sensitivity/metrics.py
=================================================
* Module docstring: 8 lines -> 1 sentence ("Proxy metrics between
reference and quantized activations. Higher = more distortion.").
* ``kl_div`` / ``mse`` / ``cos_dist`` docstrings: property-first. Each
metric now leads with its scale-sensitivity property ("Robust to
activation magnitude scale.", "Sensitive to activation magnitude
scale.", "Scale-invariant.") instead of comparative narration
("recommended default because ...", "a target whose output happens
to be large in absolute value will look more sensitive under MSE than
under KL / cosine", etc.).
tests/gpu/onnx/quantization/test_sensitivity.py
================================================
* Added a module-scoped ``coatnet_fixtures`` fixture that
``pytest.skip``\ s cleanly when the pre-staged CoAtNet-0 ONNX +
``imagenet_calib_500.npz`` are absent. Both integration tests now
receive the tuple instead of duplicating the ``_require_fixture``
calls.
* Trimmed ``test_coatnet_op_type_matches_manual_groundtruth``'s
docstring: the 13-row op-type ranking table is already documented in
the RST guide, so the test docstring just states the top-4 assertion
invariant.
* Renumbered the tiers **by cost**: Tier 1 is now the fast
synthetic-random regression guard, Tier 2 is the fast
synthetic-real deterministic test, Tier 3 is the CoAtNet op-type
integration, and Tier 4 is the CoAtNet per-node integration. The
previous numbering interleaved a fast fallback test as Tier 4 after
two slow real-model tests, which was harder to scan. The two
synthetic tests are also reordered in the source file to match. All
docstring ``Tier N:`` labels, the module-header tier list, and the
``coatnet_fixtures`` fixture docstring reference ("for tier 3 / 4
tests") were updated in the same pass.
* Fixed a stale env-var reference in the same trimmed docstring:
``MODELOPT_SENSITIVITY_FIXTURES`` never existed; the test reads
``MODELOPT_ONNX_ACCURACY_MODELS_DIR`` (see line 48).
tests/unit/onnx/quantization/test_sensitivity_picker.py
========================================================
Hoisted ``import logging`` to module scope -- the four
``TestNearTieWarning`` tests each imported it inside their body. Net
-3 lines / +1 line, and the ``caplog.at_level(logging.WARNING, ...)``
calls now reference the module-level module correctly.
Validation
==========
``ruff check`` + ``ruff format`` clean on all touched files. ``mypy``
clean on the five source files. ``pytest`` still runs on the fork's
own env (Computelab); locally we only sanity-check ruff + mypy.
Signed-off-by: gcunhase <4861122+gcunhase@users.noreply.github.com>
Co-Authored-By: modelopt-fix-agent-bot (Opus 4.7) <noreply@anthropic.com>
|
||
|
gcunhaseandClaude Opus 4.7
|
86c6ee2357 |
picker/docs: simplify docstrings; swap ViT block-picker example to max + threshold
Post-review cleanup after the block-picker feature landed. Three files touched, all documentation / comment simplification -- zero API changes, zero behavior changes. picker.py ========= Compressed verbose bulleted guidance in ``suggest_exclusion``'s docstring without dropping semantic content: * Module docstring reduced from 20 to 5 lines (single-paragraph summary). * ``coverage`` guidance: three bulleted sub-ranges collapsed to a one-line paragraph (``0.85-0.90`` balances, ``0.95-0.99`` favors accuracy, ``0.70-0.80`` favors latency). * ``threshold`` guidance: bulleted per-architecture magnitudes collapsed to a one-line paragraph. * ``blocks`` docstring: kept all rules (first-match wins, singleton fallback, group ranking replaces node ranking, expansion at return) but stripped repeated framing. * ``block_agg`` docstring: kept the two natural-pairing bullets and the paragraph explaining unit shifts under off-diagonal combinations, but removed a sentence-level restatement of each bullet. * ``max_nodes``, ``min_score_floor``, ``near_tie_ratio``: one-paragraph descriptions. Net -170 lines added / +79 lines removed. The extracted helpers (``_pick_from_scores``, ``_assign_groups``, ``_aggregate_group_scores``, ``_warn_near_tie``) are functionally identical to what landed in a3584c81. __main__.py =========== * Removed "Mirrors the flag style of ``python -m modelopt.onnx.quantization.autotune``." sentence from the module docstring. * Shrunk the ``CalibrationSource`` assert comment from two lines to one: "Sanity-check the JSON schema; score() already emits the enum's string value." _onnx_quantization.rst ====================== Revisions after empirical validation of the block-picker recipe against the tested 101-node ViT-tiny hot region: * Trimmed the "primitive reuses ``quantize`` internally, so scales are properly calibrated (not autotune's placement-only descriptors)" clause to just "The primitive reuses ``quantize`` internally for each per-target probe." The autotune-contrast note was a maintainer-facing detail that did not belong in the user guide. * Corrected the ``calibration_method`` bullet from "``entropy`` (default), ``max``, ``mse``, ``percentile``, etc." to the honest "``entropy`` (default) or ``max``". The ONNX quantize path in ``int8.py`` / ``fp8.py`` only dispatches on ``entropy`` vs falls-through-to-MinMax; ``mse`` and ``percentile`` are silently degraded to MinMax with no error. ``PercentileCalibrater`` exists in ``ort_patching.py`` but is not reachable from the public ``calibration_method`` argument. * Tightened ``granularity`` and other bullet-list descriptions. * Removed a redundant "``calibration_source`` field of the output JSON records which mode was used" sentence. * Rewrote the ViT-tiny block-picker example to use the natural ``max`` + ``threshold`` pairing recommended in the picker's docstring instead of ``sum`` + ``max_nodes``. Validated empirically: ``suggest_exclusion(scores, threshold=0.1, blocks=blocks, block_agg="max")`` produces the same 101-node exclusion set as the hand-curated regex-union (checked node-by-node against the vit_tiny_hotregion_blocks_7_11_plus_norm exclusion) and recovers 74.80% top-1 (within 1pp variance of the earlier 75.20% measurement). * Consolidated the two rendered rankings (``max_agg`` and ``sum_agg``) into one side-by-side table so the reader sees both aggregations of the same data at once. Added a note explaining that either ``max + threshold=0.1`` or ``sum + coverage=1.0, max_nodes=6`` picks the same six groups, with a small internal-ordering difference on blocks.9 vs blocks.11 that does not affect the final selection. Signed-off-by: gcunhase <4861122+gcunhase@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
gcunhaseandClaude Opus 4.7
|
1eb71aec75 |
picker: add block-aware exclusion via optional blocks + block_agg args
Extends ``suggest_exclusion`` with two new keyword arguments so operators can
turn per-node sensitivity scores into a block-level exclusion set without
reimplementing the coverage / threshold / near-tie logic themselves. Motivated
by empirical validation on ViT-tiny where per-node picking hits a ~60% top-1
ceiling due to intra-block precision fragmentation; block-level exclusion
recovers ~75% top-1 (within 1pp of native ``trtexec --int8 --fp16``).
New API surface:
* ``blocks: Mapping[str, Sequence[str | re.Pattern]] | None = None`` -- maps
group name to a list of regex patterns matching node paths. Each node is
assigned to at most one group (first-match wins across the ``blocks``
dict). Nodes matching no pattern become their own singleton group named
after themselves, so architecturally-important standalone nodes (final
``LayerNormalization`` before the head, patch-embed ``Conv``, etc.) compete
for exclusion on equal footing with multi-node blocks. When ``blocks`` is
``None`` (default) the picker behaves exactly as before -- fully
backward-compatible.
* ``block_agg: Literal["sum", "max", "mean"] = "sum"`` -- aggregation used to
compute a group's score from its members. Kept as ``Literal`` for
IDE/mypy support, plus a runtime ``ValueError`` in ``suggest_exclusion``
for defensive validation.
Semantics:
* When ``blocks`` is set, the picker computes per-group aggregated scores
and applies coverage / threshold / near-tie / ``max_nodes`` /
``min_score_floor`` semantics identically to the per-node path. The
returned exclusion list is the union of member node names across the
selected groups, ready to pass as ``nodes_to_exclude=`` to
``modelopt.onnx.quantization.quantize``.
* Natural pairings between ``block_agg`` and picker mode -- documented in
the docstring and RST guide:
- ``block_agg="sum"`` with ``coverage`` (recommended default): identical
"fraction of total KL mass" semantic as per-node coverage. Portable
across per-node and per-block grouping on the same model.
- ``block_agg="max"`` with ``threshold``: same units as per-node
threshold (excludes any group whose peak-node score exceeds the
cutoff). Preserves operator intuition when transferring per-node
threshold guidance to the block level.
- Other combinations remain valid but change what ``coverage`` and
``threshold`` mean in units; the docstring calls this out explicitly.
Implementation notes:
* The existing per-node core (coverage / threshold / near-tie logic) is
extracted into a private ``_pick_from_scores`` helper. Both the per-node
and per-block paths call it, so both share identical semantics for every
future behavior change. Zero duplication.
* Two additional private helpers: ``_assign_groups`` (regex-based
first-match-wins assignment with singleton fallback) and
``_aggregate_group_scores`` (dispatches on ``block_agg``).
* Backward compatibility: every existing ``suggest_exclusion`` call site
behaves exactly as before because ``blocks`` defaults to ``None`` and the
``block_agg`` value is only inspected when ``blocks`` is set.
Documentation:
* ``docs/source/guides/_onnx_quantization.rst`` gains a new subsection at
the end -- "Grouping per-node scores into architectural blocks" -- with:
- A ``vit_tiny_patch16_224`` (timm) worked example at ``coverage=0.95``
with ``block_agg="sum"`` that selects blocks 8, 10, 9, 11, 7 (~100
nodes across 5 whole transformer blocks), which recovers ~75% top-1
on ImageNet-1k versus ~60% for the best per-node picking.
- A depth-2 example showing how to split each transformer block into
``blocks.N.attn`` and ``blocks.N.mlp`` sub-groups.
- Guidance on the ``block_agg`` / picker-mode pairings.
- A "when block-level grouping doesn't help" note calling out
Conv-heavy architectures (MobileNet, ResNet families) where diffuse
per-node sensitivity means the per-node picker still wins.
Signed-off-by: gcunhase <4861122+gcunhase@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
gcunhaseandClaude Opus 4.7
|
0e4227812c |
onnx: add sensitivity primitive with exclusion picker for per-op-type / per-node PTQ ranking
Adds ``modelopt.onnx.quantization.sensitivity``, a per-op-type or per-node
accuracy sensitivity ranking primitive for ONNX PTQ, plus a coverage / threshold
based exclusion picker that turns the ranking into an actionable
``--nodes_to_exclude`` or ``--op_types_to_exclude`` list.
Primitive (sensitivity.score):
- For each quantizable target (op type or individual node), invokes the existing
``modelopt.onnx.quantization.quantize`` entry point to insert calibrated Q/DQ
on just that target, runs the reference and quantized ONNXs through
ONNXRuntime on the same calibration inputs, and computes a proxy metric
between the two graph-output activation sets. Higher score means the target
adds more accuracy loss if quantized -- so callers keep high-scoring targets
at higher precision.
- Op-type granularity via ``--op_types_to_quantize`` (fast; ~10-15 probes on a
typical graph); per-node granularity via ``--nodes_to_quantize`` regex (deep
dive; N_nodes probes).
- Three proxy metrics via ``metrics.py``: kl_div (default, softmax-normalized),
mse (raw), cos_dist (1 - cosine_similarity).
- Real calibration data via .npy / .npz / directory path, or synthetic random
fallback (directional-only; warned in the CLI and marked in the output JSON's
calibration_source field).
- Default op_types_scope excludes layout / copy ops via
``modelopt.onnx.op_types.is_copy_op`` -- Transpose / Reshape / Concat and
friends show up in ORT's default quantizable set, but their sensitivity
signal reflects Q/DQ insertion at data-movement boundaries rather than any
INT8-kernel trade-off, so ranking them clutters the output with
"don't do this anyway" entries.
Picker (sensitivity.suggest_exclusion, sensitivity.summarize_exclusion):
- Coverage mode (default): return the largest target set whose cumulative
sensitivity score stays at or below ``coverage * total_mass``. The actual
coverage is always less than or equal to the requested value ("at most X%"),
so the operator never gets more exclusion than they asked for.
Architecture-portable because the target is a fraction, not an absolute
number.
- Threshold mode: return every target whose individual sensitivity score
exceeds ``threshold``. Simpler and more predictable when the operator
already knows what per-target sensitivity magnitude they consider
"too sensitive to quantize" for a specific model.
- ``near_tie_ratio`` (default 0.99) emits a logger.warning when the cut-off
between included and excluded targets is a near-tie -- flags potential
intra-group precision fragmentation. Set to ``None`` to disable.
- Companion ``summarize_exclusion`` reports the effect of an exclusion set:
coverage_pct, num_excluded, num_previously_quantized, num_remaining_quantized,
excluded_mass, total_mass.
Public surface:
- Python: ``from modelopt.onnx.quantization.sensitivity import score,
suggest_exclusion, summarize_exclusion``.
- CLI: ``python -m modelopt.onnx.quantization.sensitivity --onnx_path=...
--calibration_data_path=... --granularity=op_type --metric=kl_div``.
- Output JSON schema includes scores, calibration_source,
num_calibration_samples, metric, granularity, target_precision.
Tests:
- ``tests/gpu/onnx/quantization/test_sensitivity.py``: synthetic-graph tier
(real deterministic inputs -> LayerNormalization scores above Conv) plus a
synthetic-random regression tier (calibration_data=None still preserves the
directional invariant), plus CoAtNet-0 op-type and per-node integration
stubs marked @pytest.mark.manual (gated by --run-manual and a
MODELOPT_SENSITIVITY_FIXTURES env var).
- ``tests/unit/onnx/quantization/test_sensitivity_picker.py``: coverage /
threshold / min_score_floor / max_nodes / near-tie warning tests.
- ``tests/unit/onnx/quantization/test_nodes_to_quantize.py``: validates the
existing ``--nodes_to_quantize`` include-only flag that per-node granularity
relies on.
Documentation:
- ``docs/source/guides/_onnx_quantization.rst``: new "Quantization Sensitivity
Scan" chapter plus a "Turning scores into an exclusion list" subsection with
both policy modes, ``:ref:`` cross-reference to the metric options, and a
note that the picker documentation assumes per-node granularity for
simplicity but the same logic applies to per-op-type.
Signed-off-by: gcunhase <4861122+gcunhase@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
Keval MorabiaandClaude Opus 5
|
d73278808b |
Bump nemo container requirement to 26.08 for MBridge examples (#2257)
### What does this PR do? Type of change: Bug fix Bumps the Megatron-Bridge examples, tests and launcher configs to `nemo:26.08` and removes the version-gated fallbacks they carried, plus the fixes needed to make the suites green on that container. **26.08 bump and shim removal** - Examples, CI workflows, `noxfile.py` and the `mbridge_*` launcher configs move to `nemo:26.08`. - `examples/megatron_bridge/_distillation_provider.py` is deleted — 26.08's Megatron-Bridge ships `convert_to_distillation_provider(..., distill_submodule=...)` natively, so `distill.py` imports it directly. - `prune_minitron.py` drops the `AutoBridge.from_hf_config` / config-only-export probing; `--no_moe_grouped_gemm` is no longer needed in the MoE pruning tests, and the Qwen3.5-MoE `skipif` is gone (native MoE expert mappings are in 26.08). - `_DynamicMambaMixer` targets only the raw `conv1d_weight` / `conv1d_bias` parameters that replaced the `conv1d` module in Megatron-Core. **MambaModel / MambaModelProvider removal** Megatron-Core has shipped `HybridModel` since 26.06 and `MambaModel` is a deprecated subclass that shares its `forward`, so `DMRegistry` resolves those instances to the `HybridModel` registration and the separate entry is redundant. Same for `MambaModelProvider` vs `HybridModelProvider` on the bridge side. `MambaMixer` / `MambaLayer` / `ExtendedRMSNorm` are untouched — the layers still exist. The deprecated `get_te_mamba_stack_spec` is removed; use `get_te_hybrid_stack_spec`. **Bug fix: compressed output_layer extra state** `mtq.compress` converts even a *disabled* `output_layer` into a `RealQuantLinear` (its weight is left uncompressed, since `pack_real_quantize_weight` skips disabled quantizers). The guard added in #2112 excluded `RealQuantLinear` by class, so the layer emitted extra state and every worker died in `GPTModel.sharded_state_dict`: ``` RuntimeError: Boolean value of Tensor with more than one value is ambiguous megatron/core/models/gpt/gpt_model.py:896, in sharded_state_dict output_extra_state and output_extra_state.data ``` The guard now keys off whether the weight was actually compressed (`QTensorWrapper`) instead of the class. This took out all 12 `test_homogeneous_compressed_sharded_state_dict` params, and the crashed workers poisoned the pool, which surfaced as unrelated timeouts and NCCL errors in `test_layer_sync_moe_local_experts_amax`, `test_kv_cache_quant`, `test_kv_cache_amax_sync`, `test_convert_mcore_te_gpt_model` and `test_homogeneous_sharded_state_dict_te_spec` — 21 tests in total. The e2e coverage is `skip_flaky_on_blackwell`, so CI never ran it; `test_output_layer_extra_state_empty_when_nothing_quantized` now asserts the contract directly and is not skipped. **Checkpoint import entry point** 26.08 replaced `examples/conversion/convert_checkpoints.py` with `scripts/conversion/convert.sh`, so `tools/launcher/common/megatron_bridge/import/import.sh` and the three README snippets are retargeted. `import.sh` uses the distributed GPU backend with `GPUS_PER_NODE` / `TP` / `PP` / `EP` knobs. **Megatron-LM on nemo:26.06** keeps working: `_get_mamba_conv1d` still dispatches between the `conv1d` module (26.06 and earlier) and the raw parameters (26.08+), so `import_mcore_gpt_from_hf` / `export_mcore_gpt_to_hf` handle NemotronH on both. Only the Megatron-Bridge examples and Minitron pruning of Mamba/hybrid models require 26.08. **Test consolidation** `test_export_distilled_megatron_to_hf.py` is merged into `test_distill.py`: `test_distill_llm` becomes `test_distill_llm_hf_export` and covers the standalone `--export_iterations all` run on the checkpoints it already produces, saving one full distillation (~185 s of CI time). The two mamba-named gpu test files are renamed to `hybrid`. ### Usage ```bash # HF -> Megatron import, via Megatron-Bridge's 26.08 conversion entry point bash /opt/Megatron-Bridge/scripts/conversion/convert.sh import \ --executor local \ --device gpu \ --gpus-per-node 8 \ --hf-model Qwen/Qwen3-8B \ --megatron-path /tmp/Qwen3-8B-megatron ``` ### Testing All on `nvcr.io/nvidia/nemo:26.08`, 2x RTX 6000 Ada, no timeout overrides: - `tests/examples/megatron_bridge`: 16 passed, 1 skipped (28m14s). The skip is the `gemma3vl` QAD param, now `@pytest.mark.manual` since `qwen3_5_moe_vl` covers the VLM QAD path. - `tests/gpu_megatron` (`_extensions`, `distill`, `export`, `opt`, `peft`, `sparsity`, `speculative`, `utils`): 61 passed, 5 xpassed. - `tests/gpu_megatron/torch/export` re-run after the conv1d dispatch change: 27 passed. - The 21 previously failing/hanging quantization tests: 21 passed (12 + 9). - `tests/gpu_megatron/torch/{nas,prune}`: verified separately. `import.sh` equivalence on a toy `qwen3_moe`, comparing all 12 weight tensors after flattening each dist checkpoint with `dcp_to_torch_save` — the GPU backend at 1 GPU, `--tp 2`, `--pp 2`, `--ep 2`, and `import.sh` end-to-end (`GPUS_PER_NODE=2 EP=2`) are all byte-identical to `--device cpu`. `nemo:26.06` compatibility was checked directly in that image: `megatron.core.models.hybrid.HybridModel`, the modelopt hybrid spec and `hybrid_layer_pattern` are all present, while `megatron.bridge.models.hybrid` and the bridge's `distill_submodule` are not. The NemotronH round-trip test failed there before the conv1d dispatch was restored and the dispatch is back in place; per project convention the suites themselves only run on 26.08. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ⚠️ Megatron-Bridge examples plus Minitron pruning of Mamba/hybrid models now require `nemo:26.08`. Megatron-LM quantization and checkpoint export still run on `nemo:26.06`. - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: ✅ — `test_output_layer_extra_state_empty_when_nothing_quantized` for the compress fix; existing tests extended for the merged export coverage. - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ - Did you get Claude approval on this PR?: ❌ 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added guidance for importing Hugging Face checkpoints into Megatron distributed format. * Expanded distillation workflows to export selected or all checkpoint iterations. * **Improvements** * Expanded Hybrid model support across Megatron workflows. * Updated distributed import tooling with GPU and parallelism options. * Updated supported environments and examples to NVIDIA NeMo 26.08. * **Bug Fixes** * Corrected output-layer quantization state handling when quantization is disabled. * **Documentation** * Added compatibility guidance for current and legacy NeMo containers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
realAsma
|
a0513f18bd |
Update documentation theme to Shibuya (#2242)
### What does this PR do?
Type of change: documentation.
Replaces the legacy Sphinx RTD theme with Shibuya and gives the ModelOpt
documentation a modern responsive light/dark presentation.
The configuration uses the Shibuya green palette with NVIDIA green
(#76b900) as the primary accent, follows the reader system color
preference by default, enables dark code blocks, and expands the first
level of global navigation. RTD-specific CSS is removed, while the
announcement page now uses Shibuya semantic color tokens in both light
and dark modes.
Shibuya 2026.7.12 is licensed under BSD-3-Clause.
### Usage
```python
html_theme = "shibuya"
html_theme_options = {
"accent_color": "green",
"color_mode": "auto",
"dark_code": True,
"globaltoc_expand_depth": 1,
}
```
Doc preview:
https://nvidia.github.io/Model-Optimizer/pr-preview/pr-2242/
### Testing
- `nox -N --envdir /tmp/modelopt-shibuya-nox -s docs`
- Sphinx 9.1 built all 411 pages successfully with `--fail-on-warning`.
- `pre-commit run --files docs/source/conf.py
docs/source/_static/custom.css docs/source/_static/announcements.css
pyproject.toml uv.lock --show-diff-on-failure`
- All applicable hooks passed.
- Verified generated HTML loads Shibuya assets, declares the green
accent, includes automatic light/dark mode logic, and includes the
custom theme-token CSS.
### Before your PR is "*Ready for review*"
Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).
Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).
- Is this change backward compatible?: ✅
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: ✅
- Did you write any new necessary tests?: N/A — documentation theme
change covered by the full warning-as-error build.
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A — documentation presentation change.
- Did you get Claude approval on this PR?: N/A — focused documentation
theme migration.
### Additional Information
No source code or public API behavior changes.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Updated the documentation site with the Shibuya theme.
* Added green theme accents, automatic light/dark mode, and dark code
blocks.
* Improved table-of-contents behavior and refreshed announcement
styling.
* Removed outdated layout and table customization overrides.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: realAsma <akuriparambi@nvidia.com>
|
||
|
realAsma
|
8c04ce6ee2 |
docs: add AutoQuantize mixed-precision search blog (#1979)
### What does this PR do? Type of change: documentation. Adds the AutoQuantize technical blog to the announcements system introduced by #1971. - Preserves the source derivation, deployment-aware search details, results, usage example, and references in native Sphinx RST. - Adds the MMLU accuracy-versus-effective-bits figure as a losslessly optimized asset. - Adds a newest-first landing-page card and AutoQuantize filter. - Credits the authors in this order: Asma Beevi K T, Wei Ming, Frida Hou, Juhi Mittal, Jenny Chen, Ajinkya Rasane, Meng Xin. This is a stacked PR targeting the branch for #1971. After #1971 merges, this PR can be retargeted to main. ### Usage N/A; documentation only. ### Testing - Focused pre-commit hooks on all three changed files. - git diff --check. - Focused Sphinx HTML build for the announcement and landing page. - Rendered-output checks for equations, table, Python code block, image and alt text, references, external links, card, filter, and exact author order. - Full fail-on-warning build was attempted; remaining warnings were unrelated optional autodoc environment warnings. ### Before your PR is "Ready for review" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: N/A - Did you write any new necessary tests?: N/A - Did you update Changelog?: N/A - Did you get Claude approval on this PR?: N/A ### Additional Information Depends on #1971. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Added a comprehensive announcement introducing AutoQuantize for gradient-based mixed-precision optimization. - Documented sensitivity scoring, effective-bits cost modeling, deployment-aware grouping, benchmark results, usage examples, future plans, and references. - Added the announcement to the documentation homepage with an August 24, 2026 release card from the Model Optimizer Team. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: realAsma <akuriparambi@nvidia.com> |
||
|
Chenhan D. Yu
|
ddca53b4bf |
docs: add announcements landing page (#1971)
## Summary - Add a JS/static announcements landing page for GitHub Pages - Keep existing Sphinx docs available under the `api/` subpath - Add PR-authored announcement posts with tags, search/filtering, image support, and a DSpark vs Domino sample post Jira: https://jirasw.nvidia.com/browse/OMNIML-5476 ## Verification - `python3 docs/build_site.py --output docs/build/html` - Local preview checked at `http://127.0.0.1:8088/` ## Publishing approval User explicitly approved publishing the `dspark-vs-domino` sample post and copied image assets from `modelopt-site` to public GitHub in `NVIDIA/Model-Optimizer`. ## Notes - Full `uv run nox -s docs` was attempted locally, but dependency setup/download did not complete in a reasonable time; CI should provide the authoritative full docs build and PR Pages preview. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an interactive announcements hub to the documentation homepage with date sorting, tag filtering, search, pagination, and empty-state messaging. * Improved announcement presentation with consistent cards, metadata, typography, and interactive controls. * **Documentation** * Added announcements covering the GitHub Pages announcement hub and a DSpark versus Domino comparison. * Added guidance for authoring, reviewing, and discovering future announcements. * Updated documentation navigation to feature announcements while keeping other sections accessible. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Chenhan Yu <chenhany@nvidia.com> |
||
|
ZhiyuandClaude Opus 5
|
6261f854aa |
docs: rebuild the unified HF deployment support matrix from the deploy test suite (NVBug 6550792) (#2087)
### What does this PR do? Type of change: documentation Fixes [NVBug 6550792](https://nvbugspro.nvidia.com/bug/6550792) / OMNIML-5693. The **Unified HF Checkpoint Deployment Model Support Matrix** listed 9 model families and **no VLMs**, while `tests/examples/hf_ptq/test_deploy.py` declares deployment cases for ~80 checkpoints across TRT-LLM, vLLM, and SGLang — including `Qwen2.5-VL`, `Qwen3-VL-235B`, and `Nemotron-3-Nano-Omni`. QA (the filer) could not use the doc to scope testing, and users could not tell what is actually covered. Filing also surfaced that the matrix lived in **three places that had drifted apart**: only the `.rst` listed Qwen3-VL, only the README listed Qwen3.5 MoE, and the skill reference had neither. #### Changes 1. **Rebuilt the matrix in `docs/source/deployment/3_unified_hf.rst`** from `test_deploy.py`, split into language models, vision-language/multimodal, speculative decoding drafters, and diffusion. 2. **Stated plainly what the matrix is and is not.** Review established that the original "CI-validated" framing claimed more than the suite substantiates, so a *What this matrix is based on* section now leads with two limits: - The suite is marked `release` and collects only under `--run-release`, which **no workflow passes** — these are declared cases, not PR-gated coverage. - Each case is a **load-and-generate smoke check on the text path**: no accuracy, no image/audio input, no diffusion output, no verification that speculative decoding engages. The legend follows from that: ✅ = declared in the suite, ⚠ = expected to work but not a suite entry (or an entry that does not exercise the feature the row names), `-` = not in the suite. Sections that would otherwise over-read carry their own qualifiers — VLM rows are labelled text-only smoke coverage, and Medusa and Wan 2.2 are ⚠ with the reason stated. 3. **Removed the two duplicate copies**, replacing them with links, so there is one table to maintain. 4. **Fixed stale prose**: the deployment tabs still claimed FP8-only support on vLLM v0.6.5 and a source build of SGLang main from Jan 2025, both contradicting the version table above them. The TRT-LLM floor moves to v1.2.0, qualified as the oldest version stated rather than the oldest that works. 5. **Dropped the Phi series** from the deployment matrix, following #2115 (NVBug 6563509) and confirmation that Phi-4 is being deprecated. ### Usage N/A — documentation only. ### Testing - `docutils` parse of the modified `.rst`: no warnings or errors from the new content; all 5 tables parse with every cell in the correct column. - Cell contents cross-checked against `test_deploy.py` by AST-parsing the `ModelDeployerList(...)` calls rather than by eye; the scope caveats were each verified against `tests/_test_utils/deploy_utils.py`. - `pre-commit run --files …` passes; `build-docs` green. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A — documentation only - Did you get Claude approval on this PR?: ❌ — not yet run ### Additional Information **Two known follow-ups, neither in scope here:** 1. **Nothing enforces that the doc matrix tracks `test_deploy.py`.** Consolidating to one copy removes the three-way drift but not the doc-vs-test drift; a generator plus a CI check would close it. 2. **The release deployment suite does not run in CI.** Wiring it into per-backend release CI is what would let ✅ mean "verified to pass" rather than "declared". That needs GPU capacity across three backends and should be tracked on its own. **For the filer (@Kenny Kang):** the ✅ cells are the scope the release deploy suite declares, and `test_deploy.py` carries the checkpoint, TP size, and minimum SM version per entry — but please read the legend first, since those cases are not currently executed by CI. --------- Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
vishalpandya1990
|
87c9f8cf83 |
Update documentation guide for ONNX INT4 PTQ on Windows cuda13 host (#2022)
### What does this PR do? Type of change: Documentation update - Update documentation guide for ONNX INT4 PTQ on Windows cuda13 host - mention about compatible onnxruntim-gpu and cupy-cuda13x packages. ### Testing - Windows's onnx_ptq\genai_llm INT4 PTQ example with a 1B genai-cuda-ep ONNX model + local doc building ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ / ❌ / N/A <!--- If ❌, explain why. --> - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ / ❌ / N/A <!--- Mandatory --> - Did you write any new necessary tests?: ✅ / ❌ / N/A <!--- Mandatory for new features or examples. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ / ❌ / N/A <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> - Did you get Claude approval on this PR?: ✅ / ❌ / N/A <!--- Run `/claude review`. NVIDIA org members can self-trigger for complex changes; orthogonal to CodeRabbit. --> ### Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Clarified Windows CUDA prerequisites for calibration and GPU-accelerated quantization. * Added setup guidance for CUDA 12 and CUDA 13.x, including compatible packages and cuDNN requirements. * Expanded installation verification steps for CUDA, ONNX Runtime, and CuPy. * Updated the GenAI LLM example with CUDA version compatibility guidance. * **Enhancements** * Added runtime logging of detected CUDA environment paths and version details during quantization. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: vipandya <vipandya@nvidia.com> |
||
|
Gwena Cunha
|
d984de3795 |
[6008361][ONNX][Quantization] Clarify autotune guidance (#1989)
### What does this PR do? Type of change: documentation This PR clarifies when users should use ONNX quantization with Autotune enabled versus the direct Autotune entry point. - Adds a warning to the Autotune guide explaining that direct Autotune is a lower-level Q/DQ placement tool and does not replace calibrated ONNX PTQ. - Updates the ONNX quantization guide to show `autotune=True` in the Python API and explain that it uses default Autotune settings. - Updates ONNX PTQ example documentation to prefer `python -m modelopt.onnx.quantization ... --autotune=<mode>` for accuracy-sensitive PTQ from an unquantized model. - Updates the direct Autotune CLI help text to point users back to the full ONNX quantization workflow when calibration data and accuracy validation are required. ### Usage ```python N/A — documentation/help text change. ``` ### Testing - Ran `python -m py_compile modelopt/onnx/quantization/autotune/__main__.py`. - Built the Sphinx documentation with `python -m sphinx -b html docs/source docs/build/html`; build succeeded. Remaining warnings are from optional documentation imports and existing cross-reference labels. ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Clarified that Direct Autotune is an advanced tool for Q/DQ placement experiments, not a replacement for full calibrated ONNX quantization. * Added guidance on when to use Direct Autotune versus the end-to-end ONNX PTQ workflow. * Documented the optional `autotune=True` setting, expected calibration-time impact, and representative calibration data requirements. * Expanded links and guidance across ONNX PTQ examples and updated CLI help text with the recommended workflow. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Gwenaelle Cunha Sergio <gcunhasergio@nvidia.com> |
||
|
Chad Voegele
|
7d5d3f9046 |
MiniMax-M3 mixed MXFP8-base + NVFP4-experts PTQ export (#1806)
### What does this PR do?
Type of change: New example
Adds two workflows for producing MiniMax-M3 checkpoints with an MXFP8
language-model base and NVFP4 routed experts:
- A memory-bounded exporter that preserves the vendor MXFP8 base and
quantizes routed experts from BF16 one MoE layer at a time.
- A model-specific `hf_ptq.py` recipe that quantizes the complete BF16
model using MXFP8 for language-model linear layers and MSE-calibrated
NVFP4 for routed experts.
The routed-expert NVFP4 `input_scale` is fixed to 1.0. The vision
branch, routers, `lm_head`, and KV cache remain unquantized.
Supporting changes:
- Skip MSE `amax` calibration for MX formats, which do not use a global
scale.
- Discover decoder layers through the MiniMax-M3 VLM path
`model.model.language_model.layers`.
- Add focused tests for recipe precedence, MXFP8 MSE exclusion, and VLM
decoder discovery.
- Document the streaming exporter in `examples/minimax_m3/README.md` and
the model-specific recipe in the recipe guides.
### Usage
Quantize the complete BF16 model through `hf_ptq.py`:
```bash
python examples/hf_ptq/hf_ptq.py \
--pyt_ckpt_path /models/minimax-m3-bf16 \
--recipe huggingface/minimax_m3_vl/ptq/mxfp8_nvfp4_experts \
--export_path /models/minimax-m3-mxfp8-nvfp4 \
--use_seq_device_map \
--gpu_max_mem_percentage 0.68 \
--calib_size 1
```
Compose the vendor MXFP8 base with routed experts quantized from BF16:
```bash
python examples/minimax_m3/hf_ptq_mixed_mxfp8_nvfp4.py \
--mxfp8_ckpt /models/minimax-m3-mxfp8 \
--bf16_ckpt /models/minimax-m3-bf16 \
--recipe huggingface/minimax_m3_vl/ptq/nvfp4_experts_only \
--output_ckpt /models/minimax-m3-mxfp8-nvfp4 \
--device cuda
```
### Testing
- Full unit suite: 2,939 passed, 15 skipped.
- All pre-commit hooks passed.
- The streaming exporter reproduced all 89,614 tensors in
`nvidia/MiniMax-M3-NVFP4` exactly.
- Full BF16 `hf_ptq.py` validation matched all 87,552 NVFP4 expert
tensors, 534 MXFP8 scales, and 21,888 expert input scales exactly.
- The remaining 92 reference differences are BF16 Q/K norm tensors whose
values already differ between the public BF16 and vendor MXFP8 source
checkpoints.
- Verified standard Hugging Face shard names and mixed-precision
metadata.
### Before your PR is "*Ready for review*"
- Is this change backward compatible?: ✅
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: ✅
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
✅
- Did you get Claude approval on this PR?: ❌
### Additional Information
The workflows were tested with PyTorch 26.05.
---------
Signed-off-by: Chad Voegele <cvoegele@nvidia.com>
|
||
|
Keval Morabia
|
1b03381123 |
(Deps) Pin docs and test dependencies to avoid breaking CI on new packages (#1874)
Fix broken doc building CI and pin docs/test dependencies to avoid such issues in future <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Improved several guide cross-references so links resolve more reliably in the rendered docs. * Updated references in the quantization guides for better navigation between related topics. * **Chores** * Tightened versions for documentation and test tooling to improve consistency in local and CI environments. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
Keval MorabiaandClaude Opus 4.8
|
2fc352be2d |
Add VLM pruning and PTQ with image-text calibration (Megatron-Bridge) (#1792)
### What does this PR do?
Type of change: New feature
Adds **vision-language model (VLM) support** to the Megatron-Bridge
examples for both **Minitron pruning** (`prune_minitron.py`) and **PTQ**
(`quantize.py`). Only the **language model** is pruned/quantized — the
vision tower and vision→language projector are left in full precision —
and the full VLM is saved back. `hidden_size` is skipped for pruning
when it is shared with the vision→LM projector.
Supported VLMs (tested e2e): **Qwen{3,3.5}-VL** (dense; hybrid
GatedDeltaNet + gated attention) and **Gemma3-VL** (sliding/full
attention).
### Calibration (image-text)
Calibration is conditioned on real **image-text** data so the language
model's pruning importance / quantizer statistics see vision-conditioned
activations. The modality is inferred from `--calib_dataset_name`:
- an **image-text** dataset (default for VLMs,
`nemotron_vlm_dataset_v2`) drives the **full VLM forward**;
- a **text** dataset runs text-only calibration of the language model
(for text-vs-image ablations).
A shared `get_megatron_vlm_calibration_forward_loop` (built on
`megatron_prefill`) drives the full VLM forward over image-text pairs
from `vlm_dataset_utils` (`scienceqa`, `nemotron_vlm_dataset_v2`, with
config-driven subset/shard caps to bound downloads). It shards across
**data-parallel (DP)** ranks like the text loop (#1804); **context
parallelism (CP)** applies to text-only VLM calibration (the shared text
loop), not the multimodal forward — splitting the sequence would
misalign the merged vision embeddings.
### Results - Cosmos-Reason2-2B
Validated end-to-end on **Cosmos-Reason2-2B** (Qwen3-VL). Minitron NAS
prunes the language-model tower **1.72B → ~1.59B** (vision encoder +
projector frozen), top_k=1. Calibration data drives pruning importance;
image-text calibration runs the full VLM forward.
| Model | Calibration | MMLU | BLINK Rel-Depth | RealWorldQA |
|---|---|---|---|---|
| Baseline (1.72B) | — | 0.58 | 0.76 | 0.61 |
| Pruned (1.59B) | text (`nemotron-post-training-dataset-v2`) | 0.51\* |
~0.69 | ~0.57 |
| Pruned (1.59B) | image+text (`nemotron_vlm_dataset_v2`) | 0.49\* |
**0.77** | **0.61** |
\* Pruned MMLU on the 10% split (the pruning score function); baseline
MMLU is the full set. The VLM-benchmark numbers for the text row were
measured with a different text calibration set and are expected to be
similar for `nemotron-post-training-dataset-v2` (marked `~`).
> [!NOTE]
> These numbers come from short single runs on small eval splits — read
them for **high-level trends only**, not as exact values.
Takeaways: pruning the LM tower of a VLM works end-to-end. **Image-text
calibration** (this PR's feature) preserves the VLM benchmarks better
than text-only — BLINK Rel-Depth ~0.77 vs ~0.69 and RealWorldQA ~0.61 vs
~0.57, both close to the unpruned baseline (0.76 / 0.61) — which is the
motivation for calibrating on vision-conditioned activations.
### Results - Qwen3.5-9B
| Model | MMLU | MMStar |
|----------------------------|:------:|:------:|
| Qwen3.5-9B | 0.7003 | 0.6117 |
| Pruned-7B (text calib) | 0.5527 | 0.4411 |
| Pruned-7B (image+text calib) | 0.5107 | 0.3941 |
### Key changes
- `quantize.py`: quantizes the **root** model with non-LM (vision)
quantizers disabled, so the ModelOpt state lives on the root (required
by the Megatron save) while only the language model is quantized.
- `prune_minitron.py`: image-text (or text) calibration for VLM pruning
importance.
- Shared VLM calibration forward loop (`megatron_prefill`-based, unwraps
tuple outputs, DP-sharded) + `vlm_dataset_utils`.
- Tiny VLM test fixtures (Qwen3.5-VL, Gemma3-VL) with vision tokens
derived dynamically from the reference processor; VLM prune + quantize
example tests.
- README + CHANGELOG.
### Usage
```bash
# Prune the language model of a VLM (image-text calibration by default)
torchrun --nproc_per_node 2 prune_minitron.py \
--pp_size 2 \
--hf_model_name_or_path <vlm> \
--prune_target_params 3e9 \
--output_hf_path /tmp/vlm-pruned
# PTQ the language model of a VLM
torchrun --nproc_per_node 2 quantize.py \
--hf_model_name_or_path <vlm> \
--quant_cfg fp8 \
--export_megatron_path /tmp/vlm-fp8-megatron
```
### Testing
- `test_prune_minitron.py::test_prune_minitron_vlm` — Gemma3-VL,
image-text (ScienceQA) calibration; full load → prune (depth + ffn) →
save → reload.
- `test_quantize_export.py::test_quantize_vlm` — Qwen3.5-VL, text
calibration; quantize LM → save Megatron checkpoint.
- LM regression tests (`test_prune_minitron`,
`test_quantize_and_export`) unchanged and passing.
### Not in scope
- **HF unified export of a quantized VLM** is not yet supported;
`export.py` saves the Megatron checkpoint only for VLMs (tracked by a
TODO in `export.py`). The recommended path is to route the megatron→HF
quant export through Megatron-Bridge's
`AutoBridge.export_hf_weights_quant(quantization_checker, quant_fn,
quant_block_size)`, which reuses the bridge's per-model mcore↔HF mapping
— covering Qwen3.5-VL / Gemma3-VL and the vision tower/projector (left
full precision) for free — so modelopt supplies only the checker +
pack/scale fn + `hf_quant_config` (KV-cache scales need a separate
path). This avoids re-authoring per-model mappings in modelopt (cf.
#1482's Qwen3-VL-only `mcore_qwen3vl.py`).
> [!NOTE]
> Qwen3.5-VL **MoE** is not tested e2e: the Megatron-Bridge weight
conversion expects packed (`gate_up_proj`) experts that transformers'
tiny checkpoint doesn't emit. MoE pruning itself is covered by
`test_mcore_qwen35_gdn_moe_pruning`.
### Before your PR is "*Ready for review*"
- Is this change backward compatible?: ✅
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: ✅
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
✅
- Did you get Claude approval on this PR?: ✅
### Additional Information
Follow-up to the GatedDeltaNet/MLA/latent-MoE pruning PR (#1747).
Rebased on `main` to pick up CP/DP calibration (#1804); the VLM
calibration loop now shards across DP ranks the same way. `hidden_size`
pruning for VLMs (requires resizing the vision projector) is left for a
future PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added VLM-aware Minitron pruning and post-training quantization that
target only the language-model portion, keeping the vision
tower/projector in full precision.
* Calibration now auto-selects text vs image-text datasets based on
model type, with modality validation.
* Expanded Megatron-Core CP/DP guidance and introduced a `--cp_size`
flag in quantization examples.
* **Bug Fixes**
* Improved VLM generation/prefill output handling and made vocabulary
sizing more robust for VLM wrappers.
* **Tests / Documentation**
* Updated pruning/quantization docs and refreshed/added VLM-focused
tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
ZhiyuandClaude Opus 4.8
|
f335459dc0 |
refactor(examples): rename llm_ptq → hf_ptq (symlink for back-compat) (#1759)
## What does this PR do? **Type of change:** refactor / deprecation (examples) Follow-up to #1705 (which consolidated `examples/vlm_ptq` into `examples/llm_ptq`). Since that example now covers Hugging Face **LLM and VLM** PTQ, the `llm_ptq` name is a misnomer. This renames the directory to `examples/hf_ptq` and leaves a relative symlink `examples/llm_ptq → hf_ptq` so existing paths/commands keep working during a deprecation window. Requested by @kevalmorabia97 on #1705 (with the symlink-for-back-compat approach), targeted for the **same 0.46 release** as the consolidation. ### Changes - `git mv examples/llm_ptq → examples/hf_ptq` and `tests/examples/llm_ptq → tests/examples/hf_ptq` (the CI runner maps the matrix name to both `examples/<name>` and `tests/examples/<name>`). - Add a tracked back-compat symlink `examples/llm_ptq → hf_ptq`. - Update CI matrices and all repo **path references** (docs, READMEs, agent skills, launcher/debugger tools, tests) from `llm_ptq` to `hf_ptq`. - Keep Python identifiers / test-util module names (`run_llm_ptq_command`, `llm_ptq_utils`) — they name the LLM-PTQ task, not the directory. - Preserve the CODEOWNERS team slug (`modelopt-examples-llm_ptq-codeowners`) and historical CHANGELOG entries; add a CHANGELOG deprecation note. ### Back-compat caveats (inherent to git directory symlinks) - ✅ Linux/macOS CLI usage and Python `cwd`/pytest resolution work through the symlink. - ⚠️ Windows git checkouts don't materialize symlinks by default (low impact — this example is Linux-only in practice). - ⚠️ GitHub web doesn't follow directory symlinks, so legacy external deep-links to `examples/llm_ptq/...` won't navigate in. All **internal** references are repointed to `hf_ptq`, so the symlink is only for legacy external/CLI use. ### Usage (unchanged via symlink) ```bash # New canonical path cd examples/hf_ptq scripts/huggingface_example.sh --model <hf_model> --quant fp8 # Old path still works (forwards via symlink) cd examples/llm_ptq && scripts/huggingface_example.sh --model <hf_model> --quant fp8 ``` ### Testing - `bash -n` on moved/edited shell scripts (new path + via symlink). - `py_compile` on moved/edited Python; test re-export shim repointed to `examples/hf_ptq/example_utils`. - Verified git tracks `examples/llm_ptq` as a single symlink (mode 120000), not a duplicated tree (no pre-commit / pytest double-processing). - `pre-commit run` on all changed files passes. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ (relative symlink keeps `examples/llm_ptq` paths valid; see caveats above) - Did you write any new necessary tests?: N/A (pure rename; existing tests moved with the dir) - Did you update Changelog?: ✅ ### Additional Information Follow-up (later release): remove the `examples/llm_ptq` symlink once external references have migrated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PTQ guidance now directs to the unified Hugging Face PTQ flow, including VLM quantization via the shared `--vlm` entry point. * **Documentation** * Updated README and guide links, references, and command snippets to use `hf_ptq` (replacing `llm_ptq`). * Deprecated and consolidated `vlm_ptq` into `hf_ptq`; removed VILA/NVILA coverage from the Hugging Face PTQ examples. * **Bug Fixes** * Improved detection and routing so local/manual setup uses the correct PTQ source. * **Tests / Chores** * CI and example tests updated to run the `hf_ptq` variants. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Keval MorabiaandClaude Sonnet 4.6
|
33bfa8b1fe |
CI/Dev env bump (#1818)
### What does this PR do? Type of change: chore Bumps CI/dev tooling and test containers. **Container bumps** - NeMo test containers → 26.06 - TRT-LLM container → 1.3.0rc19 - transformers max version → 5.12 **Dev tooling bumps** - ruff bump 0.12.11 → 0.15.18 - mypy 1.17.1 → 2.1.0: enable new defaults (`local_partial_types`, `strict_bytes`); fix/narrow the errors newly surfaced by mypy 2.0 in 4 modules (rather than blanket-suppressing them); remove 2 stale `# type: ignore` comments - pre-commit 4.3.0 → 4.6.0 - sphinx 8.1 → 9.1 + sphinx-rtd-theme 3.0 → 3.1: add `suppress_warnings = ["ref.python"]` to fix cross-reference ambiguity error new in sphinx 9.x - trl fix for newly released 1.7 version **Bug fixes surfaced by the bumps** - sparsity (weight): make the weight mask DTensor-aware under FSDP. The transformers→5.12 bump routes the HF Trainer FSDP optimizer-state save through torch's DTensor-based `get_optimizer_state_dict`, which triggered `aten.mul.Tensor got mixed torch.Tensor and DTensor` in the dynamic `weight` getter. The mask is now distributed to the weight's mesh/placements before masking, cached, and rebuilt only when the sharding changes (invalidated on `set_mask`). Fixes the `llm_sparsity` example test. ### Testing - `pre-commit run --all-files` ✅ (including mypy 2.1.0) - `nox -s docs` ✅ - `tests/unit/torch/sparsity` + `tests/unit/torch/nas` ✅ - `llm_sparsity` GPU example test (FSDP path) verified in CI ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A - Did you get Claude approval on this PR?: ✅ <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary * **Documentation** * Refreshed Docker pre-requisites across examples to recommend updated container image tags (and streamlined some instructions). * **Bug Fixes** * Improved sparse weight mask handling for DTensor/FSDP by aligning and caching distributed masks. * Made TensorRT engine byte retrieval return immutable `bytes`. * Reduced Sphinx cross-reference warnings and tuned Transformers compatibility warning thresholds. * **Tests** * Increased default unit test timeout on Windows runners. * **Chores** * Updated CI workflow container tags and refreshed linting/typing/docs version pins, plus related mypy configuration. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Chenjie LuoandClaude Opus 4.8
|
56c4af2333 |
feat(recipes): add kv_fp8_cast variants for partial-NVFP4 and weight-only PTQ recipes (#1652)
### What does this PR do?
Type of change: new feature (recipes)
Several `general/ptq` recipe families shipped a data-driven FP8 KV-cache
(`-kv_fp8`) variant but lacked the constant-amax `kv_fp8_cast` companion
that `fp8_default` and `nvfp4_default` already have. This PR adds the
missing cast variants so every KV-quantizing (and the weight-only)
family offers the calibration-free FP8 KV-cache option:
- `general/ptq/nvfp4_experts_only-kv_fp8_cast`
- `general/ptq/nvfp4_mlp_only-kv_fp8_cast`
- `general/ptq/nvfp4_omlp_only-kv_fp8_cast`
- `general/ptq/nvfp4_weight_only-kv_fp8_cast`
Each new recipe composes the exact same model-quant config as its
existing sibling and swaps the `kv_fp8` unit for the shared
`kv_fp8_cast` unit (constant-amax FP8 KV cache; no KV calibration
forward pass). The docs guide table/tree and the changelog are updated
to match.
### Usage
```bash
python examples/llm_ptq/hf_ptq.py \
--pyt_ckpt_path <model> \
--recipe general/ptq/nvfp4_mlp_only-kv_fp8_cast
```
### Testing
Extended the built-in PTQ smoke test
`tests/unit/recipe/test_loader.py::test_load_recipe_all_builtins` with
the four new recipe paths; all four load into a valid
`ModelOptPTQRecipe` with a populated `quantize` section.
```
$ python -m pytest tests/unit/recipe/test_loader.py tests/unit/recipe/test_presets.py -q
180 passed
```
`pre-commit` (including the `validate modelopt recipes` hook) passes on
all changed files.
### Before your PR is "*Ready for review*"
- Is this change backward compatible?: ✅ (additive — only new recipe
files)
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: ✅ (extended the builtin recipe
smoke test)
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
✅
- Did you get Claude approval on this PR?: ❌ (not yet)
### Additional Information
The two weight-only families were discussed for scope;
`nvfp4_weight_only` is included (it already names a KV mode, `kv_fp16`),
while `int4_blockwise_weight_only` is intentionally left untouched since
it carries no `-kv_` composition.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added four new NVFP4 PTQ (Post-Training Quantization) recipe variants:
experts-only, MLP-only, OMLP-only, and weight-only configurations.
* All new recipes include FP8 KV-cache cast mode support for improved
inference performance.
* **Documentation**
* Updated built-in recipes guide with new NVFP4 recipe options and
repository layout.
* **Tests**
* Expanded recipe loader test coverage for new recipe configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Grzegorz K. KarchandKeval Morabia
|
b98a59557a |
Add vLLM-based runtime statistics for subblock latency measurement (#1358)
### What does this PR do? Type of change: ? <!-- Use one of the following: Bug fix, new feature, new example, new tests, documentation. --> <!-- Details about the change. --> ### Usage ```python # Add a code snippet demonstrating how to use this ``` ### Testing <!-- Mention how have you tested your change if applicable. --> ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ / ❌ / N/A <!--- If ❌, explain why. --> - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ / ❌ / N/A <!--- Mandatory --> - Did you write any new necessary tests?: ✅ / ❌ / N/A <!--- Mandatory for new features or examples. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ / ❌ / N/A <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> ### Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Runtime-based latency optimization: collect vLLM-measured inference latency to constrain optimization. * **Configuration** * New runtime config/template for Llama-3.1-8B pruning (runtime stats enabled, NCCL timeout templating, MIP target-latency). * Validation sample defaults adjusted (one flow: 128 → 8; runtime flow uses 128). * Human constraint key renamed to target_latency_seconds. * **Documentation** * README section describing runtime-based latency optimization setup and usage. * **Tests** * Added GPU end-to-end test for runtime stats collection. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/NVIDIA/Model-Optimizer/pull/1358?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Grzegorz Karch <gkarch@nvidia.com> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
hychiangandClaude Sonnet 4.6
|
f0d2237cbc |
Add Qwen3VL MCore Export support from PR 895 (#1482)
# [Megatron Export] Add Qwen3-VL mcore ↔ HF weight mapping > This PR is duplicated from [PR #895](https://github.com/NVIDIA/Model-Optimizer/pull/895). > The original branch source is no longer available; this new branch carries the same changes forward. ## What does this PR do? **New feature:** Add Qwen3-VL (Vision-Language) model support to the Megatron Core export/import plugin, enabling HuggingFace-to-mcore weight conversion for PTQ/QAT/QAD workflows. ### Overview Qwen3-VL has a different weight structure from Qwen3 text-only models: - Language model weights are under `model.language_model.` prefix (not `model.`) - Visual encoder weights are under `model.visual.` prefix - `lm_head` is at root level, not nested under `language_model` ### What changed | File | Change | |---|---| | `modelopt/torch/export/plugins/mcore_qwen3vl.py` | New plugin: derives Qwen3-VL mcore↔HF mapping by rewriting `model.*` → `model.language_model.*` on top of the existing Qwen3 dense rules; `lm_head.` is intentionally left unchanged | | `modelopt/torch/export/plugins/mcore_common.py` | Registers `Qwen3VLForConditionalGeneration` in `all_mcore_hf_export_mapping` and `all_mcore_hf_import_mapping` | | `modelopt/torch/export/plugins/hf_checkpoint_utils.py` | Generalized `load_multimodal_components` with a `prefixes` parameter; sharded checkpoints now scan all shards (not just the first) | | `modelopt/torch/export/unified_export_megatron.py` | `save_pretrained`: added Qwen3-VL branch that copies `model.visual.*` vision-encoder weights from the original HF checkpoint into the exported directory, producing a complete, loadable checkpoint | | `tests/_test_utils/torch/transformers_models.py` | Added `get_tiny_qwen3vl` / `create_tiny_qwen3vl_dir` helpers; Qwen3VL classes are lazy-imported inside the function to avoid collection failures on older transformers builds | | `tests/gpu_megatron/torch/export/test_unified_export_megatron.py` | Integrated Qwen3-VL export/import tests into the existing `test_unified_export_megatron` / `test_unified_import_megatron` parametrized suites; removed standalone `test_mcore_qwen3vl.py` | | `docs/source/deployment/3_unified_hf.rst` | Added Qwen3-VL (FP8 / NVFP4) to the deployment support matrix for TensorRT-LLM | ### Workflow coverage | Step | Status | Files | |---|---|---| | 1. Quantize Qwen3-VL with `hf_ptq` | ✅ existing | — | | 2. Export quantized mcore → HF | ✅ this PR | `plugins/mcore_qwen3vl.py` (weight name mapping), `unified_export_megatron.py` (export path) | | 3. Vision-encoder weights merged into export dir | ✅ this PR | `plugins/hf_checkpoint_utils.py` (`load_multimodal_components` with `prefixes`), `unified_export_megatron.py` (calls it when `arch == "Qwen3VLForConditionalGeneration"`) | | 4. Import HF checkpoint back to mcore | ✅ this PR | `plugins/mcore_qwen3vl.py` (same mapping, reverse direction), `unified_export_megatron.py` (import path) | ### Design notes - **MoE not supported**: `Qwen3VLMoeForConditionalGeneration` stores expert weights as 3-D tensors (`mlp.experts.gate_up_proj`, `mlp.experts.down_proj`) that require a dedicated fused-expert mapping. A `NotImplementedError` comment in the plugin documents this explicitly. - **`copy.deepcopy` on `func_kwargs`**: each mapping entry gets its own copy to prevent shared-dict mutation when both Qwen3 and Qwen3-VL rules are loaded. - **`prefixes` parameter on `load_multimodal_components`**: backward-compatible default preserves existing LLaVA behaviour (`"multi_modal_projector"`, `"vision_model"`); Qwen3-VL callers pass `("model.visual.",)`. - **Sharded checkpoint scan**: the old code only looked in the first shard. The Qwen3-VL vision encoder can span multiple shards, so all shards are now scanned. ## Usage From the [Megatron-LM PR comment](https://github.com/NVIDIA/Megatron-LM/pull/3444#issuecomment-3911271713): > Qwen3VL is supported within [Megatron-Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge), and pretraining and PEFT recipes for Qwen3VL are [here](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/src/megatron/bridge/recipes/qwen_vl/qwen3_vl.py) and the core code logic [here](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/src/megatron/bridge/models/qwen_vl). Create `Megatron-LM/examples/post_training/modelopt/conf/Qwen/Qwen3-VL-8B-Instruct.sh`: ```bash #!/bin/bash # Qwen3-VL-8B-Instruct text-model config for Megatron-LM import/quantize. # # Text-model dimensions are identical to Qwen3-8B (4096 hidden, 36 layers, # 32 heads, GQA=8). Differences: rope_theta=5000000, checkpoint path uses # model.language_model.* prefix (handled by mcore_qwen3vl plugin). if [ -z ${HF_MODEL_CKPT} ]; then HF_MODEL_CKPT=Qwen/Qwen3-VL-8B-Instruct TOKENIZER_MODEL=Qwen/Qwen3-VL-8B-Instruct else TOKENIZER_MODEL=${HF_MODEL_CKPT} fi MODEL_ARGS=" \ --save-interval 100000 \ --micro-batch-size 1 \ --bf16 \ --no-masked-softmax-fusion \ --disable-bias-linear \ --untie-embeddings-and-output-weights \ --position-embedding-type rope \ --no-rope-fusion \ --normalization RMSNorm \ --swiglu \ --num-layers 36 \ --hidden-size 4096 \ --ffn-hidden-size 12288 \ --num-attention-heads 32 \ --group-query-attention \ --num-query-groups 8 \ --kv-channels 128 \ --qk-layernorm \ --seq-length 4096 \ --max-position-embeddings 262144 \ --tokenizer-type HuggingFaceTokenizer \ --make-vocab-size-divisible-by 1187 \ --use-mcore-models \ --rotary-percent 1.0 \ --rotary-base 5000000 \ --no-bias-swiglu-fusion \ " ``` Import Qwen3-VL from HuggingFace to MCore (local, requires GPUs): ```bash MLM_MODEL_CFG=Qwen/Qwen3-VL-8B-Instruct \ HF_MODEL_CKPT=Qwen/Qwen3-VL-8B-Instruct \ MLM_MODEL_SAVE=/tmp/qwen3vl_mcore \ TP=1 \ bash Megatron-LM/examples/post_training/modelopt/convert.sh Qwen/Qwen3-VL-8B-Instruct ``` Quantize (PTQ via Megatron-LM path): ```bash MLM_MODEL_CFG=Qwen/Qwen3-VL-8B-Instruct \ HF_MODEL_CKPT=Qwen/Qwen3-VL-8B-Instruct \ QUANT_CFG=NVFP4_DEFAULT_CFG \ TP=4 \ bash Megatron-LM/examples/post_training/modelopt/quantize.sh Qwen/Qwen3-VL-8B-Instruct ``` ## Testing - Verified round-trip import/export with Qwen3-VL-8B-Instruct with the example usage above - Unit/GPU tests covering: - Registration in global export/import mappings - Import mapping: dense keys, `model.language_model.` prefix, `lm_head.` at root, `QKVMerging`, `GatedMLPMerging`, `REPLICATE` for layernorms, TP sharding configs - Export mapping: `QKVSlicing`, `GatedMLPSlicing`, no `parallel_config` - Import/export symmetry: same mcore keys, matching HF prefixes - Qwen3-VL vs Qwen3 difference: same keys, VL adds `language_model.` prefix, `lm_head` unchanged ## Before your PR is "Ready for review" - Is this change backward compatible?: Yes, additive only - Did you write any new necessary tests?: Yes, `tests/gpu_megatron/torch/export/test_unified_export_megatron.py` - Did you add or update any necessary documentation? Yes, see `docs/source/deployment/3_unified_hf.rst` - Did you update Changelog? Yes, see `CHANGELOG.rst` ## Additional Information Companion Megatron-LM PR adds `Qwen3VLModel`, `Qwen3VLDataset`, and `pretrain_qwenvl.py`. See: https://github.com/NVIDIA/Megatron-LM/pull/3444 --------- Signed-off-by: Hung-Yueh Chiang <hungyuehc@nvidia.com> Signed-off-by: hychiang <hungyuehc@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Shengliang Xu
|
04f58166ab |
[OMNIML-3707] Model-specific PTQ recipes bootstrap (#1506)
### What does this PR do?
Type of change: new feature
Replaces the hardcoded model-type branches in `examples/llm_ptq/` with
opt-in declarative **model-specific recipes** under
`modelopt_recipes/huggingface/<model_type>/ptq/`. Any adjustment
specific to a model type or instance must live in that model's recipe —
there is no implicit model-specific path anymore. Users select a model's
recipe with `--recipe huggingface/<model_type>/ptq/<recipe>`; users on
the plain `--qformat` path get only the generic numerics.
What moved out of Python
(`examples/llm_ptq/example_utils.py::build_quant_cfg` and
`examples/llm_ptq/hf_ptq.py::mono_quantize`):
- **gemma / mpt** `w4a8_awq` → `awq_lite` with `alpha_step=1` (coarser
search to avoid TRT-LLM overflow).
- **gemma** `int8_sq` → SmoothQuant `alpha=0.5` (default `1.0` regresses
Gemma 7B).
- **phi4mm** → disable `*speech*`, `*audio*`, `*image*`, `*vision*`
(quantize only the language model).
- **Nemotron VL** → disable `*vision*`, `*image*`, `*radio*`,
`*visual*`, `*encoder*`, `*model_encoder*` (quantize only the decoder).
What stayed in Python:
- MTP dynamic layer exclusion in `hf_ptq.py` (depends on
runtime-detected layer indices).
- `is_nemotron_vl(full_model)` detection itself, which still drives the
VLM calibration loop and the post-quantize `full_model` update — only
the `quant_cfg` adjustment it triggered moved into the Nemotron VL
recipe.
`multinode_ptq.py` shares the same `build_quant_cfg` call site and was
updated to match the new 2/3-arg signature; multinode users on
`--qformat` get the generic numerics (no `--recipe` plumbing in
multinode yet, so model-specific recipes are only reachable via
`hf_ptq.py`).
Already-YAML recipes that were elsewhere in the tree are relocated into
the same `huggingface/<model_type>/ptq/` layout so all model-specific
recipes live under one convention:
- **Step3.5-Flash** — moved from
`modelopt_recipes/huggingface/step3p5/Step3.5-Flash/` to
`huggingface/step3p5/Step3.5-Flash/ptq/` to match the `<model>/ptq/`
convention.
- **Qwen3.5 / Qwen3.6** — moved from
`modelopt_recipes/models/Qwen3.5-Qwen3.6/w4a16.yaml` to per-model_type
folders, anchored on the HuggingFace `model_type` (verified against
transformers 5.8.1 + HF model hub `config.json` for `Qwen/Qwen3.6-27B`,
`Qwen/Qwen3.6-35B-A3B`, `nvidia/Qwen3.5-397B-A17B-NVFP4`):
- `huggingface/qwen3_5/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast.yaml` —
dense `qwen3_5`
- `huggingface/qwen3_5_moe/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast.yaml` —
`qwen3_5_moe`
- Both wrappers `$import` the shared `quant_cfg` snippet
`huggingface/qwen3_5/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast.quant_cfg.yaml`
(one source of truth; the two model_types share the same hybrid
linear-attention + softmax-attention architecture so the rules apply
identically).
Full recipe layout (`modelopt_recipes/huggingface/`):
```
gemma/ptq/{w4a8_awq,int8_sq}-kv_fp8_cast.yaml
mpt/ptq/w4a8_awq-kv_fp8_cast.yaml
phi4mm/ptq/{disabled_quantizers,nvfp4-kv_fp8_cast}.yaml
nemotron_vl/ptq/{disabled_quantizers,nvfp4-kv_fp8_cast}.yaml
qwen3_5/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast{,.quant_cfg}.yaml
qwen3_5_moe/ptq/w4a16_nvfp4-fp8_attn-kv_fp8_cast.yaml
step3p5/Step3.5-Flash/ptq/nvfp4-mlp-only.yaml
```
All recipes ship with FP8 KV-cache cast (`kv_fp8_cast`). For phi4mm and
nemotron_vl, `disabled_quantizers.yaml` is a multi-document list unit
that `$import`s the standard `default_disabled_quantizers` exclusions
and appends the model-specific ones — so each recipe imports a single
disabled-quantizer slot instead of layering two, with no duplication in
YAML. Each `ptq/` folder has a `README.md` describing exactly what is
model-specific.
### Usage
```bash
# Gemma W4A8 AWQ with the Gemma-specific algorithm tuning + FP8 KV cache:
python examples/llm_ptq/hf_ptq.py \
--pyt_ckpt_path google/gemma-7b \
--recipe huggingface/gemma/ptq/w4a8_awq-kv_fp8_cast \
--export_path ./out
# Nemotron VL with vision branches excluded automatically:
python examples/llm_ptq/hf_ptq.py \
--pyt_ckpt_path nvidia/<nemotron-vl-model> \
--recipe huggingface/nemotron_vl/ptq/nvfp4-kv_fp8_cast \
--export_path ./out
```
### Testing
- Pre-commit recipe validator
(`tools/precommit/check_modelopt_recipes.py`) loads every new recipe via
`load_recipe()` — passes for all new YAMLs (gemma/mpt/phi4mm/nemotron_vl
recipes + phi4mm/nemotron_vl `disabled_quantizers` snippets + qwen3_5 /
qwen3_5_moe recipe wrappers + the shared
`w4a16_nvfp4-fp8_attn-kv_fp8_cast.quant_cfg` snippet + Step3.5-Flash
relocation).
- For qwen3_5 / qwen3_5_moe specifically, `load_recipe(...)` on both
wrappers produces an identical 33-entry resolved `quant_cfg`, confirming
the shared snippet is the single source of truth.
- `yamlfmt` + `markdownlint` + `bandit` + license-insertion hooks all
pass.
- No tests reference the removed `build_quant_cfg(qformat, ...,
model_type, ...)` signature; the only call sites (`hf_ptq.py`,
`multinode_ptq.py`) were updated to the new 2/3-arg form.
### Before your PR is "*Ready for review*"
- Is this change backward compatible?: ❌ — users who relied on
**automatic** model-specific quant_cfg behavior via `--qformat`
(gemma/mpt AWQ, gemma SmoothQuant, phi4mm exclusions, Nemotron VL
exclusions) now need to pass `--recipe
huggingface/<model_type>/ptq/<recipe>` to apply the model's recipe. The
flag itself is unchanged; only the implicit behavior was removed.
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: ❌ — relies on the existing
pre-commit recipe validator that loads each new YAML.
- Did you update Changelog?: ✅
- Did you get Claude approval on this PR?: ❌
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added many model-specific PTQ recipes (Gemma, MPT, Nemotron VL,
Phi‑4‑Multimodal, Qwen3.5, Qwen3.5‑MoE) and support for AWQ block-size
and MoE calibration ratio in quantization options.
* **Documentation**
* Expanded READMEs and changelog to document recipe locations, layout,
and how to opt into model-specific PTQ recipes.
* **Refactor**
* Model-specific PTQ tweaks moved to opt‑in recipes; default behavior
uses generic numerics.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/NVIDIA/Model-Optimizer/pull/1506?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
|
||
|
Shengliang Xu
|
2b8defc14b |
[OMNIML-4158] ModelOpt config system documentation (#1472)
### What does this PR do?
Type of change: documentation
Adds a new Sphinx guide for the ModelOpt config system and updates two
adjacent guides to match the current APIs.
New file:
- `docs/source/guides/11_config_system.rst` — covers the general config
contract and semantics rather than any single consumer:
- `ModeloptBaseConfig` schemas, validation boundaries, and
`MutableMapping` access semantics
- YAML loading flow, effective-schema selection, and persistence
- Checkpoint persistence
- Composable YAML via `imports` / `$import`, including the dict / list /
multi-document forms and the type-directed list splice-vs-append rule
The guide also explains *why* ModelOpt uses a small YAML DSL for
composition (instead of OmegaConf alone, Python factories, or hard-coded
registries): config files stay self-describing, reusable fragments can
be authored as YAML, and resolved values are still validated against the
Python schemas. Recipes are presented as one application of the shared
config system; recipe-specific authoring continues to live in the
existing recipes guide.
Updates to existing guides:
- `docs/source/guides/10_recipes.rst` — `RecipeMetadataConfig` and
`QuantizeConfig` are described as `ModeloptBaseConfig` subclasses;
`quant_cfg` entries are validated `QuantizerCfgEntry` instances after
loading; the schema-comment section now explains that validation runs
against the effective schema (in-file comment or `schema_type=`
argument, with the argument winning).
- `docs/source/guides/_quant_cfg.rst` — entries described as
`QuantizerCfgEntry` Pydantic instances (with dict input accepted and
normalized via `mode="before"` validator); `cfg:` field typed as
`QuantizerAttributeConfig` rather than a free-form dict.
### Usage
`load_config` resolves YAML composition (`imports` / `$import`) and,
when an effective schema is in scope (either via `schema_type=` or a `#
modelopt-schema:` comment), returns a validated schema instance
directly:
```python
from modelopt.recipe import load_config
from modelopt.torch.quantization.config import QuantizeConfig
cfg = load_config("configs/ptq/presets/model/fp8", schema_type=QuantizeConfig)
# cfg is a validated QuantizeConfig instance.
resolved = cfg.model_dump()
```
### Testing
- Docs-only change; no code paths touched.
- Pre-commit hooks pass (RST formatting checks).
### Before your PR is "*Ready for review*"
Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).
Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.). Docs-only change — no executable
code paths affected.
- Is this change backward compatible?: ✅
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: N/A (docs-only)
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A (docs-only)
- Did you get Claude approval on this PR?: ✅
### Additional Information
N/A
---------
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
|
||
|
realAsma
|
62401e16df |
fix: layerwise calibration backward-compat, recipe split, batch-size guard (#1310)
## Summary Follow-up to #1251 (which renamed `use_sequential` → `layerwise`). Three related fixes bundled: 1. **Backward-compatible config loading.** PTQ checkpoints saved before #1251 store the legacy `use_sequential` key in the calibration-algorithm config, so loading them now raises `ValidationError: Extra inputs are not permitted (use_sequential)` because `QuantizeAlgorithmConfig` uses `extra='forbid'`. Accept `use_sequential` as an alias for `layerwise` via `AliasChoices`. The field still serializes as `layerwise`, so round-trips through the current schema are clean. 2. **Recipe split.** `nvfp4_experts_only-fp8_kv` previously enabled layerwise calibration by default, which changes the calibration flow materially. Split into two recipes: - `nvfp4_experts_only-fp8_kv.yaml` — default (no layerwise) - `nvfp4_experts_only-fp8_kv_layerwise.yaml` — layerwise variant 3. **`hf_ptq` batch-size guard.** Auto batch-size detection is not supported together with layerwise calibration. Default to `batch_size=1` when layerwise is enabled and the user hasn't set a batch size explicitly. Originally reported by Jenny Chen while resuming a PTQ checkpoint via `restore_sharded_modelopt_state`: ``` pydantic_core._pydantic_core.ValidationError: 1 validation error for MaxCalibConfig use_sequential Extra inputs are not permitted [type=extra_forbidden, input_value=False, input_type=bool] ``` ## Test plan - [x] `tests/unit/torch/quantization/test_config_validation.py` — legacy alias accepted, current name accepted, dump serializes under current name, `extra='forbid'` still rejects unknown keys. - [x] `pre-commit run` — clean. ### Before your PR is *Ready for review* - Is this change backward compatible?: ✅ (restores compatibility for pre-#1251 checkpoints) - New PIP dependency: N/A - New necessary tests: ✅ - Changelog update: N/A (bug fix) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Added new PTQ recipe for efficient layerwise calibration of large models. * Automatic batch size optimization for layerwise calibration recipes. * Backward compatibility support for legacy input naming conventions. * **Documentation** * Updated recipe guides and changelog with new layerwise calibration recipe. * **Tests** * Added validation tests for configuration compatibility. [](https://app.coderabbit.ai/change-stack/NVIDIA/Model-Optimizer/pull/1310) <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: realAsma <akuriparambi@nvidia.com> |
||
|
Keval Morabiaandclaude[bot]
|
2ce745a92e |
Deprecate gradnas pruning and bert example (#1427)
### What does this PR do? Type of change: Deprecation of dead code <!-- Use one of the following: Bug fix, new feature, new example, new tests, documentation. --> Deprecation warning already added in 0.44 as per 1-release deprecation policy GradNAS only works for Bert and GPT-J and we dont actively maintain it or test it. Keeping it creates an expectation that it works plus it adds one more option for user to choose from. We already have much better pruning algorithms (Minitron and Puzzletron) for LLM pruning already hence removing GradNas. ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ❌ No but we dont have any users of this feature either <!--- If ❌, explain why. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Deprecation** * GradNAS pruning algorithm deprecated; related examples removed. * **Documentation** * Pruning and NAS guides and changelog updated to focus on Minitron and FastNAS; GradNAS references removed. * **Chores** * Chained-optimizations example and scripts removed. * Ownership mappings updated for README and examples; license insertion now applies to a previously excluded example file. * **Tests** * Multiple unit tests and test utilities related to GradNAS/transformer NAS removed. [](https://app.coderabbit.ai/change-stack/NVIDIA/Model-Optimizer/pull/1427) <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> |
||
|
Shengliang Xu
|
f34f488a83 |
Add a general composable $import system for YAML configs, and use it to implement composable recipes (#1253)
### What does this PR do?
Type of change: New feature
Adds a general composable YAML config loading layer for ModelOpt configs
and recipes. YAML remains the source of truth for configuration data,
while Python/Pydantic-compatible types provide schema validation at load
time. This PR uses that loader to de-duplicate PTQ recipes, introduce
reusable config snippets/presets, and start migrating selected hardcoded
quantization presets to YAML.
#### Problem
1. Built-in PTQ recipes duplicated numeric format definitions, KV-cache
entries, and the default quantizer exclusion list.
2. YAML snippets were reusable only by convention; they did not declare
or validate the schema they were meant to satisfy.
3. Loading YAML-backed quantization presets from
`modelopt.torch.quantization.config` could not depend on
`modelopt.recipe` without creating circular imports.
4. Directory-format recipes exposed import-resolution details in the
recipe loader and used `recipe.yaml` plus nested `metadata:` in a way
that made metadata handling inconsistent.
#### Solution
**Shared YAML config loader**
- Adds `modelopt.torch.opt.config_loader` as the low-level loader used
by both `modelopt.recipe` and `modelopt.torch.quantization.config`.
- Keeps the public `modelopt.recipe.load_config()` entry point, while
removing the private `modelopt/recipe/_config_loader.py` shim.
- Handles YAML loading, built-in/filesystem path resolution, suffix
probing, `ExMy` conversion for `num_bits` / `scale_bits`, `$import`
expansion, and schema validation.
- Lives below `modelopt.recipe` in the dependency graph to avoid
circular imports from quantization config code.
**Composable `$import` system**
Recipes and snippets can declare an `imports` mapping, then reference
entries with `{$import: name}`.
`$import` semantics:
- **Dict value**: replaced with the imported dict. Multiple imports are
supported with ordered precedence; inline keys override imported keys.
- **List entry**: schema-driven behavior for strongly typed lists. If
the snippet schema matches the containing list type, the imported list
is spliced. If the snippet schema matches the list element type, the
imported element is appended. Other schema combinations are rejected.
- **Multi-document YAML**: supports snippets that need an `imports`
header plus a list body.
- **Recursive and scoped**: snippets can import other snippets; import
names are scoped per file.
- **Cycle detection**: circular imports report a clear error.
**Snippet schema validation**
- Every reusable snippet referenced through `imports` must declare a `#
modelopt-schema: ...` preamble.
- Snippets are validated after nested imports are resolved.
- Schema paths are restricted to the `modelopt.` package and may be
Pydantic models, `TypedDict` classes, or explicitly typed container
aliases such as `list[QuantizerCfgEntry]`.
- Untyped list imports are rejected so list append/splice behavior stays
strongly typed.
**Recipe model and directory recipe cleanup**
- `ModelOptRecipeBase` now owns a `metadata: RecipeMetadataConfig`
field.
- `ModelOptPTQRecipe` is the PTQ recipe schema; the overlapping
YAML-specific PTQ config class was removed.
- Directory recipes now use `metadata.yaml` / `metadata.yml` for
top-level metadata fields, plus section files such as `quantize.yaml`.
- Directory recipe loading now delegates import resolution to
`load_config()` instead of manually using raw config loading.
**Config snippet and preset library**
Adds reusable snippets under `modelopt_recipes/configs/`:
- `numerics/`: `fp8`, `nvfp4`, `nvfp4_static`
- `ptq/units/`: `base_disable_all`, `default_disabled_quantizers`,
`w8a8_fp8_fp8`, `w4a4_nvfp4_nvfp4`, `kv_fp8`, `kv_fp8_cast`,
`kv_nvfp4_cast`
- `ptq/presets/`: YAML presets for `FP8_DEFAULT_CFG` and `FP8_KV_CFG`
`FP8_DEFAULT_CFG` and `FP8_KV_CFG` now load from YAML presets via
`load_config()`.
**Recipe migration and naming**
- General PTQ recipes now use shared imports instead of repeating the
same quantizer fragments inline.
- General PTQ recipe paths were renamed to KV-first naming, for example:
- `general/ptq/fp8_default-fp8_kv` -> `general/ptq/fp8_default-kv_fp8`
- `general/ptq/fp8_default-fp8_cast_kv` ->
`general/ptq/fp8_default-kv_fp8_cast`
- `general/ptq/nvfp4_default-none_kv_gptq` ->
`general/ptq/nvfp4_default-kv_none-gptq`
- `general/ptq/nvfp4_default-nvfp4_cast_kv` ->
`general/ptq/nvfp4_default-kv_nvfp4_cast`
- Example docs and `examples/llm_ptq/hf_ptq.py --recipe` help text were
updated to use the new paths.
**Pre-commit and documentation**
- Recipe validation accepts `$import` entries and handles directory
recipes using `metadata.yaml`.
- The recipe validation hook skips `modelopt_recipes/configs/` because
those files are reusable snippets, not full recipes.
- `docs/source/guides/10_recipes.rst` now documents imports, schema
modelines, list append/splice semantics, built-in snippets, built-in
recipe paths, directory recipes, and the current recipe data model.
#### Backward compatibility
- Existing inline YAML recipes without `$import` continue to load.
- `modelopt.recipe.load_config()` remains public.
- The built-in recipe path renames are user-visible; callers should
update recipe path strings to the KV-first names listed above.
#### Testing
- `pytest tests/unit/recipe/test_loader.py -q` - 90 passed
- `python tools/precommit/check_modelopt_recipes.py ...` for the renamed
built-in PTQ recipes
- `pre-commit run mypy --files
modelopt/onnx/llm_export_utils/quantization_utils.py`
- `python -m py_compile examples/llm_ptq/hf_ptq.py`
- `git diff --check`
### Before your PR is "Ready for review"
- Is this change backward compatible?: Partially. Loader/API behavior is
compatible for existing inline YAML recipes, but built-in recipe path
names were renamed to KV-first paths.
- Did you write any new necessary tests?: Yes.
- Did you update Changelog?: Yes.
---------
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
|
||
|
Keval Morabiaandcoderabbitai[bot]
|
70546bdd6a |
Enable Python 3.14 wheel support to unblock NGC PyTorch container testing on Ubuntu 26.04 + Python 3.14 (#1386)
Ubuntu 26.04 is here and very soon, NVIDIA PyTorch containers will ship with Python 3.14 requiring us to enable untested support to unblock them <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * DFlash offline speculative decoding training * MXFP4→NVFP4 weight conversion support * Shared hidden-state dump utilities * Updated DeepSeek PTQ calibration defaults * **Chores** * Added Python 3.14 support; updated Python requirement to <3.15 * **Documentation** * Updated installation documentation for Python version compatibility <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
ynankani
|
9bb917d57c |
[BUG6108338] Update windows documentation for onnxruntime quantization with Cuda13.x (#1368)
### What does this PR do? Type of change: ? documentation <!-- Details about the change. --> Update windows documentation for onnxruntime quantization with Cuda13.x ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: N/A <!--- If ❌, explain why. --> - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A <!--- Mandatory --> - Did you write any new necessary tests?: N/A <!--- Mandatory for new features or examples. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated Windows installation guide with CUDA 13.x-specific setup instructions for GPU-accelerated dependencies, including CuPy and ONNX Runtime configuration with nightly builds. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: ynankani <ynankani@nvidia.com> |
||
|
Keval MorabiaandClaude Sonnet 4.6
|
e56682e34a |
docs: update installation pages with legal-approved license notices (#1322)
## Summary
- Replaces the old pip license notice ("Please review the license terms
of ModelOpt and any dependencies before use") with the Legal-approved
wording: "Model Optimizer will download and install additional
third-party open source software projects. Review the license terms of
these open source projects before use."
- Adds a generic container license review notice ("Before pulling and
using the container images, please review their respective license
terms.") to the Linux installation doc (Docker tab) and README.
- Adds a `.. note::` with the pip notice to the Windows installation
page (covers both standalone and Olive child pages).
- Expands the README container section to explicitly list all four
recommended NVIDIA container images (`pytorch`, `nemo`, `tensorrt-llm`,
`tensorrt`).
## Test plan
- [x] Verify rendered docs look correct (`nox -s docs`)
- [x] Confirm legal notices appear in Linux, Windows, and README install
sections
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Updated installation guides with explicit references to supported
NVIDIA container images (PyTorch, NeMo, TensorRT-LLM and variants),
clarified pre-installed Model Optimizer in some images, and added notes
to review each container’s license terms; clarified conditional
environment setup wording and local install license guidance.
* **Chores**
* Updated project license header year.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
|
361f7e391b |
Merge puzzletron compression algorithm (#1121)
### What does this PR do? Implement puzzletron compression algorithm based on Puzzle paper (https://arxiv.org/abs/2411.19146) <details> <summary> Th list of reviewed and merged MRs that resulted in the feature/puzzletron branch</summary> Merging dkorzekwa/any_model to feature/puzzletron [Add anymodel directories to feature/puzzletron by danielkorzekwa · Pull Request #974 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/974) - merged [Draft: anymodel activation scoring by danielkorzekwa · Pull Request #989 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/989) - merged [Draft: Merge anymodel pruning by danielkorzekwa · Pull Request #990 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/990/) - merged [Draft: Merging anymodel:build_library_and_stats by danielkorzekwa · Pull Request #993 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/993) - merged [Dkorzekwa/any model calc one block scores by danielkorzekwa · Pull Request #994 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/994) - merged [Draft: merge any_model: mip_and_realize_models by danielkorzekwa · Pull Request #995 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/995) - merged [Dkorzekwa/any model other modeqls by danielkorztiekwa · Pull Request #1007 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1007/) - merged PR to 1007: https://github.com/NVIDIA/Model-Optimizer/pull/1039 - merged [Dkorzekwa/anymodel gptoss by danielkorzekwa · Pull Request #1020 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1020) - merged [Merge any_model tutorial by danielkorzekwa · Pull Request #1035 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1035) - merged [Merge mbridge distillation for any_model by danielkorzekwa · Pull Request #1036 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1036) - merged [MR branch for the remaining difference between dkorzekwa/any_model an… by danielkorzekwa · Pull Request #1047 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1047) - merged [Dkorzekwa/decilm hf code cleanup by danielkorzekwa · Pull Request #1071 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1071) - merged [Dkorzekwa/decilm hf code cleanup 2 by danielkorzekwa · Pull Request #1073 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1073) - merged [Dkorzekwa/anymodel subblock stats by danielkorzekwa · Pull Request #1085 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1085) - merged [Dkorzekwa/anymodel subblock stats nodecilm by danielkorzekwa · Pull Request #1102 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1102) - merged [Dkorzekwa/decilm cleanup post subblockstats by danielkorzekwa · Pull Request #1103 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1103) - merged [code clean up by danielkorzekwa · Pull Request #1110 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1110) - merged Merging into main: [Activation hooks redesign (reuse hooks component across both minitron and puzzletron) by danielkorzekwa · Pull Request #1022 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1022) - merged [Dkorzekwa/puzzletron use importance hooks from prune by danielkorzekwa · Pull Request #1115 · NVIDIA/Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer/pull/1115) - merged </details> <!-- Details about the change. --> ### Usage Puzzletron tutorial: https://github.com/NVIDIA/Model-Optimizer/tree/feature/puzzletron/examples/puzzletron ### Testing The main e2e test for compressing 9 models with Puzzletron: https://github.com/NVIDIA/Model-Optimizer/blob/feature/puzzletron/tests/gpu/torch/puzzletron/test_puzzletron.py 2-gpu nightly tests: - https://github.com/NVIDIA/Model-Optimizer/actions/runs/24468209205/job/71501061203 - https://github.com/NVIDIA/Model-Optimizer/actions/runs/24470214159/job/71508152952 ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ - Did you write any new necessary tests?: ✅ - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Puzzletron: end-to-end heterogeneous pruning & NAS workflow with AnyModel support, example pipelines, deployment and evaluation utilities, and tools for converting/pruning and exporting compressed checkpoints. * **Documentation** * Comprehensive Puzzletron tutorials, model-specific guides, evaluator instructions, example configs, and changelog entry. * **Chores** * CI/workflow updates (extras installation, longer GPU test timeout), pre-commit hook exclusion updated, and CODEOWNERS entries added. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Signed-off-by: Daniel Korzekwa <dkorzekwa@nvidia.com> Signed-off-by: Liana Mikaelyan <lmikaelyan@nvidia.com> Signed-off-by: Liana Mikaelyan <45925959+LianaMikael@users.noreply.github.com> Signed-off-by: Daniel Korzekwa <daniel.korzekwa@gmail.com> Signed-off-by: jrausch <jrausch@nvidia.com> Signed-off-by: root <root@pool0-00848.cm.cluster> Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Liana Mikaelyan <lmikaelyan@nvidia.com> Co-authored-by: Liana Mikaelyan <45925959+LianaMikael@users.noreply.github.com> Co-authored-by: J Rausch <38429553+j-rausch@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Gwena Cunhaanddmoodie
|
dec2952992 |
[6034518] Downgrade TRT support for remote autotuning in Autotune from 10.16 to 10.15 (#1259)
### What does this PR do? Type of change: Bug fix Remote autotuning is supported in TensorRT from version 10.15, but fails with Autotune as it's checking for 10.16+. This PR fixes that check and updates documentation accordingly. ### Usage ```python # Add a code snippet demonstrating how to use this ``` ### Testing See bug 6034518. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A <!--- Mandatory --> - Did you write any new necessary tests?: N/A <!--- Mandatory for new features or examples. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added a Remote Autotuning guide for TensorRT 10.15+ with CLI examples; updated examples to require `--safe --skipInference`. * **Updates** * Lowered TensorRT minimum requirement for remote autotuning from 10.16 to 10.15. * Clarified CLI help text for trtexec/autotune arguments. * **Bug Fixes** * trtexec-based autotuning now verifies the trtexec executable version when checking compatibility. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: gcunhase <4861122+gcunhase@users.noreply.github.com> Signed-off-by: dmoodie <dmoodie@nvidia.com> Co-authored-by: dmoodie <dmoodie@nvidia.com> |
||
|
Shengliang Xu
|
14b78aed1f |
recipes doc (#1165)
### What does this PR do? Added an extensive guide for the ModelOpt recipe system: recipe structure, YAML schema (quantization-focused), built-in discovery and path resolution, floating-point shorthand conversion, example usage (Python/CLI), authoring guidance, repository layout, and future directions. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added comprehensive guide for ModelOpt recipes, introducing declarative YAML-based optimization specifications. Documents recipe structure, configuration loading, path resolution, and the three-layer system architecture. Includes built-in recipe discovery conventions and detailed instructions for authoring custom recipes with practical examples. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Shengliang Xu <shengliangx@nvidia.com> |
||
|
Keval Morabia
|
04cd596d79 |
Add experimental support for transformers>=5.0 + min torch 2.8 (#975)
### What does this PR do? - Add experimental support for transformers >=5.0 and remove deprecated usages: https://github.com/huggingface/transformers/blob/main/MIGRATION_GUIDE_V5.md - ⚠️ For accelerate examples that used `--warmup-ratio: float` (deprecated in 5.x), we now change it to `--warmup-steps: float | int` which works as ratio if float but only for 5.x. For 4.x, it will error out if float and prompt user to change back to `--warmup-ratio` or pass an int absolute step count. - ⚠️ Unified Hugging Face checkpoint export for quantized checkpoints may not work for some models with transformers>=5.0 yet as it requires a lot of fixes (e.g. change in how MoE experts are organized) - ~Add Workaround for TRT-LLM's import of deprecated transformers functions so trt-llm based gpu unit tests work fine. Still deployment for models needs proper fixes directly in TRT-LLM hence llm/vlm ptq example tests still run with transformers 4.57~ - Everything except PTQ and Export (mainly MoE) should work fine with transformers>=5.0 - Bump min torch to 2.8 and enable 2.11 cicd testing - NOTE: Upcoming Nemo:26.04 container comes with transformers 5.3 ### Testing <!-- Mention how have you tested your change if applicable. --> - [x] CI/CD tests passing - [x] Manually tested unit tests, gpu tests with transformers 4.56 and 5.4 - [x] Manually tested example tests (except trt-llm container tests) with transformers 4.56 and 5.4 - [x] 2-gpu nightly CICD tests manually triggered and passing: [gpu tests](https://github.com/NVIDIA/Model-Optimizer/actions/runs/23867257540), [example tests](https://github.com/NVIDIA/Model-Optimizer/actions/runs/23867260643) ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, using `torch.load(..., weights_only=True)`, avoiding `pickle`, etc.). - Is this change backward compatible?: ✅ <!--- If ❌, explain why. --> - If you copied code from any other source, did you follow IP policy in [CONTRIBUTING.md](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md#-copying-code-from-other-sources)?: N/A <!--- Mandatory --> - Did you write any new necessary tests?: ✅ <!--- Mandatory for new features or examples. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Make remote-code usage opt-in via a configurable --trust_remote_code flag across examples and tools. * **Bug Fixes** * Improve checkpoint/resume detection and related training guidance to avoid erroneous errors. * **Refactor** * Consolidate dtype/config naming, switch warmup settings from ratio → steps, and unify tokenizer invocation patterns. * **Documentation** * Simplify changelog title and add misc notes for release 0.44. * **Chores** * Remove scheduled PR-branch cleanup workflow and relax/remove several transformers version pins. * **Tests** * Adjust test gates, skips, and structures to align with updated deps and behaviors. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
Shengliang Xu
|
1cceb950d6 |
[OMNIML-3689] PTQ quant_cfg semantic correction. Design in doc _quant_cfg.rst (#1094)
### What does this PR do?
#### Summary
Redesigns the `quant_cfg` configuration format in ModelOpt's PyTorch
quantization stack, replacing the previous dict-based format with an
**ordered list of typed `QuantizerCfgEntry` dicts**.
##### Motivation
The old `quant_cfg` dict had several pain points:
- **Ambiguous precedence**: no explicit way to reason about which entry
wins when multiple keys match a quantizer
- **Mixed key namespaces**: wildcard paths and PyTorch class names lived
in the same dict level, requiring ad-hoc dispatch
- **Magic `"default"` key**: an implicit, undocumented catch-all that
was easy to misuse
- **Poor composability**: merging two configs required dict updates that
silently discarded keys
- **No YAML round-trip fidelity**: the nested structure couldn't be
expressed cleanly in YAML
##### New format
`quant_cfg` is now an ordered list of `QuantizerCfgEntry` TypedDicts.
Each entry has:
- `quantizer_name` *(required)*: `fnmatch` wildcard matched against
quantizer module names
- `cfg` *(optional)*: dict (or list of dicts) of
`QuantizerAttributeConfig` fields
- `enable` *(optional)*: toggles quantizer on/off independently of `cfg`
- `parent_class` *(optional)*: restricts match to quantizers whose
parent module is of the given PyTorch class (e.g. `"nn.BatchNorm2d"`)
Entries are applied in list order; later entries override earlier ones.
The canonical pattern is deny-all first (`_base_disable_all`), then
selectively re-enable and configure, then apply standard exclusions
(`_default_disabled_quantizer_cfg`).
##### Changes
**Core library (`modelopt/torch/quantization/`)**
- **`config.py`**:
- Added `QuantizerCfgEntry` TypedDict (line 163) and
`find_quant_cfg_entry_by_path()` helper for exact-match lookup of
entries by path.
- Added `normalize_quant_cfg_list()` (line 1539) that converts legacy
formats (flat dict, single-key dicts, `nn.*`-scoped dicts, `"default"`
key) to canonical `QuantizerCfgEntry` lists. After normalization every
entry is guaranteed to have explicit `quantizer_name`, `enable`, and
`cfg` keys.
- Converted `_default_disabled_quantizer_cfg` and
`_mamba_moe_disabled_quantizer_cfg` from dicts to lists of
`QuantizerCfgEntry`.
- Added `_base_disable_all` (line 205): canonical deny-all entry
(`[{"quantizer_name": "*", "enable": False}]`).
- Converted all ~30 built-in config constants (`INT8_DEFAULT_CFG`,
`FP8_DEFAULT_CFG`, `NVFP4_DEFAULT_CFG`, etc.) to list format using
`*_base_disable_all` and `*_default_disabled_quantizer_cfg` unpacking.
- KV-cache configs (`FP8_KV_CFG`, `NVFP4_KV_CFG`, etc.) are now minimal
lists designed to be concatenated with a primary config — they
intentionally omit `_base_disable_all` and `"algorithm"`.
- Added two `QuantizeConfig` Pydantic field validators: a
`mode="before"` validator that calls `normalize_quant_cfg_list()`, and a
`mode="after"` validator that validates `cfg` dicts against
`QuantizerAttributeConfig`.
- Updated `need_calibration()` to iterate the normalized list instead of
the old dict.
- Changed `QuantizeQuantCfgType` alias from `dict[str | Callable, ...]`
to `list[QuantizerCfgEntry]`.
- **`conversion.py`**:
- Rewrote `set_quantizer_by_cfg()` (line 217) to iterate the list
directly. Each entry's `parent_class` is resolved via
`QuantModuleRegistry[parent_class_name]` (the existing `_DMRegistryCls`
registry).
- Added `set_quantizer_attributes_full()` (line 314): full replacement
of quantizer attributes from a `QuantizerAttributeConfig`. Unspecified
fields revert to defaults, enforcing entry atomicity. Can also upgrade
`TensorQuantizer` → `SequentialQuantizer` or downgrade the reverse.
- Added `set_quantizer_attributes_partial()` (line 384): merges a
partial `dict` of attributes into existing quantizer state. Does NOT
change quantizer structure. Used for enable-only entries.
- Added `set_quantizer_by_cfg_context()` context manager (line 447) that
temporarily applies a `quant_cfg` list and restores original quantizer
state on exit.
- Deprecated `set_quantizer_attribute()` (line 525) with a
`DeprecationWarning` pointing to the new functions.
- **`tensor_quantizer.py`**:
- `TensorQuantizer.set_from_attribute_config()`: narrowed type hint from
`dict` to `dict[str, Any]`.
- Added `_axis_setter` and `_block_sizes_setter` custom setters so that
`axis` and `block_sizes` changes properly propagate to the calibrator
and maintain mutual exclusivity.
- `SequentialQuantizer.set_from_attribute_config()`: narrowed signature
to `list[QuantizerAttributeConfig] | list[dict[str, Any]]` (removed the
old union with single values).
- **`algorithms.py`**:
- Updated `_match_quantizer_cfg()` to iterate the list and return
`(matched_cfg, matched_enable)` tuple with last-match-wins.
- Updated `_cfg_to_dict()`, `estimate_quant_compression()`, and
`QuantRecipe` to work with the list-based format.
- Updated `get_auto_quantize_config()` to emit list-format `quant_cfg`.
- **`model_quant.py`**: `disable_quantizer()` / `enable_quantizer()` now
call `set_quantizer_attributes_partial()` directly instead of the
deprecated `set_quantizer_attribute()`. Updated docstrings and code
examples to show the list format.
- **`utils/core_utils.py`**: `disable_lora_quantizers_in_config()` and
`update_quant_cfg_with_kv_cache_quant()` updated to append
`QuantizerCfgEntry` dicts to the list.
- **Other**: minor updates to `backends/fp8_per_tensor_gemm.py`,
`backends/nvfp4_gemm.py`, `compress.py`, `model_calib.py`,
`export/unified_export_hf.py`, and
`sparsity/attention_sparsity/conversion.py` to use the list format.
- **`onnx/llm_export_utils/quantization_utils.py`**: Updated
quantization config construction to use list format.
**YAML recipes (`modelopt_recipes/`)**
- Converted all 5 general PTQ recipes to the new list format:
- `general/ptq/fp8_default-fp8_kv.yml`
- `general/ptq/nvfp4_default-fp8_kv.yml`
- `general/ptq/nvfp4_experts_only-fp8_kv.yml`
- `general/ptq/nvfp4_mlp_only-fp8_kv.yml`
- `general/ptq/nvfp4_omlp_only-fp8_kv.yml`
- Converted model-specific recipe:
`models/Step3.5-Flash/nvfp4-mlp-only.yaml`
**Documentation (`docs/`)**
- New guide: `docs/source/guides/_quant_cfg.rst` — comprehensive
reference covering entry format, ordering semantics, entry atomicity,
`enable` vs `cfg` independence, `parent_class` filtering, and common
patterns (deny-all-then-enable, customizing a built-in config, building
from scratch).
- Updated `_pytorch_quantization.rst` code examples to show the list
format with `copy.deepcopy` and `.append()`.
- Added `_quant_cfg.rst` to the quantization guide table of contents.
**Examples**
- Updated all quantization examples to use the list format:
`deepseek/ptq.py`, `diffusers/quantization/config.py`,
`llm_ptq/hf_ptq.py`, `llm_qat/main.py`, `vllm_serve/vllm_ptq_utils.py`,
`llm_autodeploy/run_auto_quantize.py`, `llm_eval/quantization_utils.py`,
`llm_ptq/example_utils.py`,
`windows/torch_onnx/diffusers/qad_example/sample_example_qad_diffusers.py`,
and 2 notebooks.
**Tests**
- New test file:
`tests/unit/torch/quantization/test_config_validation.py` — unit tests
for `need_calibration()`, `normalize_quant_cfg_list()` (new format,
legacy format conversions, error cases),
`find_quant_cfg_entry_by_path()`, `_match_quantizer_cfg()`, and
`QuantizeConfig` Pydantic validators.
- Extended `tests/unit/torch/quantization/test_quantize_cpu.py` with
tests for `set_quantizer_attributes_full()` (atomicity, parent_class
filtering, SequentialQuantizer creation), list ordering, enable-only
entry behavior, and end-to-end legacy dict format.
- Updated 20+ existing test files across `tests/unit/`, `tests/gpu/`,
`tests/gpu_megatron/`, and `tests/_test_utils/` to use the list format.
##### Backward compatibility
`normalize_quant_cfg_list()` is called automatically by the
`QuantizeConfig` Pydantic `mode="before"` validator, so existing code
passing the old dict-based format (flat dict like `{"*weight_quantizer":
{"num_bits": 8}}`, single-key dict lists, or `nn.*`-scoped dicts with
`parent_class` semantics) continues to work without modification. The
legacy `"default"` key is converted to `quantizer_name: "*"`.
`set_quantizer_attribute()` is preserved as a deprecated wrapper around
`set_quantizer_attributes_partial()`.
#### Test coverage
- **Unit tests**: new `test_config_validation.py` with tests for
normalization, validation, path lookup, and cfg matching. Extended
`test_quantize_cpu.py` with tests for full/partial attribute setting,
ordering, atomicity, and legacy backward compatibility.
- **System testing**:
```
python examples/llm_ptq/hf_ptq.py \
--model Qwen/Qwen3-8B \
--recipe general/ptq/fp8_default-fp8_kv \
--export_path=build/fp8_default-fp8_kv42 \
--calib_size=16 \
--batch_size=0 \
--trust_remote_code \
--export_fmt=hf
```
### Additional Information
<!-- E.g. related issue. -->
---------
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
|
||
|
vishalpandya1990
|
20a46e04a7 |
Update ModelOpt-with-Olive documentation to mention CUDA EP commands (#1099)
### What does this PR do? Type of change: Minor documentation update - Update documentation (olive installation instructions) to mention install commands for CUDA EP packages for ORT / ORT-genai. ### Testing - Locally checked the readme, doc. ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ / ❌ / N/A <!--- If ❌, explain why. --> - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ / ❌ / N/A <!--- Mandatory --> - Did you write any new necessary tests?: ✅ / ❌ / N/A <!--- Mandatory for new features or examples. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ / ❌ / N/A <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> ### Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated Windows Olive installation guidance to recommend CUDA-based ONNX Runtime packages instead of DirectML and added a link to ONNX Runtime’s Execution-Provider docs for alternative EPs and requirements. * Simplified Windows examples by removing explicit package install commands and pointing users to the consolidated Olive setup instructions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: vipandya <vipandya@nvidia.com> |
||
|
Keval Morabia
|
d0bf0bef96 |
Remove deprecated Nemo 2.0 references / examples (#1098)
### What does this PR do? - Remove `examples/nemo_run` and other deprecated Nemo 2.0 references - Add Megatron-Bridge example links where missing <!-- Details about the change. --> ### Testing <!-- Mention how have you tested your change if applicable. --> ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ <!--- If ❌, explain why. --> - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A <!--- Mandatory --> - Did you write any new necessary tests?: N/A <!--- Mandatory for new features or examples. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Deprecations** * Removed deprecated NeMo 2.0 support and related example flows and utilities. * **Documentation** * Updated docs and examples to emphasize Megatron-Bridge / Megatron-LM and refreshed technique/deployment guidance and links. * **New Features** * Added CLI options for additional parallelism (context/expert tensor/expert model) in Megatron-Bridge distillation. * **Chores** * Removed legacy CI configs and refreshed container image tags across examples and docs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
|
cb1ff321ee |
Add Python 3.13 support (#1048)
Fixes https://github.com/NVIDIA/Model-Optimizer/issues/217 ## Summary - Bump `requires-python` from `>=3.10,<3.13` to `>=3.10,<3.14` to formally include Python 3.13 - Add explicit Python 3.10–3.13 PyPI classifiers for better discoverability - Add `py313` to tox CPU unit test and partial-install environment matrices - Add Python 3.10–3.13 to the `multi-py` CI matrix in `unit_tests.yml` ## Background Python 3.13 was previously excluded by the `<3.13` upper bound. Testing in a related repo with `--ignore-requires-python` confirmed that the library installs and runs correctly under Python 3.13. This PR lifts the restriction and wires up CI to verify it going forward. ## Test plan - [ ] CI `multi-py` job passes on `py313-torch210-tf_latest-unit` - [ ] `tox -e py313-torch210-tf_latest-unit` passes locally (requires Python 3.13 installed) - [ ] `tox -e py313-partial-unit-torch` passes locally - [ ] No regressions on existing Python 3.10/3.11/3.12 matrix jobs 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Extended Python support: minimum remains 3.10; added official support up through 3.13 (upper bound advanced accordingly). * **Tests** * CI and test matrix expanded to include experimental Python 3.13 coverage. * **Documentation** * Installation docs and changelog updated to reflect Python 3.13 support. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Ivan Basov <ibasov@nvidia.com> Signed-off-by: Ivan Basov <5455484+ivanbasov@users.noreply.github.com> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
Gwena Cunha
|
69c0d47946 |
[OMNIML-3252][ONNX] MOQ + Autotune moq integration docs (#1026)
### What does this PR do? **Type of change**: documentation **Overview**: This PR updates the documentation and does some folder re-structuring and file re-naming related to https://github.com/NVIDIA/Model-Optimizer/pull/951. ### Usage Documentation ### Testing Documentation ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ - Did you write any new necessary tests?: N/A <!--- Mandatory for new features or examples. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ (renamed `AutoQDQ` to `Autotune`) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Renamed AutoQDQ to Autotune across guides and changelog. * Updated Autotune guide descriptions and wording. * Added a new section on optimizing Q/DQ node placement with Autotune, including CLI usage and API links (appears twice in one README). * Applied minor grammar and capitalization corrections. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: gcunhase <4861122+gcunhase@users.noreply.github.com> |
||
|
|
695c8e8522 |
Integrate Automated QDQ placement tool - part 4.3 (#843)
## What does this PR do? This PR upload user guide of Automated QDQ placement tool. This tool automatically search QDQ insertion points with better performance. **Overview:** ? ## Usage <!-- You can potentially add a usage example below. --> ```python # Add a code snippet demonstrating how to use this ``` ## Testing <!-- Mention how have you tested your change if applicable. --> ## Before your PR is "*Ready for review*" <!-- If you haven't finished some of the above items you can still open `Draft` PR. --> - **Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)** and your commits are signed. - **Is this change backward compatible?**: Yes/No <!--- If No, explain why. --> - **Did you write any new necessary tests?**: Yes/No - **Did you add or update any necessary documentation?**: Yes/No - **Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?**: Yes/No <!--- Only for new features, API changes, critical bug fixes or bw breaking changes. --> ## Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added comprehensive guide for Automated Q/DQ Placement Optimization workflow, including quick start instructions, advanced usage patterns, configuration options, best practices, and troubleshooting. * **New Features** * Exposed public API for CLI parser programmatic access. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Will Guo <willg@nvidia.com> Signed-off-by: Gwena Cunha <4861122+gcunhase@users.noreply.github.com> Co-authored-by: Gwena Cunha <4861122+gcunhase@users.noreply.github.com> Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
Keval Morabia
|
860d0b4a70 |
Merge Linux and Windows Changelog (#954)
Since we no longer have any compiled packages, all releases are for all platforms so we dont have separate windows releases hence merging changelog as well. Going forward, windows can test on latest linux version and if fixes needed, they can go in next usual monthly linux release <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Reorganized changelog structure to consolidate Windows and Linux release information in a unified view. * Expanded Windows Support documentation across recent release versions. * **Chores** * Updated Windows example release badge to dynamically reflect the latest PyPI release version. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
Keval Morabia
|
82f1d216d1 |
Add Security and IP related contributing guide and configure coderabbit to catch such issues (#935)
### What does this PR do? - Add Security related coding practices in `SECURITY.md` and merge with `2_security.rst` - Update `CONTRIBUTING.md` for instructions to follow if copying code from other repositories - Update PR template - Cleanup dependency files - New API `mto.load_modelopt_state` doing the insecure `torch.load(f, weights_only=False)` instead of doing it separately everywhere. This also allows us to later improve the input validation for `modelopt_state_path` or use safer alternatives to `torch.load` ### Testing <!-- Mention how have you tested your change if applicable. --> N/A ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=True)`, `pickle`, etc.). - Is this change backward compatible?: ✅ <!--- If ❌, explain why. --> - If you copied code from any other source, did you follow IP policy in [CONTRIBUTING.md](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md#-copying-code-from-other-sources)?: ✅ <!--- Mandatory --> - Did you write any new necessary tests?: NA <!--- Mandatory for new features or examples. --> - Did you add or update any necessary documentation and update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: NA <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Expanded and reorganized security guidance and contributor procedures; updated PR template and several READMEs with clearer security, submission, and installation instructions * Replaced an older security document with an enhanced, centralized security guidance * **Chores** * Adjusted example dependency lists and optional extras (adds, removals, and version constraints) * Enabled automated incremental reviews, added pre-merge security checks, and introduced a knowledge-base of coding/security guidelines <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
Gwena CunhaandKeval Morabia
|
3fe7e65b70 |
[5615343,5597780,5371126] Upgrade ORT to 1.24 (#928)
## What does this PR do? **Type of change:** Bug fix **Overview:** Upgrade ORT to 1.24.x to fix various bugs (5615343, 5597780, 5371126). TODO: Verify no regressions by bumping ORT once @ajrasane is back ## Usage See each bug. ## Testing See each bug. ## Before your PR is "*Ready for review*" <!-- If you haven't finished some of the above items you can still open `Draft` PR. --> - **Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)** and your commits are signed. - **Is this change backward compatible?**: Yes <!--- If No, explain why. --> - **Did you write any new necessary tests?**: No - **Did you add or update any necessary documentation?**: No - **Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?**: Yes <!--- Only for new features, API changes, critical bug fixes or bw breaking changes. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Bug Fixes * Upgraded ONNX Runtime to version 1.24.2, addressing multiple reported issues * Updated system requirements and installation documentation to reflect the new ONNX Runtime version <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: gcunhase <4861122+gcunhase@users.noreply.github.com> Signed-off-by: Gwena Cunha <4861122+gcunhase@users.noreply.github.com> Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
Keval Morabia
|
9e23c6c312 |
Upgrade Dev containers for CICD to latest (#891)
## What does this PR do? - Upgrade CICD test containers to latest - Enable torch 2.10 testing in CICD ## Testing <!-- Mention how have you tested your change if applicable. --> CI/CD in this PR should pass <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for mixed-precision gradient handling with FSDP2. * **Documentation** * Updated Linux installation guide with CUDA 13.x support and cupy dependency guidance. * **Chores** * Updated CI/CD workflows and test infrastructure to support PyTorch 2.10 and CUDA 13. * Updated container image versions and test environment configurations. * Updated TensorRT-LLM version requirements. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
jingyu-ml
|
10efcb65f6 |
[3.1/4] Diffusion Quantized ckpt export - WAN 2.2 14B (#855)
## What does this PR do?
**Type of change:** documentation <!-- Use one of the following: Bug
fix, new feature, new example, new tests, documentation. -->
**Overview:**
1. Added multi‑backbone support for quantization: --backbone now accepts
space- or comma-separated lists and resolves to a list of backbone
modules.
2. Introduced PipelineManager.iter_backbones() to iterate named backbone
modules and updated get_backbone() to return a single module or a
ModuleList for multi‑backbone.
3. Updated ExportManager to save/restore per‑backbone checkpoints when a
directory is provided, with {backbone_name}.pt files, and to create
target directories when missing.
4. Simplified save_checkpoint() calls to rely on the registered
pipeline_manager by default.
**Usage: **
```bash
python quantize.py --model wan2.2-t2v-14b --format fp4 --batch-size 1 --calib-size 32 \
--n-steps 30 --backbone transformer transformer_2 --model-dtype BFloat16 \
--quantized-torch-ckpt-save-path ./wan22_mo_ckpts \
--hf-ckpt-dir ./wan2.2-t2v-14b
```
Plans
- [x] [1/4] Add the basic functionalities to support limited image
models with NVFP4 + FP8, with some refactoring on the previous LLM code
and the diffusers example. PIC: @jingyu-ml
- [x] [2/4] Add support to more video gen models. PIC: @jingyu-ml
- [x] [3/4] Add test cases, refactor on the doc, and all related README.
PIC: @jingyu-ml
- [ ] [4/4] Add the final support to ComfyUI. PIC @jingyu-ml
## Testing
<!-- Mention how have you tested your change if applicable. -->
## Before your PR is "*Ready for review*"
<!-- If you haven't finished some of the above items you can still open
`Draft` PR. -->
- **Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)**
and your commits are signed.
- **Is this change backward compatible?**: No <!--- If No, explain why.
-->
- **Did you write any new necessary tests?**:No
- **Did you add or update any necessary documentation?**: Yes
- **Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?**:
Yes <!--- Only for new features, API changes, critical bug fixes or bw
breaking changes. -->
## Additional Information
<!-- E.g. related issue. -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Unified Hugging Face export support for diffusers pipelines and
components
* LTX-2 and Wan2.2 (T2V) support in diffusers quantization workflow
* Comprehensive ONNX export and TensorRT engine build documentation for
diffusion models
* **Documentation**
* Updated to clarify support for both transformers and diffusers models
in unified export API
* Expanded diffusers examples with LoRA fusion guidance and additional
model options (Flux, SD3, SDXL variants)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Jingyu Xin <jingyux@nvidia.com>
|
||
|
vishalpandya1990
|
4a848c4f52 |
Modelopt-windows documentation update (#812)
## What does this PR do? Documentation **Overview:** - Update support matrix, changelog, deployment page, example readmes as per recent feature and model support on Windows side. ## Testing - No testing, its just documentation change ## Before your PR is "*Ready for review*" <!-- If you haven't finished some of the above items you can still open `Draft` PR. --> - **Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)** and your commits are signed. - **Is this change backward compatible?**: Yes/No <!--- If No, explain why. --> - **Did you write any new necessary tests?**: Yes/No - **Did you add or update any necessary documentation?**: Yes/No - **Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?**: Yes/No <!--- Only for new features, API changes, critical bug fixes or bw breaking changes. --> ## Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added ONNX Mixed Precision Weight-only quantization (INT4/INT8) support. * Introduced diffusion-model quantization on Windows. * Added new accuracy benchmarks (Perplexity and KL-Divergence). * Expanded deployment with multiple ONNX Runtime Execution Providers (CUDA, DirectML, TensorRT-RTX). * **Bug Fixes** * Fixed ONNX 1.19 compatibility issue with CuPy during INT4 AWQ quantization. * **Documentation** * Updated installation guides with system requirements and multiple backend options. * Reorganized deployment documentation with comprehensive execution provider guidance. * Expanded example workflows with improved setup instructions and support matrices. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: vipandya <vipandya@nvidia.com> |
||
|
Keval Morabia
|
792806f00e |
Add Security considerations in docs (#803)
Add security considerations to docs suggested by Nvidia Security team <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * Added comprehensive security considerations documentation for ModelOpt covering multiple risk areas including untrusted input handling, deserialization safety, input validation, resource exhaustion prevention, data protection for transit and storage, logging and observability practices, supply chain security, and structured mitigation approaches with practical implementation examples. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
Gal Hubara-Agam
|
391f6cb004 |
[5750013][5591945][5360813]: AutoCast standalone implementation for type inference (#719)
## What does this PR do? **Type of change:** New feature **Overview:** AutoCast runs full type inference to get the new types after adding casts. ONNX doesn't have a separate function for type inference, and it is done as part of shape inference. Shape inference is a much more complex task than type inference, especially when dynamic shapes are involved. We're seeing some shape inference related bugs in AutoCast. Typically we can WAR, but it's cumbersome. A local implementation might allow users to WAR shape inference related issues. This is opt-in and marked as experimental. ## Usage python -m modelopt.onnx.autocast --onnx_path /path/to/input.onnx [options] --use_standalone_type_inference ## Testing Added use_standalone_type_inference=True to all existing PrecisionConverter tests. ## Before your PR is "*Ready for review*" <!-- If you haven't finished some of the above items you can still open `Draft` PR. --> - **Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)** and your commits are signed. - **Is this change backward compatible?**: Yes - **Did you write any new necessary tests?**: Yes - **Did you add or update any necessary documentation?**: Yes - **Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?**: Yes <!--- Only for new features, API changes, critical bug fixes or bw breaking changes. --> ## Additional Information A more permanent fix would be to decouple type and shape inference in ONNX, we should invest in that when we have the resources - see https://github.com/onnx/onnx/issues/7100 . This is a quick fix, which is also why it is opt-in and not the default mode. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added `--use_standalone_type_inference` flag to ONNX AutoCast, enabling type-only inference as an alternative to standard shape inference. Useful as a workaround when shape inference fails or to reduce computational overhead. * **Documentation** * Added "Type Inference Control" section with usage examples and caveats for the new standalone type inference option. * **Tests** * Extended test coverage to validate both standard and standalone type inference paths. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Gal Hubara Agam <96368689+galagam@users.noreply.github.com> |
||
|
Keval Morabia
|
9de4877d27 |
[1/2] Address security concerns in code (#626)
- [x] Address feedback on Threat and Vuln Analysis (TAVA) doc by ProdSec team - [x] Add note on safe usage of pickle deserialization of modelopt-generated state files **TODO: [Separate PR]** Replace pickle usage in `modelopt/torch/opt/plugins/megatron.py` - Needs fix on TransformerEngine first as we copy from https://github.com/NVIDIA/TransformerEngine/blob/3ff0b8d4/transformer_engine/pytorch/module/base.py#L863 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added `--trust_calibration_data` CLI flag for secure ONNX quantization with pickle data files. * **Improvements** * Enhanced security validation for generated quantization code. * Simplified data loading by removing pickle-based caching—data is now always loaded fresh. * Added security guidance throughout model state loading operations. * **Documentation** * Updated guides with security best practices for model state handling. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
Keval Morabia
|
10d0fec907 |
Minitron pruning refactor [1/2]: Remove num_query_groups pruning and simplify megatron dynamic modules for follow-up PRs (#690)
## What does this PR do? - Deprecate `num_query_groups` parameter pruning support in `mcore_minitron` mode. This parameter pruning was not part of the paper but added hacky support in the past which makes refactoring current implementation a bit challenging hence removed its support. - Besides, `num_query_groups` should be multiple of TP so its less likely to be pruned. - Users can still prune it if they want to by using ModelOpt 0.40.0 or earlier - Going forward, `num_query_groups` if passed in pruning `export_config` should be same as original else raises an Exception. - As a follow-up PR, I will de-couple DynamicModule with forward hooks / hparam importance logic so its easy for users to change to different scoring logic ## Testing <!-- Mention how have you tested your change if applicable. --> - [x] CI/CD Tests pass - [x] Compare MMLU for pruning Qwen3-8B (nmm-sanbox) with previous and current implementation ## Before your PR is "*Ready for review*" <!-- If you haven't finished some of the above items you can still open `Draft` PR. --> - **Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)** and your commits are signed. - **Is this change backward compatible?**: No. See reasoning above <!--- If No, explain why. --> - **Did you write any new necessary tests?**: No - **Did you add or update any necessary documentation?**: Yes - **Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?**: Yes <!--- Only for new features, API changes, critical bug fixes or bw breaking changes. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **Deprecations** * num_query_groups parameter in Minitron pruning is deprecated. Use ModelOpt 0.40.0 or earlier if pruning with this parameter is needed. * **Documentation** * Updated NAS search parameters documentation * Refined pruning scope documentation for Transformer, MoE, and Mamba models * Updated example configurations and support matrix reflecting current supported pruning parameters <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> |
||
|
Ajinkya Rasane
|
51c3614db0 |
[OMNIML-2244] Add E2E example for mixed precision quantization and ONNX export (#656)
## What does this PR do? **Type of change:** New Feature **Overview:** - Enable ONNX export for auto quantized models - Update documentation and changelog ## Usage <!-- You can potentially add a usage example below. --> ``` python torch_quant_to_onnx.py --quantize_mode=auto \ --onnx_save_path=./vit_base_patch16_224.nvfp4_fp8.onnx \ --calibration_data_size 64 \ --auto_quantization_formats NVFP4_AWQ_LITE_CFG FP8_DEFAULT_CFG \ --batch_size 128 ``` ## Testing <!-- Mention how have you tested your change if applicable. --> ``` python evaluate.py --onnx_path=vit_base_patch16_224.nvfp4_fp8.onnx \ --model_name=vit_base_patch16_224 \ --results_path=./results.txt \ --batch_size 128 ``` Accuracy results ``` The top1 accuracy of the model is 84.15% The top5 accuracy of the model is 97.396% ``` Reference accuracy for fp16 ``` The top1 accuracy of the model is 85.102% The top5 accuracy of the model is 97.526% ``` ## Before your PR is "*Ready for review*" <!-- If you haven't finished some of the above items you can still open `Draft` PR. --> - **Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CONTRIBUTING.md)** and your commits are signed. - **Is this change backward compatible?**: Yes - **Did you write any new necessary tests?**: No - **Did you add or update any necessary documentation?**: Yes - **Did you update [Changelog](https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CHANGELOG.rst)?**: Yes <!--- Only for new features, API changes, critical bug fixes or bw breaking changes. --> --------- Signed-off-by: ajrasane <131806219+ajrasane@users.noreply.github.com> |
