mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-16 05:21:00 +08:00
pull-request/2240
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Keval MorabiaandClaude Opus 5
|
7ff81dd795 |
Add day-0 verbosity gate + harden release skills from a live run (#2254)
### What does this PR do?
Type of change: Bug fix, new feature, documentation
Day-0 release treats verbosity as a hard gate, but nothing in the skill
measured it — `gate_compare.py` only does accuracy. A run could complete
Step 5 and report a publish recommendation with a gate silently
unmeasured. This adds the missing gate and folds in fixes for problems
that cost GPU-h on a live release run.
**New: `gate_verbosity.py` + Step 5b.** Pure `evaluate_verbosity()` plus
an artifact harvester, matching `gate_compare.py`'s conventions and
`failure_class` vocabulary.
Two details it encodes, both of which produced wrong verdicts before
they were understood:
- **Read `response_stats.avg_completion_tokens`.** The
`reasoning.*_tokens` fields are always `0`, which reads as "the harness
never captured tokens" and pushes you to word counts. Only the
reasoning/content *split* is missing, not the total. Words disagree with
the gate: one task read **+6.10% FAIL** in words and **+1.32% PASS** in
tokens.
- **Two run-hygiene filters.** Pooling a mismatched reasoning-effort run
reported **+43.06% FAIL** on a task that is **+1.24% PASS** matched; one
truncated run (n=200 vs 294) reported **+9.58% FAIL** on a task that is
**+1.71% PASS** without it. Tasks with no common sample count are
reported `not_comparable` rather than as a delta.
**Skill hardening**, each from a specific failure:
- **Step 2b canary** — poll ceiling must exceed load time (a 50 min poll
against a 51 min load failed a checkpoint that serves fine), print an
explicit `RESULT:` on every path (a fall-through exits 0 and reads as
PASS), log to shared storage, and canary the **as-exported** artifact
rather than a copy modified to make it work.
- **Step 4 config parity** — assert the candidate config differs from
the baseline's in nothing but checkpoint path and served-model name. A
mismatched `parallelism` was worth ~2 pp, enough to invert the sign of a
delta, and cost four re-runs.
- **Statistical power** — re-running does not guarantee fresh samples:
with a warm NEL response cache two runs came back bit-identical to 16
digits.
- **Step 6 closeout** — verify the published path against the evaluated
one by inode, and prefix rejected sibling exports.
- **Size gate** — growth is blocking by default and waived only when the
validation summary's
`source_precision` shows an already-sub-8-bit source (which cannot
shrink further under a
4-bit recipe) and the growth is within what that explains.
`source_precision` is now a
recorded field in the ptq validation table, so the waiver is reachable
from the normal
pipeline, and `SIZE_NOT_REDUCED` has a triage row pointing at declaring
it.
- **`ptq.py`** — `--calib_seq` matters more than `--calib_size`, and
`--mse_calibrate` is a no-op under `--cast_mxfp4_to_nvfp4` (it tunes
weight quantizers only, and the cast overwrites `weight_scale`).
- **`.gitignore workspaces/`** — the skills create scratch directories
inside the repo; nothing excluded them.
### Usage
```bash
python "$SKILL_DIR/scripts/gate_verbosity.py" \
--baseline <baseline_eval_root> --candidate <candidate_eval_root> \
--glob 'eval_*' --threshold 0.05
```
Exit codes match the sibling gates: `0` pass, `1` the gate ran and
failed, `2` the gate could not
read its input (wrong root, `--glob` matched nothing, everything
excluded). Prints per-task tokens,
delta, `within_threshold`, `sample_count`, run counts,
`dropped_mismatched_runs`, any
`truncated_comparison`, `not_comparable`, `harvest_diagnostics`, and a
`max_abs_delta` summary.
### Testing
- 8 new unit tests in `test_gates.py`, one per real failure mode
(two-sided threshold, partial-run filtering, unequal sample counts,
short-output warning, one-sided tasks, empty input). Full suite: **36
passed**, no GPU or network.
- `gate_verbosity.py` validated end-to-end against a real day-0 run's
artifacts: reproduces the hand-computed result (`max_abs_delta =
0.0171`, pass) and correctly marks the two unequal-sample tasks
`not_comparable`.
- `pre-commit run --files <changed>` clean, including ruff, mypy,
bandit, markdownlint, and the `.claude/skills` symlink sync.
- Verified `max_sample_length` is a real `get_dataset_dataloader`
parameter with default 512, matching `--calib_seq`'s default, so
existing `ptq()` callers are unaffected.
### Before your PR is "*Ready for review*"
- Is this change backward compatible?: ✅ — `--calib_seq` defaults to the
pre-existing 512; `gate_ptq.py` reclassifies size growth from
`QUANT_COVERAGE_FAILURE` to `SIZE_NOT_REDUCED`, which is a more precise
class for an already-4-bit source and is covered by a new test asserting
a real coverage failure still outranks it.
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A — no new
dependencies, stdlib only.
- Did you write any new necessary tests?: ✅ — 8 new tests for the new
gate.
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A
- Did you get Claude approval on this PR?: ❌ — not yet run.
### Additional Information
The measured figures come from a completed day-0 NVFP4 release
qualification. Model-specific results were removed from the general
skills where the rule stands on its own; two references were kept
deliberately — a model card citation illustrating per-scenario sampling,
and a model-specific vLLM MoE kernel crash where the model name *is* the
evidence.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable calibration sequence-length limits for DeepSeek-V4
quantization.
* Added automated release checks for output verbosity, evaluation
comparability, serving readiness, and configuration parity.
* **Bug Fixes**
* Improved quantization size-ratio reporting by distinguishing
explainable growth from blocking failures.
* Clarified handling of deployment memory-access errors and infeasible
evaluations.
* **Documentation**
* Expanded guidance for calibration, remote execution, workspace
management, evaluation setup, deployment troubleshooting, and
statistical reliability.
* Added task-specific guidance for SciCode and GDPVal feasibility
checks.
* **Chores**
* Excluded workspace session directories from version control.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Chenjie LuoandClaude Opus 5
|
73d7784223 |
docs(eval-skill): add NVFP4 model-card sampling reference (#2224)
### What does this PR do? Type of change: documentation Adds `plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md` — the published `temperature` / `top_p` / max generation length for the **2026 NVFP4 checkpoints under [huggingface.co/nvidia](https://huggingface.co/nvidia/models) that disclose them** — and points `SKILL.md` Step 3 and `model-card-research.md` at it. **Why.** Config generation currently re-derives sampling params per run from one model card, which is slow and silently wrong in two ways: 1. **Quickstart boilerplate reads as an eval setting.** Most NVIDIA NVFP4 cards paste a TensorRT-LLM snippet containing `SamplingParams(temperature=0.8, top_p=0.95)`. That string is byte-identical across `Llama-3.1-8B`, `Llama-3.3-70B`, `Llama-4-Scout`, `Phi-4-reasoning-plus`, `Phi-4-multimodal`, `Qwen2.5-VL-7B` and six `Qwen3-*` repos. It is template text, not what the accuracy table was measured with — but it is the most prominent `temperature=` in the card. 2. **A generic fallback beats an available answer.** When a card is silent, Step 3 falls back to 65536/16384 even where a sibling in the same family publishes an exact value. **Scope — 25 rows.** All 69 NVFP4 checkpoints in the org were read. A row exists only for a 2026 **target** checkpoint whose card discloses usable settings, so absence means "read the card", not "not yet checked". Excluded by construction: pre-2026 releases, cards that publish nothing, and — via the curation regex `-NVFP4(-V\d+|-QAD)?$` — `-DSpark`/`-DFlash` speculative-decoding variants (verified against the target, so identical accuracy; they would only duplicate the base row), `-Eagle3` draft heads, and `-MLPerf-Inference-Closed-*` snapshots. | provenance | rows | meaning | | --- | --- | --- | | `eval` | 20 | card ties the values to its accuracy table — authoritative | | `rec` | 5 | recommended inference sampling, not tied to the eval | A `—` in one value column means that field specifically is unpublished — four rows give `temperature`/`top_p` but state no generation cap (`Mistral-Medium-3.5`, `Nemotron-3.5-Lightning`, `Nemotron-3-Super-120B`, `Nemotron-Labs-3-Elastic-30B`). Per-task exceptions are recorded where cards state them: GLM-5.2 GPQA Diamond `100000` vs `64000`; Qwen3.5-397B-V2 τ²-Bench Telecom `128000`; Qwen3.6 SciCode `temperature=0.6`; Kimi-K3 uncapped for Terminal-Bench. **Posture: the card is the source of truth; the table is a reference, not a constraint.** It is there to confirm a value you read, fill a gap when the card is silent, and catch a misreading — never to override what a card states. Listed and in agreement → proceed; listed and different → the card wins, re-read, surface the discrepancy. The `max_num_tokens` column records the card's *headline* cap, so Step 3's existing take-the-highest rule still governs when a card names more than one. Per-task `temperature`/`top_p` in the notes is precedent rather than mandate — engineers do tune sampling per benchmark — so the guidance is to follow the card and escalate only on a regime change (greedy vs sampled), not a nudge (`0.95` vs `1.0`). ### Usage Not an API change; the reference is consumed by the `evaluation` skill when generating a NEL config. ```yaml # nvidia/GLM-5.2-NVFP4 -> references/nvfp4-modelcard-sampling.md (provenance: eval) nemo_evaluator_config: config: params: max_new_tokens: 100000 # card: GPQA Diamond 100000, others 64000 -> take the highest temperature: 1.0 top_p: 0.95 ``` ### Testing **Coverage — enumeration cross-verified two ways.** Rather than paging `https://huggingface.co/nvidia/models?p=N` by hand, the candidate list came from the HF API (`author=nvidia&limit=1000` → 917 repos, one page), then verified against the website pagination: 918 unique model links across `p=0..31`, and the 68 NVFP4-named repos were **identical in both** (`api-only: []`, `web-only: []`). The committed curation regex was then re-run against that full set and confirmed to select every one of the 25 table rows plus 7 further 2026 target checkpoints whose cards disclose nothing — i.e. the documented filter reproduces the table. **Accuracy — every `eval` row diffed against its source sentence.** A script re-parsed the table and printed each row beside the matching card line. To be precise about what that covers: the diff was run when the table was larger, and all 28 machine-checkable *Benchmarked with* rows matched exactly (`Qwen3.6-35B-A3B` states its settings on the second line of a blockquote and was checked by hand). Every edit since has been a row **removal** — the scope cut to 2026 and the spec-decode removal — each verified by re-parsing the table before and after and confirming no surviving row's `temperature`/`top_p`/`max_num_tokens`/`provenance` changed. Extraction was restricted to *"Benchmarked with…"* / *"…were evaluated with…"* / *"We evaluate the model using…"* sentences, "Recommended Sampling" rows, and accuracy-table footnotes — never the quickstart snippets. **Two extraction traps found while building it**, both now in the file's refresh recipe as a second mandatory grep: some cards publish the cap **only** as a footnote beneath the accuracy table (`*Max OSL for evals can be as high as 64K`), and DeepSeek states sampling in its `## Input:` usage block rather than a *Benchmarked with* sentence. Neither is reachable from the obvious grep. **Hygiene.** `pre-commit run --files <the 3 files>` passes, including `markdownlint-cli2` and the `sync .claude/skills/ symlinks` hook; the reference resolves through both `.claude/skills/evaluation/references/` and `.agents/skills/evaluation/references/`. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A <!-- documentation-only; verification described under Testing --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A <!-- agent-skill documentation; no user-facing API or behavior change --> - Did you get Claude approval on this PR?: ✅ <!-- reviewed; findings addressed or answered below --> ### Additional Information Data collected 2026-08-20; the file is an explicitly dated snapshot and says to trust the card for anything newer. It ends with a refresh recipe (API enumeration → `-NVFP4(-V\d+|-QAD)?$` filter → `createdAt >= 2026-01-01` → card fetch → the grep phrasings that carry eval settings) so the table can be regenerated as new checkpoints ship — roughly four NVFP4 target checkpoints per month over 2026 so far. Review findings addressed: the mandatory-match framing was softened to reference-only, per-task sampling reframed as precedent with a human-escalation trigger, `mkdir -p cards` added to the recipe, the HF token no longer interpolated into a `curl` argument, and explicit uncapped generation (`max_new_tokens: null`) distinguished from an unpublished cap. The provenance question on the Nemotron-3.5-Lightning rows is answered in a review reply — the differing labels were correct (the base card says "Recommended Sampling", the sibling cards say "Benchmarked with"), and those sibling rows have since been removed as spec-decode duplicates. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Chenjie Luo <chenjiel@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Asha Anoosheh
|
94915a1e5d |
Fix typo for Tau2 eval config (#2210)
### What does this PR do? Type of change: Documentation <!-- Details about the change. --> ### Usage ```python # Add a code snippet demonstrating how to use this ``` ### Testing <!-- Mention how have you tested your change if applicable. --> ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: N/A - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A - Did you get Claude approval on this PR?: N/A ### Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated Tau2 evaluation guidance to consistently use “judge” and “user-simulator” terminology. * Clarified rate-limit guidance and the related YAML configuration terminology for telecom evaluations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> |
||
|
Chenjie LuoandClaude Opus 5
|
d32c2c2a56 |
docs(eval-skill): add MRCR (NeMo Gym) benchmark (#2192)
### What does this PR do?
Type of change: documentation (agent skill)
Adds **MRCR** — OpenAI's Multi-Round Co-reference Resolution, a
long-context
retrieval benchmark — to the `evaluation` skill as a standalone NeMo Gym
task,
derived from the reviewed `nemotron_nano_v35_nvfp4_mrcr_gym` golden;
regroups the
gym tasks/examples under one `gym/` dir; and fixes several latent bugs
in the
shared gym command block that were found by running the benchmark
end-to-end.
MRCR tasks are long multi-turn conversations containing N near-identical
"needle"
responses; the model must reproduce the Nth verbatim behind a random
prefix.
Grading is deterministic (`SequenceMatcher.ratio()`, gated on the
prefix). Unlike
GDPVal it uses the `simple_agent`: no SIF, no judge, no Tavily —
`HF_TOKEN` is the
only secret, and the cost is context length (up to 1M tokens), not agent
turns.
**It is not an AA benchmark** and is never generated for an "AA"
request.
### Layout
| File | |
| --- | --- |
| `recipes/tasks/gym/mrcr.md` | new recipe — variants, 1M serving
envelope, canary, score extraction |
| `recipes/examples/gym/example_mrcr.yaml` | new self-contained SLURM +
vLLM config |
| `SKILL.md` | MRCR branch + gym index table |
| `references/quantization-benchmarks.md` | table row + comparability
notes |
`examples/gym_{gdpval,mrcr}/` → `examples/gym/example_<task>.yaml` and
`tasks/aa_gym/gdpval.md` → `tasks/gym/gdpval.md`, all tracked as git
renames.
`aa_gym` encoded "in the AA suite" in the *path*; since GDPVal is AA and
MRCR is
not, membership is now stated explicitly in both recipe headers and an
"In AA
suite?" column in the SKILL.md index. `gym/` groups by harness, not
suite.
### Bugs fixed (found by running it, not by reading it)
The first three are in the **shared** gym command block, so they also
affect
`example_gdpval.yaml` — GDPVal was broken independently of MRCR.
1. **Invalid OmegaConf interpolation.** A literal `${...}` inside a
comment in the
gym `command:` block is parsed as an interpolation and rejected, so
*every*
`nel run --dry-run` of either gym template died with
`hydra.errors.ConfigCompositionException`.
2. **Hardcoded `ray==2.49.2`** injected into each sub-server's
requirements. Against
an image carrying `ray[default]==2.55.1` this makes `uv` unsatisfiable
and the
gym resources server exits at startup. Now derived from the image at
runtime.
3. **`--max-num-seqs` sized against the wrong topology.** The template
shipped DP1
→ 4 replicas at 64 concurrent while citing a golden that is TP2×DP2 → 8
replicas
at 32. Following it ran double the reference's per-replica load, which
on
1M-token prompts is what decides whether the KV cache fits.
4. **Score extraction pointed at an empty map.** `results.yml` →
`groups.nemo_gym.metrics` holds only `key_metrics/mean/*` telemetry; the
scores
are in `artifacts/evaluator_rollouts_aggregate_metrics.json` →
`[0].agent_metrics`.
Also documents that `pass@1/accuracy` is already 0-100 while
`mean/reward` is the
same number as a 0-1 fraction, and records `mean/prefix_matched ≈ 0.55`
as the
healthy calibration.
5. **The template shipped a container its own bootstrap rejects.** After
adding the
hard-fail on an unpinned Gym, `container:` still defaulted to public
`nemo-gym:26.05` — the image the docs say has a non-git `/opt/Gym`. Now
`???`,
so `--dry-run`'s mandatory-value check catches it instead of the job
dying at
startup.
### Review feedback
All items from @meenchen and CodeRabbit addressed or answered inline.
Two worth
surfacing here:
- **`process_reasoning_traces` vs `use_reasoning`** — verified against
`nemo_evaluator/adapters/adapter_config.py`: both exist, `use_reasoning`
is the
**deprecated** one and they are bidirectionally aliased. Documented
rather than
switched to the deprecated name.
- **`tiktoken` / `transformers` left unpinned** — deliberate. The n3
prepare path
uses `transformers.AutoTokenizer` to decide which samples exceed the
cap, so a
bump can shift dataset membership; but pinning would diverge from the
golden's
`pre_cmd` and therefore from the run that produced the reference number.
Risk is
now documented under "Deferred, know the risk" instead of being
implicit.
Topology, `gres` and TP/DP were aligned to the **existing** sibling
templates
rather than to a new convention: `gres` stays a comment (already the
convention in
`example_eval.yaml` and `example_gdpval.yaml`), TP is a concrete `1`
like both
siblings, and `num_nodes`/`num_instances`/`--max-num-seqs` are guidance
rather than
baked values. `--max-num-seqs` sizing follows **AA-LCR**, since MRCR is
the same
KV-bound problem at ~1M tokens vs LCR's ~120K: the formula gives a
ceiling, not a
target, because oversubscribing causes preemption and recomputing a
1M-token
prefill makes the run slower.
### Usage
```bash
cp plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml mrcr.yaml
# fill checkpoint_path / served_model_name / container / SLURM ??? values
export NEMO_EVALUATOR_TRUST_PRE_CMD=1 NEMO_EVALUATOR_TRUST_UNLISTED_TASKS=1
nel run --config mrcr.yaml --dry-run && nel run --config mrcr.yaml
```
### Testing
Docs/config-only; no library code touched. `pre-commit` clean on every
commit.
- Both gym YAMLs parse; asserted the folded-scalar rule (no `#` inside
`>-`), that
every string survives `OmegaConf.create` (bug 1), and that the variant
is
identical in `data_prep_params` and `collect_rollout_params`.
- After the rename: zero stale `aa_gym`/`gym_gdpval`/`gym_mrcr` refs and
every
`recipes/**` path referenced across both skill trees resolves on disk —
this
caught `env.example`, which an extension-filtered grep missed. The
`nemo_gym_gdpval_stirrup_agent` metric names contain the substring
`gym_gdpval`
and were deliberately left untouched.
- **Ran the benchmark end-to-end.** A full run on gcp-nrt (B200)
completed
2363/2363 rollouts and scored; that run produced bugs 2 and 4 and the
corrected
metric paths. A second attempt on aws-cmh was preempted and later
cancelled.
`--dry-run` now passes (it did not before bug 1 was fixed), and the bare
template
correctly fails validation on unresolved mandatory values.
- Regenerated the config from scratch with a fresh agent against the
fixed
templates as a regression check; its dry-run passed and its findings
drove the
topology/`gres`/TP-DP alignment above.
Model-specific scores are deliberately **not** recorded in the recipe —
the
reference there stays the golden's BF16 Nano 3.5 shape, since quoting a
different
model's number beside it invites exactly the false comparison the recipe
warns
about.
### Before your PR is "*Ready for review*"
- Backward compatible?: ✅ — additive plus a doc-tree rename; all
referencing files
updated and verified. The template fixes change only broken behaviour.
- Copied code / new PIP dependency: N/A.
- New tests: N/A — agent-skill documentation.
- Changelog: N/A — skill/docs change; consistent with prior skill-only
commits.
- Claude approval: ❌ not yet — will trigger `/claude review`.
### Additional Information
Upstream drift flagged to reviewers (**no change made to that repo**):
in
`nvidia-eval-factory-benchmarking`, `configs/benchmarks/mrcr/bench.yaml`
is still
old-style `ng_*` while `configs/models/nemotron_nano_v35/gym.yaml` moved
to the
`gym eval` CLI on 2026-07-29 — RULER migrated, MRCR did not, so the
layers no
longer compose. This template follows `ng_*`, which is what MRCR's own
bench.yaml
specifies and what the sign-off run executed.
NVIDIA-internal companion (container specifics kept out of this public
tree):
Model-Optimizer-Internal MR !116.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Keval MorabiaandClaude Opus 5
|
fbcdc16c2d |
Remove deprecations marked in 0.45 and 0.46 (#2182)
### What does this PR do?
Type of change: Backward breaking change (deprecation removal)
Ahead of the 0.47 code freeze, this removes every deprecation still
outstanding from the previous two releases (0.45 and 0.46). Two are
intentionally left in place: the **Python 3.10** drop and the
**transformers 4.x** drop
| Deprecation | Marked in | Replacement |
| --- | --- | --- |
| `--auto_quantize_bits` / `_method` / `_score_size` / `_cost_model` /
`_active_moe_expert_ratio` | 0.46 | AutoQuantize `--recipe` |
| `examples/llm_ptq` symlink + `examples/vlm_ptq/` forwarder | 0.46 |
`examples/hf_ptq` (`--vlm` for VLMs) |
| `QuantizationArgumentsWithConfig` alias | 0.45 |
`QuantizationArguments` |
| `QFORMAT_ALIASES` short names | 0.45 | canonical preset basenames |
| `layerwise` bool + flat `layerwise_checkpoint_dir` | 0.45 | nested
`layerwise: {enable, checkpoint_dir}` |
| in-trainer `quant_cfg` / `--quant_cfg` | 0.45 | `--recipe` |
#### Two things worth a closer look
**1. The `use_sequential` alias goes too.** It is the pre-#1251 alias on
`QuantizeAlgorithmConfig.layerwise` and only ever carried a bool. Once
the bool form is rejected it cannot accept a valid value, so keeping it
would only produce a differently-worded validation error. Note the
direction is breaking either way (`extra="forbid"`): a pre-0.45
`modelopt_state` carrying `use_sequential: True` or a top-level
`layerwise_checkpoint_dir` now fails validation instead of being
migrated.
**2. Removing in-trainer `--quant_cfg` required two new recipes.** The
`examples/gpt-oss` QAT flow ran on `--quant_cfg
MXFP4_MLP_WEIGHT_ONLY_CFG` and no `general/ptq/` recipe covered it. This
PR adds `general/ptq/mxfp4_mlp_weight_only` and
`general/ptq/nvfp4_mlp_weight_only`, verified to `model_dump` identical
to `mtq.MXFP4_MLP_WEIGHT_ONLY_CFG` / `mtq.NVFP4_MLP_WEIGHT_ONLY_CFG`,
and migrates the gpt-oss README, both SFT configs, `sft.py` and
`tests/examples/gpt-oss/test_gpt_oss_qat.py`. `examples/llm_qat` was
already recipe-only.
### Usage
```bash
# AutoQuantize: --auto_quantize_* flags -> an AutoQuantize recipe
scripts/huggingface_example.sh --model $HF_PATH \
--recipe general/auto_quantize/nvfp4_fp8_at_5p4bits --calib_batch_size 4
# --qformat / --quant_cfg: short name -> canonical preset basename
# int8_sq -> int8_smoothquant nvfp4_mse -> nvfp4_w4a4_weight_mse_fp8_sweep
# int8_wo -> int8_weight_only nvfp4_local_hessian -> nvfp4_w4a4_weight_local_hessian
# w4a8_awq -> w4a8_awq_beta fp8_pb_wo -> fp8_2d_blockwise_weight_only
# nvfp4_awq -> nvfp4_awq_lite fp8_pc_pt -> fp8_per_channel_per_token
scripts/huggingface_example.sh --model $HF_PATH --quant int8_smoothquant
# VLM PTQ: examples/vlm_ptq -> examples/hf_ptq with --vlm
scripts/huggingface_example.sh --model $HF_PATH --quant fp8 --vlm
# gpt-oss QAT: --quant_cfg <CFG name> -> --recipe <recipe path>
accelerate launch --config_file configs/zero3.yaml sft.py \
--config configs/sft_full.yaml --model_name_or_path openai/gpt-oss-20b \
--recipe general/ptq/mxfp4_mlp_weight_only --output_dir gpt-oss-20b-qat
```
```python
# Layerwise calibration: bool / flat key -> nested LayerwiseConfig
quant_cfg["algorithm"] = {"method": "gptq", "layerwise": {"enable": True, "checkpoint_dir": "/path"}}
```
### Testing
- `tests/unit/recipe` (229 passed),
`tests/unit/torch/quantization/test_config_validation.py` (79 passed),
`tests/examples/hf_ptq/test_hf_ptq_args.py` (23 passed).
- Verified the two new recipes `model_dump` identical to the `mtq.*_CFG`
constants they replace.
- `ruff check modelopt/ examples/ tests/` clean; `ruff format --check`
clean on all changed Python files.
- GPU suites
(`tests/gpu/torch/export/test_unified_hf_export_and_check_safetensors.py`,
`test_accelerate_gpu.py`, `test_gptq.py`) had their preset / layerwise
literals updated but were not run locally — relying on CI.
- `examples/llm_qat/ARGUMENTS.md` is hand-edited to match what the
`generate-arguments-md` hook emits; the generator could not run locally
(missing `transformers` package metadata in this environment).
### Before your PR is "*Ready for review*"
- Is this change backward compatible?: ❌ — that is the point of the PR:
it removes shims deprecated in 0.45/0.46. Callers must move to the
replacements in the table above. Additionally, a pre-0.45
`modelopt_state` carrying `use_sequential` or a top-level
`layerwise_checkpoint_dir` will now fail config validation.
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: ✅ — existing tests migrated to
the surviving APIs;
`TestLayerwiseNestedConfig::test_legacy_forms_rejected` pins that the
bool form, the `use_sequential` alias and the flat checkpoint-dir key
are all rejected. Tests covering the removed shims were deleted.
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
✅
- Did you get Claude approval on this PR?: ❌
### Additional Information
Follow-up: the transformers 4.x drop deprecated in 0.46 is still
outstanding and will need its own PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## New Features
- Added MXFP4 and NVFP4 weight-only quantization recipes for MLP and MoE
layers.
- Added shared layer exclusions for more accurate effective-bits
calculations.
## Improvements
- Updated PTQ, QAT, GPT-OSS, deployment, and quantization-format
examples with current recipe names and configuration formats.
- Standardized layerwise settings under nested configuration fields.
## Breaking Changes
- Removed deprecated AutoQuantize options, `quant_cfg` usage, format
aliases, legacy layerwise settings, and compatibility example paths.
- Recipe-based and nested configuration forms are now required.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Zhiyu
|
53ccec614f |
Pin GDPVal evaluator launcher to 0.2.6 (#2188)
### What does this PR do? Type of change: Bug fix Pins GDPVal evaluation commands to the currently validated `nemo-evaluator-launcher==0.2.6` through a dedicated `uvx` wrapper and updates the GDPVal skill, recipe, example, and reference documentation to use it. The reference also documents how to validate and adopt a future launcher release without mixing harness versions in baseline-versus-candidate comparisons. Older launchers generate a runtime re-export for `NEL_INVOCATION_ID` without first assigning the launcher-created invocation ID. The resulting Slurm script exits under `set -u` with `NEL_INVOCATION_ID: unbound variable` before the evaluation client starts. Launcher 0.2.6 writes the stable invocation ID into the script before the re-export, including for walltime-resume chains. ### Usage ```bash "$SKILL_DIR/scripts/nel-gdpval.sh" --version "$SKILL_DIR/scripts/nel-gdpval.sh" run --config gdpval.yaml --dry-run "$SKILL_DIR/scripts/nel-gdpval.sh" run --config gdpval.yaml ``` ### Testing - `bash -n plugins/modelopt/skills/evaluation/scripts/nel-gdpval.sh` - Verified the wrapper reports `nemo_evaluator_launcher: 0.2.6`. - Ran a real GDPVal dry-run and verified the generated script assigns `NEL_INVOCATION_ID="<invocation-id>"` before the runtime re-export. - Added a hermetic shell-wrapper test with a stub `uvx`; a conflicting ambient conflicting ambient version/spec values still invoke the validated 0.2.6 package. - `python -m pytest plugins/modelopt/skills/evaluation/ -o addopts="" -p no:cacheprovider -v` - Ran pre-commit on all six changed files. ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors). - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: ✅ — added a hermetic wrapper test that stubs `uvx` and verifies the exact package spec despite a conflicting environment value. - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A - Did you get Claude approval on this PR?: N/A ### Additional Information The fix preserves a stable GDPVal cache namespace across Slurm walltime resumes; using `SLURM_JOB_ID` would not. The failing run reused an existing base-environment `nel` 0.2.4 because the pre-existing prerequisite checked only whether `nel` was installed. The skill already identified 0.2.6 as the default launcher for Steps 1–9 and GDPVal. Version 0.2.6 is also the latest public `nemo-evaluator-launcher` release, so there is no newer public launcher version to validate. A dedicated `uvx` wrapper keeps the fix narrow and deterministic: - `nel-next.sh` manages the separate `nemo-evaluator` 0.4.x package, CLI, and schema; generalizing it would couple incompatible workflows. - A skill-level lock/project adds state for a single exact package invocation, while `uvx --from ...==0.2.6` already provides an isolated cached environment. - A version preflight would only reject the stale shared environment or require mutating it; the wrapper guarantees the known-good launcher on every GDPVal invocation without changing the existing `nel`. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added a GDPVal launcher wrapper that consistently uses the validated launcher version and forwards commands. - Added help output and checks for required runtime tooling. - **Documentation** - Clarified GDPVal launch, dry-run, reproducibility, resumed-run, and invocation ID guidance. - Documented version requirements and known sample limitations. - Updated evaluation examples and task instructions with the new launch procedure. - **Tests** - Added coverage verifying the pinned launcher version and argument forwarding. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com> |
||
|
ZhiyuandClaude Opus 5
|
6261f854aa |
docs: rebuild the unified HF deployment support matrix from the deploy test suite (NVBug 6550792) (#2087)
### What does this PR do? Type of change: documentation Fixes [NVBug 6550792](https://nvbugspro.nvidia.com/bug/6550792) / OMNIML-5693. The **Unified HF Checkpoint Deployment Model Support Matrix** listed 9 model families and **no VLMs**, while `tests/examples/hf_ptq/test_deploy.py` declares deployment cases for ~80 checkpoints across TRT-LLM, vLLM, and SGLang — including `Qwen2.5-VL`, `Qwen3-VL-235B`, and `Nemotron-3-Nano-Omni`. QA (the filer) could not use the doc to scope testing, and users could not tell what is actually covered. Filing also surfaced that the matrix lived in **three places that had drifted apart**: only the `.rst` listed Qwen3-VL, only the README listed Qwen3.5 MoE, and the skill reference had neither. #### Changes 1. **Rebuilt the matrix in `docs/source/deployment/3_unified_hf.rst`** from `test_deploy.py`, split into language models, vision-language/multimodal, speculative decoding drafters, and diffusion. 2. **Stated plainly what the matrix is and is not.** Review established that the original "CI-validated" framing claimed more than the suite substantiates, so a *What this matrix is based on* section now leads with two limits: - The suite is marked `release` and collects only under `--run-release`, which **no workflow passes** — these are declared cases, not PR-gated coverage. - Each case is a **load-and-generate smoke check on the text path**: no accuracy, no image/audio input, no diffusion output, no verification that speculative decoding engages. The legend follows from that: ✅ = declared in the suite, ⚠ = expected to work but not a suite entry (or an entry that does not exercise the feature the row names), `-` = not in the suite. Sections that would otherwise over-read carry their own qualifiers — VLM rows are labelled text-only smoke coverage, and Medusa and Wan 2.2 are ⚠ with the reason stated. 3. **Removed the two duplicate copies**, replacing them with links, so there is one table to maintain. 4. **Fixed stale prose**: the deployment tabs still claimed FP8-only support on vLLM v0.6.5 and a source build of SGLang main from Jan 2025, both contradicting the version table above them. The TRT-LLM floor moves to v1.2.0, qualified as the oldest version stated rather than the oldest that works. 5. **Dropped the Phi series** from the deployment matrix, following #2115 (NVBug 6563509) and confirmation that Phi-4 is being deprecated. ### Usage N/A — documentation only. ### Testing - `docutils` parse of the modified `.rst`: no warnings or errors from the new content; all 5 tables parse with every cell in the correct column. - Cell contents cross-checked against `test_deploy.py` by AST-parsing the `ModelDeployerList(...)` calls rather than by eye; the scope caveats were each verified against `tests/_test_utils/deploy_utils.py`. - `pre-commit run --files …` passes; `build-docs` green. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A — documentation only - Did you get Claude approval on this PR?: ❌ — not yet run ### Additional Information **Two known follow-ups, neither in scope here:** 1. **Nothing enforces that the doc matrix tracks `test_deploy.py`.** Consolidating to one copy removes the three-way drift but not the doc-vs-test drift; a generator plus a CI check would close it. 2. **The release deployment suite does not run in CI.** Wiring it into per-backend release CI is what would let ✅ mean "verified to pass" rather than "declared". That needs GPU capacity across three backends and should be tracked on its own. **For the filer (@Kenny Kang):** the ✅ cells are the scope the release deploy suite declares, and `test_deploy.py` carries the checkpoint, TP size, and minimum SM version per entry — but please read the legend first, since those cases are not currently executed by CI. --------- Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Chad Voegele
|
d3fe8117ff |
Add ModelOpt agent plugin marketplace (#2025)
### What does this PR do? Type of change: new feature Packages the existing ModelOpt agent skills as installable Codex and Claude plugins: - Adds a repo-scoped Codex marketplace and Claude-compatible marketplace. - Adds the canonical `plugins/modelopt/` plugin tree and manifests. - Moves the skill tree into the plugin and keeps `.agents/skills` as a compatibility symlink. - Adds a minimal `common` placeholder skill required by Codex validation. - Documents installation from this repository. ### Usage ```bash codex plugin marketplace add NVIDIA/Model-Optimizer ``` Then open `/plugins`, select the `modelopt` marketplace, and install `modelopt`. For Claude Code: ```bash claude plugin marketplace add https://github.com/NVIDIA/Model-Optimizer.git claude plugin install modelopt@modelopt ``` ### Testing - Codex plugin validator - `claude plugin validate . --strict` - `claude plugin validate plugins/modelopt --strict` 1. Install the marketplace plugin with Codex and Claude from an unrelated temporary workspace. 2. Exercise packaged evaluation helpers, a day-0 gate, and the shared remote helper from that workspace. 3. Run `uv run --frozen --extra dev python -m pytest -q plugins/modelopt/skills/day0-release/tests/test_gates.py plugins/modelopt/skills/benchmark-model-kernels/tests`. 4. Run pre-commit hooks for all changed files. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: ✅ — added a plugin-path validator; existing focused skill tests and installed-plugin smoke tests pass. - Did you update Changelog?: N/A — agent tooling and distribution only. - Did you get Claude approval on this PR?: N/A ### Additional Information Skills remain available through `.agents/skills`; bundled helpers are packaged under the plugin and resolved from `$SKILL_DIR` so installed workflows do not depend on the current workspace. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added installable ModelOpt plugins for Claude Code and Codex. * Added skills for PTQ, deployment, evaluation, monitoring, debugging, benchmarking, MLflow access, EAGLE3 workflows, and release management. * Added deployment helpers, evaluation recipes, checkpoint validation, and release-gating tools. * **Documentation** * Expanded setup, credential, SLURM, benchmarking, deployment, evaluation, troubleshooting, and workspace guidance. * Added installation instructions and updated agent-skill discovery guidance. * **Maintenance** * Updated skill references and compatibility links for reliable use across supported plugin environments. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Chad Voegele <cvoegele@nvidia.com> |