mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-16 05:21:00 +08:00
pull-request/2602
24
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ye YuandClaude Opus 5
|
c75fe4b9c6 |
skills(specdec): point at examples/speculative_decoding
Every launcher script the skill tells you to run wraps that directory, but the tree referenced it only twice, both inside dflash.md. Triage and configure had no route from a wrapper to the code it calls, and the non-launcher path was invisible. Signed-off-by: Ye Yu <yeyu@nvidia.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Ye Yu <yeyu@nvidia.com> |
||
|
Ye YuandClaude Opus 5
|
b8cbcab8a3 |
skills(specdec): add DFlash2 and LiLiCorr algorithm sheets
Three recipes shipped after the skill tree landed in #2201 and had no sheet: dflash2.yaml, lilicorr.yaml and lilicorr_conv.yaml. The stage procedures already applied to them; only the per-algorithm values were missing. dflash2.md and lilicorr.md follow the variant shape README.md defines -- delta only, pointing at dflash.md for the shared pipeline. One sheet covers both LiLiCorr recipes, since lilicorr_conv.yaml is the same algorithm with DFlash2's sublayer convolutions rather than a separate one; the delta has its own section. Facts are sourced from the recipes, the launcher examples and the plugin validation paths rather than from the papers, so the Known failures tables carry the error strings a run actually prints. Signed-off-by: Ye Yu <yeyu@nvidia.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Ye Yu <yeyu@nvidia.com> |
||
|
sychen52
|
be74001256 |
Add concise AgentX benchmark skill (#2573)
### What does this PR do? Type of change: Documentation. Adds a concise AgentX skill covering harness installation, automatic dataset downloads, benchmark execution, and result reporting. Reuses the existing deployment skill and adds a Claude discovery link. ### Usage Use run-agentx to benchmark my deployed model with a concurrency sweep. ### Testing • Skill structure and metadata validation passed. • Shell syntax checks passed. • Benchmark arguments parsed and produced a valid configuration using the pinned harness. • All applicable pre-commit checks passed. • No GPU benchmark was launched. ### Before your PR is "Ready for review" Contributor guidelines and security practices were reviewed. The commit is signed and signed off. • Is this change backward compatible?: ✅ • If you copied code from other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md?: N/A. No copied implementation or project dependency changes. • Did you write any new necessary tests?: N/A. Documentation changes were validated as described above. • Did you update Changelog?: N/A. Skill documentation only. • Did you get Claude approval on this PR?: ❌ Not run. ### Additional Information <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added guidance for configuring and running SemiAnalysis AgentX serving benchmarks, including endpoint, model, tokenizer, context limits, caching, and dataset setup. * Documented using a pinned benchmark harness in a separate client environment and running each concurrency level in a fresh artifact directory with a fixed seed. * Expanded reporting guidance to cover overlapping requests, cache and preemption metrics, errors, unfinished requests, warmup failures, and submission validity. * Clarified that missing server-reported usage makes cache-hit data unknown, invalid or missing submission validity should be flagged, and smoke runs are not benchmark results. * Directed AgentX agentic workloads from the optional AIPerf guidance to the AgentX benchmark instructions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Shiyang Chen <shiychen@nvidia.com> |
||
|
Chenjie LuoandClaude Opus 5.5
|
80e04b8816 |
docs(eval): align Terminal-Bench 2.1 / SWE-bench / MRCR with upstream configs (#2479)
### What does this PR do? Type of change: documentation (agent skill) + template bug fixes Aligns the `evaluation` skill's three upstream-tracked benchmarks (Terminal-Bench 2.1, SWE-bench Verified, MRCR) with the current `nvidia-eval-factory-benchmarking` configs, and fixes guidance that turned out to be wrong when a full three-benchmark campaign was run with the skill end to end. Rebased on #2499. The first commit is the alignment. The rest address review and a fresh config-generation test: MRCR serving scoped per variant, `limit_samples` canary guidance, the template's serve command passing `--gpu-memory-utilization` (replacing `command:` had silently dropped it, so the 1M golden's 0.95 never reached vLLM), upstream's 128K values, and regression tests for the `++limit` gate and the serve command. The skill text keeps only the rules; the evidence behind them is below. GDPVal is out of scope. It was removed from this skill in #2470, and this PR replaces #2464. **Alignment with upstream** - Sandbox region via `HARBOR_ECS_REGION`. TB2.1's ECR repo name tracks the region; SWE-bench's stays in us-west-2. - One interceptor order for both harbor benchmarks, with `http_pairs_dump` **last**. Upstream is split on its position, which changes only what the dump records, never the score. Interceptor lists replace wholesale on merge, so a leaf must restate the whole chain. - `capture_request_body` goes on the service. A shared block injects an alias-only entry with no `type`. - MLflow tags gain `task_name` and `nemo-evaluator-next-version`. - TB2.1 `max_concurrent`: 50 for nano-class models, 15 for larger ones (all upstream non-nano leaves override it to 15). - `proxy.request_timeout` must always be set explicitly. Otherwise it inherits the model fragment's serving value, which ranges from 3600 to 36000 upstream, and TB2.1 has no benchmark key for it. - MRCR: - `parallelism` is 512, deliberately above server capacity, so `--max-num-seqs` must no longer be derived from it. - `limit_samples` now reaches the gym through a gated `++limit`. - Observability capture is on. - 128K is its own upstream benchmark on the condensed gym schema. **Corrections found by running it** | what the skill said | what actually happens | |---|---| | `username: ${oc.env:USER}` | nel-next only expands `${VAR}` / `${VAR:-default}`. The config passes `--dry-run` and fails at `--submit` with *"remote username contains invalid characters"*. | | MRCR canary via `++limit` edited into `collect_rollout_params` | `limit_samples` is now gated through. Under the 0.2.6 launcher the `-o` path is `++evaluation.nemo_evaluator_config.config.params.limit_samples`. `++config.params…` creates a bogus top-level key. | | the condensed gym schema's bootstrap is in the runtime image | It lives in upstream `configs/models/gym_eval_command.yaml`, which is composed in. A standalone config must carry the `command:` block. | | `mean/prefix_matched ~0.55 is healthy` | That value is calibrated to the 1M golden. A 128K run at `pass@1` ≈ 95 measured ≈ 1.0. The signal is a collapse toward 0. | | NVFP4 MoE `VLLM_*` env vars as reliable knobs | They are build-dependent: one vLLM build logged them as unknown and ignored them. Check the server log once per image. | **Rules the skill lacked** - **MRCR variant.** Pick the largest variant within the checkpoint's trained context. On a 262K-context model, 1M needs `VLLM_ALLOW_LONG_MAX_MODEL_LEN` and measures extrapolation, which a quantization comparison would then entangle with quantization damage. The serving setup follows the variant: 128K serves at the trained context without the override. - **SWE-bench `reasoning_effort`.** openhands-sdk sends `reasoning_effort: high` on every call, and canonical `bench.yaml` doesn't strip it. A server whose accepted set excludes `high` returns HTTP 400 on the first call of every trial, so `pass@1` is 0. The fix is to overwrite it with the server default via `proxy.extra_body`. terminus-2 (TB2.1) and Gym's `simple_agent` (MRCR) never send the key (46/46 and 110/110 requests checked). - **Reasoning toggles** go in `extra_body.chat_template_kwargs`. Don't write out no-op sampling defaults; `top_k` is *not* one (vLLM's default is `-1`). - **Upstream model fragment.** Consult `configs/models/<model>/` when it exists, for serving flags, the thinking toggle and `reasoning_replay.mode`. ### Usage No API change. Regenerating a config from the skill now yields the aligned values: ```yaml # recipes/examples/example_eval_next.yaml services: model: proxy: request_timeout: 3600 # always explicit benchmarks: - max_concurrent: 50 # nano-class; larger models 15 sandbox: region: ${HARBOR_ECS_REGION:-us-east-1} cluster: username: ${USER} # NOT ${oc.env:USER} ``` ### Testing - `python -m pytest plugins/modelopt/skills/ -o addopts=""`: 7/7 pass, including the new `tests/test_example_mrcr.py`. It checks that `example_mrcr.yaml` emits `++limit=N` only when `limit_samples` is set, and that the folded vLLM serve command shell-parses with every flag intact, including `--gpu-memory-utilization`. Each check fails when its defect is reintroduced. - `markdownlint-cli2` on the changed Markdown: 0 errors. Both example YAMLs parse. The full `pre-commit` suite was not run after the squash, because the sandbox could not fetch hook repos. The pre-squash commits passed it. - **Exercised end to end.** Configs built from this skill ran a BF16 campaign for a 262K-context MoE reasoning model on an internal cluster to completion: - MRCR-128K: 1470/1470 rollouts - Terminal-Bench 2.1: 712/712 trials - SWE-bench Verified: 2500/2500 trials Each correction above is a defect that campaign surfaced. - **Differential check.** Terminal-Bench configs generated from the pre- and post-change skill, from the same brief and in isolation, differ on: - interceptor chain - concurrency - `top_k` - region interpolation - MLflow tags Not run: a scored evaluation of this PR itself. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ <!-- Docs + a template fix; no ModelOpt API surface touched. --> - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A <!-- Documentation and config-template values. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A <!-- Agent skill docs, not a user-facing ModelOpt feature/breaking change/deprecation. --> - Did you get Claude approval on this PR?: ❌ <!-- Not run. --> ### Additional Information The companion internal `eval-config` change now carries only the internal values: the ECR URLs, the region default and the cluster image notes. It points here for the generic rules. **Known gaps not fixed here**, worth a follow-up: - `references/nel-next.md:136` and `references/launcher-workflow.md:207` derive `--max-num-seqs` from `parallelism / DP`. nel-next has no `parallelism` field (its analogue is `max_concurrent`), and MRCR's 512 is deliberately not a server cap. - `references/launcher-workflow.md:200-201` makes `--max-num-batched-tokens` and `--enable-chunked-prefill` always-include defaults, but `example_eval_next.yaml` omits both. - `references/launcher-workflow.md:27` says `sbatch_comment` belongs under `execution:` and is otherwise inert, yet all three shipped examples put it under `cluster:`. - MoE detection (`--enable-expert-parallel`) is unresolvable from the facts the skill asks for when the model handle has no `-A*B` suffix. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Chenjie Luo <chenjiel@nvidia.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Zhiyu
|
f2ee751089 |
Validate eval limit accounting and raise TB 2.1 timeout (#2499)
### What does this PR do? Type of change: documentation Successful evaluations can still contain timed-out trials or responses stopped by output limits. Existing skills scan for errors but do not require counts or rates, allowing serving-speed effects to be mistaken for quantization accuracy changes. Require per-task timeout and output-limit accounting before scores are treated as validated, with explicit denominators, telemetry coverage, effective limits, and artifact evidence. Separate request retries, terminal trial failures, and resumable SLURM walltime events. Carry that evidence into baseline/candidate comparisons; unknown accounting or unresolved infrastructure effects prevent an acceptable verdict. Preserve benchmark-defined failures and label diagnostic protocol changes explicitly. Allow valid-with-warnings results for small fractions of limit-hit responses/trials when coverage, protocol, and other checks pass, without automatically retrying. Clarify that a BF16 acceptance gate cannot be satisfied with an FP8/INT4 baseline. Correct Terminal-Bench 2.1 guidance that sharding cannot affect scores and require accounting across the full trial set. Raise the ModelOpt TB 2.1 agent budget from 7200 to 14400 seconds in the recipe and set it explicitly in the template. With timeout_strategy=max, this is a minimum budget, not a hard ceiling. Set sandbox lifetime to six hours and example SLURM walltime to eight hours to allow setup and verification; partition limits and effective task budgets must still be checked. Baseline and candidate must use the same policy, and old two-hour results require remeasurement for a matched comparison. This changes skill defaults, not the upstream benchmark protocol or harness instrumentation. The upstream-vendored launching-evals skill is unchanged per repository policy; its standalone workflow still needs an upstream update. Reduce the evaluation entrypoint from 6,156 to 888 words (86%) by moving launcher Steps 1–8 into an on-demand reference without changing their instructions. Preserve step headings/links for existing callers. Shorten timeout accounting from 550 to 352 words (36%) while retaining the checks and warning policy. This reduces initial context; full launcher workflows still load the relevant detailed sections. ### Usage TB 2.1 recipe/template defaults now include: ```yaml solver: timeout_strategy: max run_timeout: 14400 sandbox: max_task_lifetime_sec: 21600 ``` The example uses `cluster.walltime: "08:00:00"` where the partition permits it. Request timeout remains 3600 seconds. For leaderboard comparisons, use the benchmark protocol rather than assuming the ModelOpt override is comparable. ### Testing - Pre-commit passed on all six changed Markdown/YAML files. - skill-creator quick_validate.py passed for evaluation and compare-results. - git diff --check passed. - Validated recipe and template solver/sandbox settings against NEL schemas with the pinned TB playbook; checked max and task timeout resolution. - Verified extracted launcher Steps 1–8 match the original text, apart from a trailing blank line. - Manually reviewed handling of recovered retries, resumed artifacts, incomplete telemetry, benchmark-defined limits, and timeout-sensitive comparisons. No evaluation jobs were launched; increased timeout defaults have not been benchmarked. ### Before your PR is "*Ready for review*" Contributor guidelines and security coding practices reviewed. - Is this change backward compatible?: ✅ Existing configs remain valid. New TB configs use longer budgets and may consume more runtime; comparison requires matched timeout policies. - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: N/A - Did you write any new necessary tests?: N/A — skill/config changes validated as listed above. - Did you update Changelog?: N/A — internal skill guidance. - Did you get Claude approval on this PR?: ❌ Not requested yet. ### Additional Information A nonzero benchmark-defined timeout or output-cap rate does not automatically invalidate a score. The change requires evidence and protocol-aware interpretation, without choosing a universal acceptable rate or silently excluding affected trials. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Expanded evaluation guidance with a centralized launcher workflow covering setup, configuration, execution, monitoring, authentication, and failure handling. * Required timeout and output-limit accounting for every task, including successful and non-reasoning runs, with rates, denominators, telemetry coverage, effective limits, and recovery status. * Clarified that unknown telemetry, mismatched limits, or unresolved infrastructure effects prevent an acceptable verdict. * Updated comparison guidance to require matched reruns, aligned precision baselines, and limit-hit rate comparisons. * Clarified distributed benchmark timeout handling and score extraction; GDPVal support is no longer documented. * Updated example evaluation time limits and SLURM walltime. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com> |
||
|
Zhehao-Hu
|
051d6adb20 |
Require PTQ recipe guidance before selecting quantization candidates (#2478)
### What does this PR do? Type of change: documentation Updates `quant-recipe-search` to require reading `modelopt_recipes/ptq.md` before selecting initial or subsequent quantization candidates. The skill uses the guide to inform quantization scope, KV-cache scheme, and calibration method. It requires inspecting candidate YAMLs and supporting configs, citing relevant guide sections, and explaining deviations. Existing coverage, runtime compatibility, and evaluation checks remain required. ### Usage Invoke `quant-recipe-search` as usual. The skill reads the guide from the ModelOpt source checkout. ### Testing - Passed a local Codex skill smoke test recommending an initial NVFP4 candidate for Qwen/Qwen3-8B targeting high-concurrency inference on Blackwell. - The test prompt named the skill without explicitly requesting that the agent read ptq.md. - Verified successful reads of the updated skill, ptq.md, the selected recipe YAML, and supporting configs before the final recommendation. - The agent selected general/ptq/nvfp4_mlp_only-kv_fp8_cast and cited relevant guide sections. It explicitly left coverage, accuracy, and throughput pending validation. ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A - Did you get Claude approval on this PR?: ✅ ### Additional Information This change places the reading requirement in the shared recipe-search skill so consumers receive it directly. Consumers that pin the ModelOpt plugin to a commit must update their pin to adopt it. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Updated the quantization recipe search workflow to require reviewing the PTQ reference documentation before selecting a recipe. - Clarified that available quantization schemes and model-specific exceptions should be considered during recipe evaluation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Zhehao Hu <zhehaoh@nvidia.com> |
||
|
Chenjie LuoandClaude Opus 5
|
a7166965e3 |
Drop GDPVal support from the evaluation skill (#2470)
### What does this PR do? Type of change: deprecation (agent skill) Removes GDPVal support from the `evaluation` agent skill. Its task recipe, example config and Apptainer SIF build helper are deleted. GDPVal was not cleanly separable, so this is not just a delete: - **MRCR depended on GDPVal's infrastructure.** `scripts/nel-gdpval.sh` was a generic pinned-0.2.6 `nel` launcher that only happened to be GDPVal-named, and MRCR ran through it; `references/gym-gdpval.md` documented the gym bootstrap machinery (prepare/reap, `install_on_the_fly` pin↔container coupling, the trust env vars) that both examples share. So the shared parts are kept and renamed rather than dropped: `scripts/nel-gym.sh` and `references/gym.md`. The launcher pin itself is unchanged (0.2.6), as are its env-override semantics. - **GDPVal is an AA-suite member**, so the "AA rule" in `SKILL.md` and `references/quantization-benchmarks.md` had to change. An "AA" / "Artificial Analysis" request now generates the `aa/` tasks as one multi-task config and nothing else. Both files now state that the resulting set omits GDPVal and is therefore not directly comparable to a published AA Index — report per-task scores rather than an aggregate. `SKILL.md` also tells the agent to say GDPVal is unsupported rather than reconstruct a config from an older copy of the skill. Everything GDPVal-specific is gone from `references/gym.md`: the SIF sandbox and its silent-unsandboxed-exec failure mode, the 3-member judge panel, rubric vs. comparison scoring, the deliverables/MLflow `*cache*` trap as a GDPVal concern, and Stirrup-agent deploy sizing. `recipes/env.example` loses `GDPVAL_SIF_DIR`, `GDPVAL_MAX_TURNS` and `TAVILY_API_KEY`, and gains a documented `NEMO_EVALUATOR_TRUST_UNLISTED_TASKS` (required by every gym task, previously only mentioned in prose). One correction carried along: the old reference said the bootstrap pins `ray==2.49.2`, but `example_mrcr.yaml` actually pins to whatever ray version the image already carries. `references/gym.md` now describes what the template does. ### Usage Not an API change. The skill-facing surface that moved: ```text scripts/nel-gdpval.sh -> scripts/nel-gym.sh (NEL_GDPVAL_* -> NEL_GYM_*) references/gym-gdpval.md -> references/gym.md tests/test_nel_gdpval.py -> tests/test_nel_gym.py ``` ### Testing - `pytest plugins/modelopt/skills/evaluation/tests/` — 1 passed. The launcher test was renamed rather than deleted: it is the only coverage for the pinned launcher, which MRCR still depends on. - `pre-commit run --files <changed>` — clean. `sync-claude-skills` fails in my working copy because `.claude/skills/` is a read-only harness mount there; it is unrelated to this change (it trips on `speculative-decoding`) and was skipped for the commit. - Grepped the repo for `gdpval` (case-insensitive): the only remaining hits are the deliberate "no longer supported" notes in `SKILL.md` and `references/quantization-benchmarks.md`. No dangling pointers to the deleted files, and no other skill referenced GDPVal. - Checked `tests/evals.json` for both `evaluation` and `day0-release`: no eval case expected a GDPVal companion config, so no expectations needed updating. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ❌ — anyone with a saved GDPVal config keeps it, but the skill no longer generates one and the SIF helper is gone. - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A — existing launcher test renamed and kept passing. - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ — under **Deprecations**. - Did you get Claude approval on this PR?: ❌ — not yet run. ### Additional Information A follow-up is needed in the modelopt-internal repo: `modelopttools:eval-config` Step 3c is the GDPVal SIF / comparison-mode conversion checklist, and Step 3d names the gym image. Step 3c is now dead and the pointers this skill used to make into it are gone. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Evaluation Updates** * Added standalone NeMo Gym support for MRCR tasks with pinned launcher and Gym configuration requirements. * Added shared guidance for Gym setup, validation, execution, and recovery. * AA requests now generate only `aa/` tasks and report per-task scores. * **Removed Support** * Removed GDPVal evaluation recipes, documentation, task guidance, Apptainer helper tooling, and AA-suite inclusion. * Updated default quantized-checkpoint validation recommendations to exclude GDPVal. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Chenjie Luo <chenjiel@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
yeyu-nvidiaandClaude Opus 5
|
216f28a6e0 |
Consolidate speculative-decoding agent skills into one stage/algorithm tree (#2201)
### What does this PR do?
Type of change: documentation
Reorganizes the EAGLE3 agent skills into a single speculative-decoding
skill, then adds algorithm sheets for DFlash, DSpark, and Domino.
**The problem.** The four `eagle3-*` skills each baked the algorithm
into a *stage* of the same draft-model pipeline:
```
skills/eagle3-new-model/ skills/eagle3-review-logs/
skills/eagle3-triage/ skills/eagle3-validate/
```
Adding DFlash would have meant four more near-duplicate skills, since
the stages are shared and only the algorithm differs.
**The change.** One skill dir shaped like `ptq/` (SKILL.md +
references/), split along the two real axes:
```
plugins/modelopt/skills/speculative-decoding/
├── SKILL.md # router: stage table x algorithm table
└── references/
├── stages/ # the procedure — algorithm-independent
│ ├── configure.md # <- eagle3-new-model
│ ├── review-logs.md # <- eagle3-review-logs
│ ├── triage.md # <- eagle3-triage
│ └── validate.md # <- eagle3-validate
└── algorithms/ # the data sheet — per-algorithm
├── README.md # contract: 6 required sections
├── eagle3.md
├── dflash.md
├── dspark.md # DFlash variant — delta only
└── domino.md # DFlash variant — delta only
```
Stage docs cite algorithm-sheet sections by heading (*Pipeline tasks*,
*Success markers*, *Quality gate*, *Known failures*, ...), so a new
algorithm means one new file plus a table row — no stage edits. Every
recipe in `modelopt_recipes/general/speculative_decoding/` now has a
sheet.
DSpark and Domino are documented as **DFlash variants**, not separate
pipelines: same `recipe_type: speculative_dflash`, same training script,
same `dflash.*` config namespace, selected by
`dflash_architecture_config.projector_type`. Their sheets carry only the
delta.
Writing the sheets surfaced three things the old EAGLE3-only skills got
wrong or missed:
- **Task counts are not fixed.** The old skills hardcoded "4-step
pipeline, task_0 through task_3". DFlash offline is 2 tasks, DFlash
online is 3, Domino is 2. The stage docs no longer assume a count.
- **`--aux-layers` couples the dump to the draft.** For DFlash the
dump's layer count must equal the draft's `num_hidden_layers`; a
mismatch doesn't error, it silently captures the wrong layers. Recorded
under *Known failures*.
- **In-training AR is meaningless for DSpark and Domino.** Both recipes
pin `estimate_ar: false` / `ar_validate_steps: 0` because eval runs the
DFlash backbone with the new head bypassed. Each sheet says so under
*Quality gate* so nobody reads a backbone-only number as a result.
**Behavior change:** the four `/eagle3-*` slash commands are replaced by
one `/speculative-decoding`. This isn't optional —
`tools/precommit/sync_claude_skills.sh` iterates `.agents/skills/*/` one
level deep and plugin discovery is `skills/<name>/SKILL.md`, so a
directory is either one skill or a container of skills, not both.
`tools/launcher/docs/claude_code.md` is updated accordingly.
### Usage
```
/speculative-decoding
```
Or by description — the skill triggers on EAGLE3 / DFlash / DSpark /
draft model / acceptance rate. For a new model, follow the stages in
order:
```bash
# 1. Configure: copy the closest examples/<Org>/<Model>/hf_<mode>_<algo>.yaml and adapt
cd tools/launcher
uv run launch.py --yaml examples/<Org>/<Model>/<config>.yaml --dryrun # preview
uv run launch.py --yaml examples/<Org>/<Model>/<config>.yaml --yes # submit
# 2. review-logs -> 3. triage (if anything failed) -> 4. validate
```
### Testing
- `claude plugin validate . --strict` and `claude plugin validate
plugins/modelopt --strict` — both pass
- `pre-commit run --files ...` over all changed files — passes,
including `markdownlint-cli2` and the `sync-claude-skills` symlink hook
(it agrees with the new `.claude/skills/speculative-decoding` symlink)
- Verified the new skill is discovered and its description loads
- Every relative link across the skill tree resolves; every repo path
cited in the sheets exists; no dangling `eagle3-*` reference remains
anywhere in the repo
- Each factual claim in the sheets was checked against its source — the
launcher example YAMLs, the four recipes, `dflash_online_training.sh`,
`vllm_smoke_test.sh`, `check_regression.py`, and
`plugins/hf_{dflash,dspark,domino}.py` — rather than written from memory
### Before your PR is "*Ready for review*"
- Is this change backward compatible?: ❌ — the four `/eagle3-*` slash
commands become `/speculative-decoding`. Agent tooling only; no library
or API surface is touched. The three YAML comment fixes are
comment-only, no behavior change.
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: N/A — documentation; covered
by plugin validation and pre-commit
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A — agent tooling only, matching how #2025 handled it
- Did you get Claude approval on this PR?: ✅ — `/claude review` run; 2
findings, both fixed in `9b3c568`
### Additional Information
Follows #2025, which moved the skill tree into the installable plugin.
Two stale in-repo comments were found while sourcing the sheets, and are
**fixed in this PR** (`c2696b1`, comment-only):
1. `modelopt_recipes/general/speculative_decoding/dflash.yaml` pointed
`chat_template` at a `chat_templates/` directory under
`modelopt_recipes` that does not exist — templates live per-model beside
each launcher example.
2. Both offline DFlash example YAMLs annotated `--aux-layers dflash`
with "Must match the draft model's num_hidden_layers". `--aux-layers` is
a preset keyword accepting only `eagle`, `dflash`, or an explicit id
list, so it carries no count. The constraint is real but belongs to the
draft depth the preset resolves to: `--num-draft-layers` on the vLLM
dump, and no override at all on the HF/TRT-LLM dumps, which hardcode 5
via `resolve_aux_layers`. This comment had already misled this PR's own
first draft, which is why it's fixed rather than just documented.
Because of (1), this PR now touches `modelopt_recipes/`, which adds
**@NVIDIA/modelopt-recipes-codeowners** to the required reviewers.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added comprehensive speculative-decoding guidance for configuration,
training, validation, troubleshooting, and supported algorithms.
* Added workflow references for DFlash, Domino, DSpark, and EAGLE3,
including quality checks and failure diagnosis.
* **Documentation**
* Generalized experiment-log review and pipeline triage across
algorithms.
* Clarified DFlash draft-depth configuration, resource sizing, task
recovery, and validation.
* Replaced the EAGLE3-specific workflow entry with the broader
speculative-decoding workflow.
* Removed standalone EAGLE3 skill documentation as guidance is now
consolidated under speculative decoding.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Ye Yu <yeyu@nvidia.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Chad Voegele
|
c118d359c1 |
docs: replace legacy TensorRT-LLM engine deployment guidance (#2436)
Chad's Agent ### What does this PR do? Type of change: documentation. Replace legacy TensorRT checkpoint export, support matrix, and engine-build instructions with `export_hf_checkpoint` and TensorRT-LLM's PyTorch backend. Preserve the existing 0.48.0 deprecation / 0.49.0 removal notice and page URL. Update the customized-model guide and deployment skill to match. ### Usage Follow the linked unified HF export guide. No API changes. ### Testing - `git diff --check` passed. - `uvx pre-commit run --files docs/source/deployment/1_tensorrt_llm.rst docs/source/guides/_customized_model_quantization.rst plugins/modelopt/skills/deployment/references/trtllm.md` passed. - Full Sphinx build delegated to the Docs workflow; preview expected after deployment. ### Before your PR is "Ready for review" - Is this change backward compatible?: ✅ Documentation only; page URL retained. - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A — documentation only. - Did you update Changelog?: N/A — guidance correction; no new API deprecation. - Did you get Claude approval on this PR?: ❌ Not requested yet. ### Additional Information Removes instructions for the TensorRT backend that current TensorRT-LLM releases no longer support. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Updated TensorRT-LLM deployment guidance to use `export_hf_checkpoint` with the PyTorch backend. - Clarified that this workflow does not require TensorRT engine construction. - Updated DBRX customization instructions for exporting and deploying quantized models. - Added TensorRT-LLM version requirements and links to unified Hugging Face deployment guidance. - Removed guidance for the legacy TensorRT-LLM checkpoint exporter and outdated troubleshooting steps. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Chad Voegele <cvoegele@nvidia.com> |
||
|
Shengliang Xu
|
c7ed23a103 |
Rename modelopt_recipes/huggingface to model_type with backward-compat alias (#2328)
### What does this PR do?
Type of change: Refactor + deprecation (recipe-library restructure,
backward compatible), plus an unrelated transformers-compat test fix.
Rename the architecture-specific recipe tier
`modelopt_recipes/huggingface/` to
`modelopt_recipes/model_type/`, making explicit that it holds recipes
**shared across
every checkpoint of a Hugging Face `model_type`** — as opposed to the
checkpoint-mirror
`models/<org>/<model_id>/` tier. The old `huggingface/` path keeps
working as a
deprecated backward-compat alias (a source-tree symlink plus a loader
alias), so no
saved `--recipe` path breaks.
- **Loader alias** (`modelopt/recipe/loader.py`): generalized so saved
`--recipe huggingface/<model_type>/...` paths rewrite to
`model_type/...`, alongside
the existing `huggingface/models/... -> models/...` rewrite (checked
first as the more
specific prefix). This keeps old paths resolving for pip-installed
wheels, where the
source-tree symlinks don't survive.
- **Internal `$import`s**: rewritten from `huggingface/... ->
model_type/...` inside the
shipped recipes so they resolve without the symlink — mandatory for
wheels, since
`$import` resolution goes through `config_loader` (no alias there).
- **Packaging** (`pyproject.toml`, `MANIFEST.in`): extended the
symlink-exclusion globs
so the recursive `**/*.yaml` package-data glob doesn't double-ship
recipes through the
`huggingface -> model_type` and `model_type/models -> ../models`
symlinks.
- **Docs / examples / skills / tests**: migrated all internal references
to the canonical
`model_type/`; `huggingface/` remains only in the deprecated-alias tests
and explanatory
notes.
- **Unrelated fix (2nd commit):**
`tests/unit/torch/export/test_quant_aware_conversion.py`
failed on transformers>=5.9, which dropped `base_model_prefix` from
`WeightTransform.__slots__` (the scoped-rule tests assigned it on the
now-slotted
object). Production `_scope_prefixes` already reads it via `getattr(...,
None)` and
degrades correctly, so there is no runtime change — the tests now set it
through a
helper that suppresses `AttributeError` across the supported
transformers range.
### Usage
```bash
# New canonical path
python examples/hf_ptq/hf_ptq.py --model <ckpt> \
--recipe model_type/qwen3_vl/ptq/fp8_vision-kv_none
# Old path still works (deprecated backward-compat alias)
python examples/hf_ptq/hf_ptq.py --model <ckpt> \
--recipe huggingface/qwen3_vl/ptq/fp8_vision-kv_none
```
```python
from modelopt.recipe import load_recipe
load_recipe("model_type/vit/ptq/fp8") # canonical
load_recipe("huggingface/vit/ptq/fp8") # deprecated alias, resolves to the same recipe
```
### Testing
- `tests/unit/recipe/` — **336 passed**, including the new
`test_load_recipe_huggingface_arch_backward_compat_alias` and the
updated
structural/doc tests (`test_recipe_docs.py`).
- `tests/unit/torch/export/test_quant_aware_conversion.py` — **16
passed** (was 4 failed
on transformers 5.9.0).
- Built an sdist **and** a wheel and inspected both manifests: each
recipe ships exactly
once (29 `model_type/`, 13 `models/`, 2 `timm/`, 162 total) with
**zero** `huggingface/` or
`model_type/models/` duplicates and no build error on the symlinks.
- Simulated a wheel install (symlink-free extracted tree) and confirmed
`huggingface/<arch>/...`, `model_type/...`, and `huggingface/models/...`
all resolve via
the loader alias — including a recipe that pulls internal `$import`s.
### Before your PR is "*Ready for review*"
- Is this change backward compatible?: ✅ — old `huggingface/...` recipe
paths keep resolving via the symlink + loader alias.
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: ✅ — backward-compat alias test
added; structural/doc tests updated to the new layout.
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
✅ — Deprecations entry under 0.48.0. (The transformers-compat test fix
is not changelog-worthy.)
- Did you get Claude approval on this PR?: ❌ — not yet.
### Additional Information
The `model_type/models -> ../models` symlink is kept purely as a
backward-compat alias for
old `huggingface/models/<org>/<model_id>/...` paths; `model_type/` is
otherwise
architecture-only. If we ever want it strictly architecture-only, that
symlink can be
dropped later without breaking anything, since the loader rewrites
`huggingface/models/...`
straight to the top-level `models/` tier.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added post-training quantization recipes for Gemma, Gemma 4,
MiniMax-M3, Nemotron, Qwen, Step-3.7, ViT, and other architectures.
- Added vision, multimodal, mixed-precision, and experts-only
quantization options.
- **Documentation**
- Standardized architecture-specific recipes under `model_type/` and
updated examples and guidance.
- **Compatibility**
- Legacy `huggingface/` recipe paths remain supported with deprecation
warnings.
- Local recipe files now take precedence over built-in recipes.
- Deprecated quantization-format flags warn when explicitly provided.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
|
||
|
Chad Voegele
|
30f89908f0 |
Audit missing labels before release cherry-picks (#2373)
Chad's Agent ## Summary - query NVBugs MCP for the exact `Committed_ModelOpt_<VERSION>` keyword and inspect linked PRs - audit PRs merged after the latest RC tag for missing cherry-pick labels - verify audit result pagination, report findings, and require confirmation before adding labels ## Testing - `npx --yes markdownlint-cli2@0.18.1 plugins/modelopt/skills/release-cherry-pick/SKILL.md` - exercised the 0.47.0 audit using `0.47.0rc1`; GitHub returned 13 complete results - verified the exact NVBug keyword query returned 33 bugs and inspected their comments - `git diff --check` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Updated the release cherry-pick workflow to exclude pull requests whose merge commits are already included in the target release branch. - Applied this audit to both NVBug-linked and recently merged pull request candidates. - Updated recent pull request queries to reference the selected version and release branch. - Renumbered the remaining workflow steps for consistency. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Chad Voegele <cvoegele@nvidia.com> |
||
|
Chenjie LuoandClaude Opus 5
|
b80e164472 |
Carry a checkpoint's ModelOpt PTQ run into its eval's MLflow run (#2407)
### What does this PR do? Type of change: documentation (agent skill) Follow-up to #2374, which made a tracked `hf_ptq.py --mlflow` run leave `.experiment.json` in the checkpoint it writes, naming the MLflow run that quantized it. Nothing on the eval side read that file, so an eval of a quantized checkpoint recorded no link back to the quantization that produced it. The `evaluation` skill now reads it. **Step 3** `cat`s the file as soon as `checkpoint_path` is known; **Step 4** carries it into `export.mlflow`: | `.experiment.json` field | goes to | | --- | --- | | `experiment_name` | `export.mlflow.experiment_name`, verbatim | | `run_name` / `run_id` / `run_url` | tags `modelopt_run_name` / `modelopt_run_id` / `modelopt_run_url` | | `tracking_uri`, `experiment_id` | deliberately unmapped | The `modelopt_` prefix keeps them from reading as the eval's own run. Values are quoted, or an all-digit `run_id` (or a `run_name` like `20260910`) is YAML-coerced to an int or a date. **`tracking_uri` is deliberately not inherited**, and the consequence is documented rather than implied. Evals go to whatever `$MLFLOW_TRACKING_URI` names. When the PTQ tracked to a different server — the usual case, since `modelopttools:eval-config` points evals at `mlflow.frontier-evals` while `hf_ptq --mlflow` typically writes to `mlflow-modelopt` — the inherited name creates a *same-named, empty* experiment on the eval server, and `modelopt_run_url` is the only route back to the PTQ run. Evals of one checkpoint still group under a stable name. Both cases are spelled out in Step 4 so nobody goes looking for the PTQ run beside the eval. Also updated: `references/nel-next.md` (nel-next configures MLflow through its own `export_config.mlflow`) and `accessing-mlflow` (the `tags.modelopt_run_id` query that closes the loop — without it the tag is write-only). ### Usage ```bash cat "$CHECKPOINT_PATH"/.experiment.json # absent → name the experiment as usual ``` ```yaml export: mlflow: tracking_uri: ${oc.env:MLFLOW_TRACKING_URI} # NOT the file's experiment_name: alice/hf_ptq/Qwen3.8-27B-NVFP4 # verbatim from .experiment.json tags: modelopt_run_name: '20260910-175422' modelopt_run_id: '7bec239a3a154970b062f3024a5ff20e' modelopt_run_url: 'https://<modelopt-mlflow-server>/#/experiments/36/runs/7bec239a3a154970b062f3024a5ff20e' ``` Finding every eval of a checkpoint a given PTQ run produced: ```python MLflow:query_runs(experiment_id, "tags.modelopt_run_id = '<ptq_run_id>'") ``` ### Testing Docs-only, so it was tested by having an agent follow the new wording end to end and checking what NEL actually submitted. A GPQA Diamond config (single repeat) was generated against an NVFP4 checkpoint carrying a `.experiment.json`, dry-run clean, then submitted on a SLURM cluster. Verified in the `export_config.yml` heredoc **inside the submitted `run.sub`** — the file the export job actually consumes, not just the source YAML: ```yaml experiment_name: chenjiel/hf_ptq/Qwen3.8-27B-NVFP4 # inherited verbatim tags: modelopt_run_name: 20260910-000000-synthetic-test-fixture modelopt_run_id: 00000000fake0000fake0000fake0000 modelopt_run_url: https://<modelopt-mlflow-server>/#/experiments/999/runs/... tracking_uri: https://<eval-mlflow-server>/ # env var, not the file's ``` This confirmed the open question behind the change: NEL accepts arbitrary `modelopt_*` keys in `export.mlflow.tags`. The `.experiment.json` used was a synthetic fixture, since the checkpoints predate #2374. Repeated on a **second cluster with a second checkpoint** (different account, QoS-based scheduler, FP8 rather than NVFP4) and carried through to completion, so the result is confirmed as *stored by MLflow* rather than merely submitted. Canary scored `gpqa pass@1 symbolic_correct: 90.0` (card: 88.01) with `no_answer: 0.0`, then the auto-export wrote: ``` experiment chenjiel/hf_ptq/Qwen3.8-27B-FP8 (id 2106) run eval-<invocation>-ns_gpqa FINISHED modelopt_run_name 20260910-000000-synthetic-test-fixture modelopt_run_id 00000000fake0000fake0000fake0000 modelopt_run_url https://<modelopt-mlflow-server>/#/experiments/999/runs/... ``` Experiment 2106 did not exist on the eval server beforehand — the export created it. That is the shadow-experiment case this PR documents, observed rather than predicted. The same exercise corrected the first draft of the wording: it had claimed the rule makes a quantization and its evals "sit together on one server", which is false whenever the two servers differ. Confirmed by API — `chenjiel/hf_ptq/Qwen3.8-27B-NVFP4` returns `RESOURCE_DOES_NOT_EXIST` on `mlflow.frontier-evals`. Rewritten as described above. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ — additive guidance; the absent-file path is unchanged. - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A — skill documentation; validated by a real submission, above. - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ❌ — agent-skill guidance, not a library/example feature. - Did you get Claude approval on this PR?: ❌ ### Additional Information Follow-up to #2374. Three adjacent problems were found while validating and deliberately **left out of scope** — happy to split them into their own PR: 1. `recipes/examples/example_eval.yaml` puts `sbatch_comment` under a top-level `cluster:` key, but the executor reads `cfg.execution.sbatch_comment` (`executors/slurm/executor.py:688`) and SKILL.md Step 1 says the same. As shipped the template silently drops the idle-GPU reaper exemption, so long evals get reaped. 2. Step 4's `execution.gres` bullet names the wrong key for `internal/slurm/<cluster>` configs, which supply `gpus_per_node` and no `gres`; following it literally emits a redundant flag. 3. Step 3's `max_new_tokens` rules can contradict each other — "take the card's highest" can equal `max_model_len`, leaving no room for the prompt. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Clarified MLflow evaluation configuration requirements, including literal experiment and sampling values, CPU partition settings, and ModelOpt provenance. - Documented validation and fallback behavior for unresolved `${...}` placeholders in provenance values. - Added guidance for querying cross-server evaluations by run ID and handling same-named local experiments. - Clarified that provenance files are optional, independent of quantization detection, and do not control deployment flags. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Chenjie Luo <chenjiel@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Chad Voegele
|
28dc117594 |
Add Day 0 Sub-agent Roles (#2006)
### What does this PR do? Type of change: new feature Adding sub-agent definition for Day 0 workflows. I'm adding subagents as an alternative, while we evaluate which is better. They are duplicated because Claude & Codex have different formats & different locations. Deriving from a shared vendor-neutral format at build-time is too complicated. ### Usage Tell your agent "Use the modelopt_model_quantizer agent to quantize the model" ### Testing Ran a trial using Qwen-2.5, Muse Glimmer, Qwen-2.8, and GLM-5.3-Flash. ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ - Did you write any new necessary tests?: ✅ - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ / ❌ / N/A <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> - Did you get Claude approval on this PR?: ✅ / ❌ / N/A <!--- Run `/claude review`. NVIDIA org members can self-trigger for complex changes; orthogonal to CodeRabbit. --> ### Additional Information See design doc. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## New Features - Added specialized Model Optimizer agents for downloading, quantization, recipe search, evaluation, deployment, and performance benchmarking. - Added coordinated Codex and Claude Code access with validation requirements, artifact preservation, and concise handoffs. - Improved skill discovery across plugin and repository installations. ## Documentation - Clarified agent discovery, layout, and canonical editing locations. - Updated configuration guidance for supported agent definitions. ## Tests - Added synchronization checks for agent definitions and links. - Improved test compatibility for Python versions below 3.11. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Chad Voegele <cvoegele@nvidia.com> |
||
|
Chenjie Luo
|
19ce447d62 |
[skill] evaluation: mandate 8 SciCode runs and report the mean (#2327)
### What does this PR do? Type of change: documentation SciCode's repeat count was pinned to `num_repeats: 1` in #1945, which dropped its effective sample count from the original `8` (set when the recipe was written in #1561) to `1`. #2254 later documented that a single run cannot gate on — scored single-shot at `temperature 1.0`, a paired comparison moved **3.92 pp and changed sign** once repeated, and the day-0 skill's own table records a DeepSeek-V4-Pro drop reading 2.96 pp (`REGRESSION`) at 1 run versus **-0.96 pp** (`PASS`) at 8. But #2254 left the remedy as a judgment call — *"pool until the standard error is below the threshold, or report the task `INDETERMINATE`"* — so a one-run SciCode number was still reportable. This restores the original avg-of-8 statistics as a hard requirement, without reintroducing the in-run repeats that #1945 removed for a reason (repeating inside one run multiplies exposure to code-execution sandbox errors). The task keeps `num_repeats: 1` per run; the repeat budget of 8 is spent as **8 independent submissions, reported as their mean** — 8 per side for a comparison, `INDETERMINATE` below 8. Changes: - **`recipes/tasks/aa/scicode.md`** — mandatory *8 runs* section: how to submit the 8 inside a multi-task AA config, fresh-run requirement (no `run.sub` replay off a warm cache), duplicate-score check, per-run validation before averaging, and mean + `stdev/sqrt(8)` + run-count reporting. - **`compare-results` / `day0-release`** — 8 runs per side is a floor, not a variance-dependent choice. - **`references/quantization-benchmarks.md`** — the repeat-count table still listed SciCode at `num_repeats: 8`, stale since #1945. - **`evaluation/SKILL.md`** — AA rule points at the 8 submissions; the walltime section distinguishes them from the forbidden practice of splitting a heavy task across configs to dodge the 4h cap. - **`evaluation/tests/evals.json`** — behavioral eval case for the rule. ### Usage ```bash # Full AA suite (= SciCode run 1), then SciCode alone for runs 2-8. nel run --config <cfg>.yaml for _ in $(seq 7); do nel run --config <cfg>.yaml -t ns_scicode; done # Report the mean of scicode_pass_at_1_avg-of-1_subtask_accuracy over the 8 runs, # plus stdev/sqrt(8) and the run count. ``` ### Testing - `pre-commit run --files <changed files>` — all hooks pass (`markdownlint-cli2`, `check json`, symlink sync), no hook-applied modifications. - `json.load` on `evaluation/tests/evals.json` parses; 4 cases, existing three unchanged (append-only diff, original formatting preserved). - Cross-checked every remaining SciCode reference in the plugin (`grep -rn -i scicode plugins/modelopt/`) so no doc still claims `num_repeats: 8` or a variance-dependent pool size. **Ran the protocol end-to-end** on Qwen3.8-27B-FP8 (gcp-nrt, 8xB200, `temperature 1.0`), 8 independent `nel run` submissions, all `COMPLETED`, each scoring the full 80 problems / 338 subtasks. MLflow experiment 2017: | passed/338 | 168 | 167 | 166 | 164 | 163 | 161 | 157 | 155 | |---|---|---|---|---|---|---|---|---| | score | 49.70 | 49.41 | 49.11 | 48.52 | 48.22 | 47.63 | 46.45 | 45.86 | **Result 48.11, stdev 1.39, stderr 0.49.** Pooled 1301/2704 subtasks reproduces 48.1139 exactly. This is the evidence for the change: **the spread is 3.85 pp and the worst single run sits 2.26 pp from the mean**, so against a 1% gate any one of these eight, reported alone, would have been defensible and wrong. Previously this PR rested on the variance figures inherited from #2254; it now rests on a measured pool. Also validated by that campaign: an agent given only "run a full SciCode eval, follow the evaluation skill" — with no mention of the protocol — read the recipe, submitted 8 runs and reported the mean with a standard error. And the corrected score key is what made the numbers harvestable at all; the `avg-of-1` name returns nothing. Two operational findings from the run, one of which is folded into the recipe: - An MLflow export job timed out (`nel-export-ns_scicode.0`, elapsed `00:30:11` against a `00:30:00` limit) because all 8 exports hit the CPU partition at once and each reinstalls the launcher. Caused by the fan-out this PR mandates, so the recipe now warns about it and says to re-submit `export.sbatch` rather than treat the run as lost. - Out of scope, filed for separate fix: `.claude/agents` is a 0-byte read-only placeholder, so the `monitor` skill's instruction to create a session registry under it fails with `Not a directory`. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: ✅ — added `scicode-eight-run-average` to `evaluation/tests/evals.json` - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A — agent-skill guidance, not a user-facing library change - Did you get Claude approval on this PR?: ❌ — not yet run ### Additional Information Restores the sampling behavior of #1561 while keeping the sandbox-load fix from #1945 and honoring the variance evidence from #2254. Note: `origin/feature/puzzletron_v2` still carries the pre-#1945 version of this file (`num_repeats: 8`, single submission) and rewrites it to source the repeat count from `examples/llm_eval/task_contracts.yaml`. If that branch lands it will need to be reconciled with this protocol. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Clarified SciCode evaluation requirements: eight independent, valid runs per comparison side with one repeat each. - Standardized score reporting using per-run metrics, mean, standard error, and run count. - Added provenance checks and replacement runs for invalid or replayed results, including timeout recovery. - Fewer than eight valid runs are reported as **INDETERMINATE**. - Updated benchmarking guidance to distinguish separate submissions from repeated runs. - **Tests** - Added coverage for eight-run averaging, validation, replacement runs, and **INDETERMINATE** outcomes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Chenjie Luo <chenjiel@nvidia.com> |
||
|
Keval MorabiaandClaude Opus 5
|
7ff81dd795 |
Add day-0 verbosity gate + harden release skills from a live run (#2254)
### What does this PR do?
Type of change: Bug fix, new feature, documentation
Day-0 release treats verbosity as a hard gate, but nothing in the skill
measured it — `gate_compare.py` only does accuracy. A run could complete
Step 5 and report a publish recommendation with a gate silently
unmeasured. This adds the missing gate and folds in fixes for problems
that cost GPU-h on a live release run.
**New: `gate_verbosity.py` + Step 5b.** Pure `evaluate_verbosity()` plus
an artifact harvester, matching `gate_compare.py`'s conventions and
`failure_class` vocabulary.
Two details it encodes, both of which produced wrong verdicts before
they were understood:
- **Read `response_stats.avg_completion_tokens`.** The
`reasoning.*_tokens` fields are always `0`, which reads as "the harness
never captured tokens" and pushes you to word counts. Only the
reasoning/content *split* is missing, not the total. Words disagree with
the gate: one task read **+6.10% FAIL** in words and **+1.32% PASS** in
tokens.
- **Two run-hygiene filters.** Pooling a mismatched reasoning-effort run
reported **+43.06% FAIL** on a task that is **+1.24% PASS** matched; one
truncated run (n=200 vs 294) reported **+9.58% FAIL** on a task that is
**+1.71% PASS** without it. Tasks with no common sample count are
reported `not_comparable` rather than as a delta.
**Skill hardening**, each from a specific failure:
- **Step 2b canary** — poll ceiling must exceed load time (a 50 min poll
against a 51 min load failed a checkpoint that serves fine), print an
explicit `RESULT:` on every path (a fall-through exits 0 and reads as
PASS), log to shared storage, and canary the **as-exported** artifact
rather than a copy modified to make it work.
- **Step 4 config parity** — assert the candidate config differs from
the baseline's in nothing but checkpoint path and served-model name. A
mismatched `parallelism` was worth ~2 pp, enough to invert the sign of a
delta, and cost four re-runs.
- **Statistical power** — re-running does not guarantee fresh samples:
with a warm NEL response cache two runs came back bit-identical to 16
digits.
- **Step 6 closeout** — verify the published path against the evaluated
one by inode, and prefix rejected sibling exports.
- **Size gate** — growth is blocking by default and waived only when the
validation summary's
`source_precision` shows an already-sub-8-bit source (which cannot
shrink further under a
4-bit recipe) and the growth is within what that explains.
`source_precision` is now a
recorded field in the ptq validation table, so the waiver is reachable
from the normal
pipeline, and `SIZE_NOT_REDUCED` has a triage row pointing at declaring
it.
- **`ptq.py`** — `--calib_seq` matters more than `--calib_size`, and
`--mse_calibrate` is a no-op under `--cast_mxfp4_to_nvfp4` (it tunes
weight quantizers only, and the cast overwrites `weight_scale`).
- **`.gitignore workspaces/`** — the skills create scratch directories
inside the repo; nothing excluded them.
### Usage
```bash
python "$SKILL_DIR/scripts/gate_verbosity.py" \
--baseline <baseline_eval_root> --candidate <candidate_eval_root> \
--glob 'eval_*' --threshold 0.05
```
Exit codes match the sibling gates: `0` pass, `1` the gate ran and
failed, `2` the gate could not
read its input (wrong root, `--glob` matched nothing, everything
excluded). Prints per-task tokens,
delta, `within_threshold`, `sample_count`, run counts,
`dropped_mismatched_runs`, any
`truncated_comparison`, `not_comparable`, `harvest_diagnostics`, and a
`max_abs_delta` summary.
### Testing
- 8 new unit tests in `test_gates.py`, one per real failure mode
(two-sided threshold, partial-run filtering, unequal sample counts,
short-output warning, one-sided tasks, empty input). Full suite: **36
passed**, no GPU or network.
- `gate_verbosity.py` validated end-to-end against a real day-0 run's
artifacts: reproduces the hand-computed result (`max_abs_delta =
0.0171`, pass) and correctly marks the two unequal-sample tasks
`not_comparable`.
- `pre-commit run --files <changed>` clean, including ruff, mypy,
bandit, markdownlint, and the `.claude/skills` symlink sync.
- Verified `max_sample_length` is a real `get_dataset_dataloader`
parameter with default 512, matching `--calib_seq`'s default, so
existing `ptq()` callers are unaffected.
### Before your PR is "*Ready for review*"
- Is this change backward compatible?: ✅ — `--calib_seq` defaults to the
pre-existing 512; `gate_ptq.py` reclassifies size growth from
`QUANT_COVERAGE_FAILURE` to `SIZE_NOT_REDUCED`, which is a more precise
class for an already-4-bit source and is covered by a new test asserting
a real coverage failure still outranks it.
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A — no new
dependencies, stdlib only.
- Did you write any new necessary tests?: ✅ — 8 new tests for the new
gate.
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A
- Did you get Claude approval on this PR?: ❌ — not yet run.
### Additional Information
The measured figures come from a completed day-0 NVFP4 release
qualification. Model-specific results were removed from the general
skills where the rule stands on its own; two references were kept
deliberately — a model card citation illustrating per-scenario sampling,
and a model-specific vLLM MoE kernel crash where the model name *is* the
evidence.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable calibration sequence-length limits for DeepSeek-V4
quantization.
* Added automated release checks for output verbosity, evaluation
comparability, serving readiness, and configuration parity.
* **Bug Fixes**
* Improved quantization size-ratio reporting by distinguishing
explainable growth from blocking failures.
* Clarified handling of deployment memory-access errors and infeasible
evaluations.
* **Documentation**
* Expanded guidance for calibration, remote execution, workspace
management, evaluation setup, deployment troubleshooting, and
statistical reliability.
* Added task-specific guidance for SciCode and GDPVal feasibility
checks.
* **Chores**
* Excluded workspace session directories from version control.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Chenjie LuoandClaude Opus 5
|
73d7784223 |
docs(eval-skill): add NVFP4 model-card sampling reference (#2224)
### What does this PR do? Type of change: documentation Adds `plugins/modelopt/skills/evaluation/references/nvfp4-modelcard-sampling.md` — the published `temperature` / `top_p` / max generation length for the **2026 NVFP4 checkpoints under [huggingface.co/nvidia](https://huggingface.co/nvidia/models) that disclose them** — and points `SKILL.md` Step 3 and `model-card-research.md` at it. **Why.** Config generation currently re-derives sampling params per run from one model card, which is slow and silently wrong in two ways: 1. **Quickstart boilerplate reads as an eval setting.** Most NVIDIA NVFP4 cards paste a TensorRT-LLM snippet containing `SamplingParams(temperature=0.8, top_p=0.95)`. That string is byte-identical across `Llama-3.1-8B`, `Llama-3.3-70B`, `Llama-4-Scout`, `Phi-4-reasoning-plus`, `Phi-4-multimodal`, `Qwen2.5-VL-7B` and six `Qwen3-*` repos. It is template text, not what the accuracy table was measured with — but it is the most prominent `temperature=` in the card. 2. **A generic fallback beats an available answer.** When a card is silent, Step 3 falls back to 65536/16384 even where a sibling in the same family publishes an exact value. **Scope — 25 rows.** All 69 NVFP4 checkpoints in the org were read. A row exists only for a 2026 **target** checkpoint whose card discloses usable settings, so absence means "read the card", not "not yet checked". Excluded by construction: pre-2026 releases, cards that publish nothing, and — via the curation regex `-NVFP4(-V\d+|-QAD)?$` — `-DSpark`/`-DFlash` speculative-decoding variants (verified against the target, so identical accuracy; they would only duplicate the base row), `-Eagle3` draft heads, and `-MLPerf-Inference-Closed-*` snapshots. | provenance | rows | meaning | | --- | --- | --- | | `eval` | 20 | card ties the values to its accuracy table — authoritative | | `rec` | 5 | recommended inference sampling, not tied to the eval | A `—` in one value column means that field specifically is unpublished — four rows give `temperature`/`top_p` but state no generation cap (`Mistral-Medium-3.5`, `Nemotron-3.5-Lightning`, `Nemotron-3-Super-120B`, `Nemotron-Labs-3-Elastic-30B`). Per-task exceptions are recorded where cards state them: GLM-5.2 GPQA Diamond `100000` vs `64000`; Qwen3.5-397B-V2 τ²-Bench Telecom `128000`; Qwen3.6 SciCode `temperature=0.6`; Kimi-K3 uncapped for Terminal-Bench. **Posture: the card is the source of truth; the table is a reference, not a constraint.** It is there to confirm a value you read, fill a gap when the card is silent, and catch a misreading — never to override what a card states. Listed and in agreement → proceed; listed and different → the card wins, re-read, surface the discrepancy. The `max_num_tokens` column records the card's *headline* cap, so Step 3's existing take-the-highest rule still governs when a card names more than one. Per-task `temperature`/`top_p` in the notes is precedent rather than mandate — engineers do tune sampling per benchmark — so the guidance is to follow the card and escalate only on a regime change (greedy vs sampled), not a nudge (`0.95` vs `1.0`). ### Usage Not an API change; the reference is consumed by the `evaluation` skill when generating a NEL config. ```yaml # nvidia/GLM-5.2-NVFP4 -> references/nvfp4-modelcard-sampling.md (provenance: eval) nemo_evaluator_config: config: params: max_new_tokens: 100000 # card: GPQA Diamond 100000, others 64000 -> take the highest temperature: 1.0 top_p: 0.95 ``` ### Testing **Coverage — enumeration cross-verified two ways.** Rather than paging `https://huggingface.co/nvidia/models?p=N` by hand, the candidate list came from the HF API (`author=nvidia&limit=1000` → 917 repos, one page), then verified against the website pagination: 918 unique model links across `p=0..31`, and the 68 NVFP4-named repos were **identical in both** (`api-only: []`, `web-only: []`). The committed curation regex was then re-run against that full set and confirmed to select every one of the 25 table rows plus 7 further 2026 target checkpoints whose cards disclose nothing — i.e. the documented filter reproduces the table. **Accuracy — every `eval` row diffed against its source sentence.** A script re-parsed the table and printed each row beside the matching card line. To be precise about what that covers: the diff was run when the table was larger, and all 28 machine-checkable *Benchmarked with* rows matched exactly (`Qwen3.6-35B-A3B` states its settings on the second line of a blockquote and was checked by hand). Every edit since has been a row **removal** — the scope cut to 2026 and the spec-decode removal — each verified by re-parsing the table before and after and confirming no surviving row's `temperature`/`top_p`/`max_num_tokens`/`provenance` changed. Extraction was restricted to *"Benchmarked with…"* / *"…were evaluated with…"* / *"We evaluate the model using…"* sentences, "Recommended Sampling" rows, and accuracy-table footnotes — never the quickstart snippets. **Two extraction traps found while building it**, both now in the file's refresh recipe as a second mandatory grep: some cards publish the cap **only** as a footnote beneath the accuracy table (`*Max OSL for evals can be as high as 64K`), and DeepSeek states sampling in its `## Input:` usage block rather than a *Benchmarked with* sentence. Neither is reachable from the obvious grep. **Hygiene.** `pre-commit run --files <the 3 files>` passes, including `markdownlint-cli2` and the `sync .claude/skills/ symlinks` hook; the reference resolves through both `.claude/skills/evaluation/references/` and `.agents/skills/evaluation/references/`. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A <!-- documentation-only; verification described under Testing --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A <!-- agent-skill documentation; no user-facing API or behavior change --> - Did you get Claude approval on this PR?: ✅ <!-- reviewed; findings addressed or answered below --> ### Additional Information Data collected 2026-08-20; the file is an explicitly dated snapshot and says to trust the card for anything newer. It ends with a refresh recipe (API enumeration → `-NVFP4(-V\d+|-QAD)?$` filter → `createdAt >= 2026-01-01` → card fetch → the grep phrasings that carry eval settings) so the table can be regenerated as new checkpoints ship — roughly four NVFP4 target checkpoints per month over 2026 so far. Review findings addressed: the mandatory-match framing was softened to reference-only, per-task sampling reframed as precedent with a human-escalation trigger, `mkdir -p cards` added to the recipe, the HF token no longer interpolated into a `curl` argument, and explicit uncapped generation (`max_new_tokens: null`) distinguished from an unpublished cap. The provenance question on the Nemotron-3.5-Lightning rows is answered in a review reply — the differing labels were correct (the base card says "Recommended Sampling", the sibling cards say "Benchmarked with"), and those sibling rows have since been removed as spec-decode duplicates. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Chenjie Luo <chenjiel@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Asha Anoosheh
|
94915a1e5d |
Fix typo for Tau2 eval config (#2210)
### What does this PR do? Type of change: Documentation <!-- Details about the change. --> ### Usage ```python # Add a code snippet demonstrating how to use this ``` ### Testing <!-- Mention how have you tested your change if applicable. --> ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: N/A - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A - Did you get Claude approval on this PR?: N/A ### Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated Tau2 evaluation guidance to consistently use “judge” and “user-simulator” terminology. * Clarified rate-limit guidance and the related YAML configuration terminology for telecom evaluations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> |
||
|
Chenjie LuoandClaude Opus 5
|
d32c2c2a56 |
docs(eval-skill): add MRCR (NeMo Gym) benchmark (#2192)
### What does this PR do?
Type of change: documentation (agent skill)
Adds **MRCR** — OpenAI's Multi-Round Co-reference Resolution, a
long-context
retrieval benchmark — to the `evaluation` skill as a standalone NeMo Gym
task,
derived from the reviewed `nemotron_nano_v35_nvfp4_mrcr_gym` golden;
regroups the
gym tasks/examples under one `gym/` dir; and fixes several latent bugs
in the
shared gym command block that were found by running the benchmark
end-to-end.
MRCR tasks are long multi-turn conversations containing N near-identical
"needle"
responses; the model must reproduce the Nth verbatim behind a random
prefix.
Grading is deterministic (`SequenceMatcher.ratio()`, gated on the
prefix). Unlike
GDPVal it uses the `simple_agent`: no SIF, no judge, no Tavily —
`HF_TOKEN` is the
only secret, and the cost is context length (up to 1M tokens), not agent
turns.
**It is not an AA benchmark** and is never generated for an "AA"
request.
### Layout
| File | |
| --- | --- |
| `recipes/tasks/gym/mrcr.md` | new recipe — variants, 1M serving
envelope, canary, score extraction |
| `recipes/examples/gym/example_mrcr.yaml` | new self-contained SLURM +
vLLM config |
| `SKILL.md` | MRCR branch + gym index table |
| `references/quantization-benchmarks.md` | table row + comparability
notes |
`examples/gym_{gdpval,mrcr}/` → `examples/gym/example_<task>.yaml` and
`tasks/aa_gym/gdpval.md` → `tasks/gym/gdpval.md`, all tracked as git
renames.
`aa_gym` encoded "in the AA suite" in the *path*; since GDPVal is AA and
MRCR is
not, membership is now stated explicitly in both recipe headers and an
"In AA
suite?" column in the SKILL.md index. `gym/` groups by harness, not
suite.
### Bugs fixed (found by running it, not by reading it)
The first three are in the **shared** gym command block, so they also
affect
`example_gdpval.yaml` — GDPVal was broken independently of MRCR.
1. **Invalid OmegaConf interpolation.** A literal `${...}` inside a
comment in the
gym `command:` block is parsed as an interpolation and rejected, so
*every*
`nel run --dry-run` of either gym template died with
`hydra.errors.ConfigCompositionException`.
2. **Hardcoded `ray==2.49.2`** injected into each sub-server's
requirements. Against
an image carrying `ray[default]==2.55.1` this makes `uv` unsatisfiable
and the
gym resources server exits at startup. Now derived from the image at
runtime.
3. **`--max-num-seqs` sized against the wrong topology.** The template
shipped DP1
→ 4 replicas at 64 concurrent while citing a golden that is TP2×DP2 → 8
replicas
at 32. Following it ran double the reference's per-replica load, which
on
1M-token prompts is what decides whether the KV cache fits.
4. **Score extraction pointed at an empty map.** `results.yml` →
`groups.nemo_gym.metrics` holds only `key_metrics/mean/*` telemetry; the
scores
are in `artifacts/evaluator_rollouts_aggregate_metrics.json` →
`[0].agent_metrics`.
Also documents that `pass@1/accuracy` is already 0-100 while
`mean/reward` is the
same number as a 0-1 fraction, and records `mean/prefix_matched ≈ 0.55`
as the
healthy calibration.
5. **The template shipped a container its own bootstrap rejects.** After
adding the
hard-fail on an unpinned Gym, `container:` still defaulted to public
`nemo-gym:26.05` — the image the docs say has a non-git `/opt/Gym`. Now
`???`,
so `--dry-run`'s mandatory-value check catches it instead of the job
dying at
startup.
### Review feedback
All items from @meenchen and CodeRabbit addressed or answered inline.
Two worth
surfacing here:
- **`process_reasoning_traces` vs `use_reasoning`** — verified against
`nemo_evaluator/adapters/adapter_config.py`: both exist, `use_reasoning`
is the
**deprecated** one and they are bidirectionally aliased. Documented
rather than
switched to the deprecated name.
- **`tiktoken` / `transformers` left unpinned** — deliberate. The n3
prepare path
uses `transformers.AutoTokenizer` to decide which samples exceed the
cap, so a
bump can shift dataset membership; but pinning would diverge from the
golden's
`pre_cmd` and therefore from the run that produced the reference number.
Risk is
now documented under "Deferred, know the risk" instead of being
implicit.
Topology, `gres` and TP/DP were aligned to the **existing** sibling
templates
rather than to a new convention: `gres` stays a comment (already the
convention in
`example_eval.yaml` and `example_gdpval.yaml`), TP is a concrete `1`
like both
siblings, and `num_nodes`/`num_instances`/`--max-num-seqs` are guidance
rather than
baked values. `--max-num-seqs` sizing follows **AA-LCR**, since MRCR is
the same
KV-bound problem at ~1M tokens vs LCR's ~120K: the formula gives a
ceiling, not a
target, because oversubscribing causes preemption and recomputing a
1M-token
prefill makes the run slower.
### Usage
```bash
cp plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml mrcr.yaml
# fill checkpoint_path / served_model_name / container / SLURM ??? values
export NEMO_EVALUATOR_TRUST_PRE_CMD=1 NEMO_EVALUATOR_TRUST_UNLISTED_TASKS=1
nel run --config mrcr.yaml --dry-run && nel run --config mrcr.yaml
```
### Testing
Docs/config-only; no library code touched. `pre-commit` clean on every
commit.
- Both gym YAMLs parse; asserted the folded-scalar rule (no `#` inside
`>-`), that
every string survives `OmegaConf.create` (bug 1), and that the variant
is
identical in `data_prep_params` and `collect_rollout_params`.
- After the rename: zero stale `aa_gym`/`gym_gdpval`/`gym_mrcr` refs and
every
`recipes/**` path referenced across both skill trees resolves on disk —
this
caught `env.example`, which an extension-filtered grep missed. The
`nemo_gym_gdpval_stirrup_agent` metric names contain the substring
`gym_gdpval`
and were deliberately left untouched.
- **Ran the benchmark end-to-end.** A full run on gcp-nrt (B200)
completed
2363/2363 rollouts and scored; that run produced bugs 2 and 4 and the
corrected
metric paths. A second attempt on aws-cmh was preempted and later
cancelled.
`--dry-run` now passes (it did not before bug 1 was fixed), and the bare
template
correctly fails validation on unresolved mandatory values.
- Regenerated the config from scratch with a fresh agent against the
fixed
templates as a regression check; its dry-run passed and its findings
drove the
topology/`gres`/TP-DP alignment above.
Model-specific scores are deliberately **not** recorded in the recipe —
the
reference there stays the golden's BF16 Nano 3.5 shape, since quoting a
different
model's number beside it invites exactly the false comparison the recipe
warns
about.
### Before your PR is "*Ready for review*"
- Backward compatible?: ✅ — additive plus a doc-tree rename; all
referencing files
updated and verified. The template fixes change only broken behaviour.
- Copied code / new PIP dependency: N/A.
- New tests: N/A — agent-skill documentation.
- Changelog: N/A — skill/docs change; consistent with prior skill-only
commits.
- Claude approval: ❌ not yet — will trigger `/claude review`.
### Additional Information
Upstream drift flagged to reviewers (**no change made to that repo**):
in
`nvidia-eval-factory-benchmarking`, `configs/benchmarks/mrcr/bench.yaml`
is still
old-style `ng_*` while `configs/models/nemotron_nano_v35/gym.yaml` moved
to the
`gym eval` CLI on 2026-07-29 — RULER migrated, MRCR did not, so the
layers no
longer compose. This template follows `ng_*`, which is what MRCR's own
bench.yaml
specifies and what the sign-off run executed.
NVIDIA-internal companion (container specifics kept out of this public
tree):
Model-Optimizer-Internal MR !116.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Keval MorabiaandClaude Opus 5
|
fbcdc16c2d |
Remove deprecations marked in 0.45 and 0.46 (#2182)
### What does this PR do?
Type of change: Backward breaking change (deprecation removal)
Ahead of the 0.47 code freeze, this removes every deprecation still
outstanding from the previous two releases (0.45 and 0.46). Two are
intentionally left in place: the **Python 3.10** drop and the
**transformers 4.x** drop
| Deprecation | Marked in | Replacement |
| --- | --- | --- |
| `--auto_quantize_bits` / `_method` / `_score_size` / `_cost_model` /
`_active_moe_expert_ratio` | 0.46 | AutoQuantize `--recipe` |
| `examples/llm_ptq` symlink + `examples/vlm_ptq/` forwarder | 0.46 |
`examples/hf_ptq` (`--vlm` for VLMs) |
| `QuantizationArgumentsWithConfig` alias | 0.45 |
`QuantizationArguments` |
| `QFORMAT_ALIASES` short names | 0.45 | canonical preset basenames |
| `layerwise` bool + flat `layerwise_checkpoint_dir` | 0.45 | nested
`layerwise: {enable, checkpoint_dir}` |
| in-trainer `quant_cfg` / `--quant_cfg` | 0.45 | `--recipe` |
#### Two things worth a closer look
**1. The `use_sequential` alias goes too.** It is the pre-#1251 alias on
`QuantizeAlgorithmConfig.layerwise` and only ever carried a bool. Once
the bool form is rejected it cannot accept a valid value, so keeping it
would only produce a differently-worded validation error. Note the
direction is breaking either way (`extra="forbid"`): a pre-0.45
`modelopt_state` carrying `use_sequential: True` or a top-level
`layerwise_checkpoint_dir` now fails validation instead of being
migrated.
**2. Removing in-trainer `--quant_cfg` required two new recipes.** The
`examples/gpt-oss` QAT flow ran on `--quant_cfg
MXFP4_MLP_WEIGHT_ONLY_CFG` and no `general/ptq/` recipe covered it. This
PR adds `general/ptq/mxfp4_mlp_weight_only` and
`general/ptq/nvfp4_mlp_weight_only`, verified to `model_dump` identical
to `mtq.MXFP4_MLP_WEIGHT_ONLY_CFG` / `mtq.NVFP4_MLP_WEIGHT_ONLY_CFG`,
and migrates the gpt-oss README, both SFT configs, `sft.py` and
`tests/examples/gpt-oss/test_gpt_oss_qat.py`. `examples/llm_qat` was
already recipe-only.
### Usage
```bash
# AutoQuantize: --auto_quantize_* flags -> an AutoQuantize recipe
scripts/huggingface_example.sh --model $HF_PATH \
--recipe general/auto_quantize/nvfp4_fp8_at_5p4bits --calib_batch_size 4
# --qformat / --quant_cfg: short name -> canonical preset basename
# int8_sq -> int8_smoothquant nvfp4_mse -> nvfp4_w4a4_weight_mse_fp8_sweep
# int8_wo -> int8_weight_only nvfp4_local_hessian -> nvfp4_w4a4_weight_local_hessian
# w4a8_awq -> w4a8_awq_beta fp8_pb_wo -> fp8_2d_blockwise_weight_only
# nvfp4_awq -> nvfp4_awq_lite fp8_pc_pt -> fp8_per_channel_per_token
scripts/huggingface_example.sh --model $HF_PATH --quant int8_smoothquant
# VLM PTQ: examples/vlm_ptq -> examples/hf_ptq with --vlm
scripts/huggingface_example.sh --model $HF_PATH --quant fp8 --vlm
# gpt-oss QAT: --quant_cfg <CFG name> -> --recipe <recipe path>
accelerate launch --config_file configs/zero3.yaml sft.py \
--config configs/sft_full.yaml --model_name_or_path openai/gpt-oss-20b \
--recipe general/ptq/mxfp4_mlp_weight_only --output_dir gpt-oss-20b-qat
```
```python
# Layerwise calibration: bool / flat key -> nested LayerwiseConfig
quant_cfg["algorithm"] = {"method": "gptq", "layerwise": {"enable": True, "checkpoint_dir": "/path"}}
```
### Testing
- `tests/unit/recipe` (229 passed),
`tests/unit/torch/quantization/test_config_validation.py` (79 passed),
`tests/examples/hf_ptq/test_hf_ptq_args.py` (23 passed).
- Verified the two new recipes `model_dump` identical to the `mtq.*_CFG`
constants they replace.
- `ruff check modelopt/ examples/ tests/` clean; `ruff format --check`
clean on all changed Python files.
- GPU suites
(`tests/gpu/torch/export/test_unified_hf_export_and_check_safetensors.py`,
`test_accelerate_gpu.py`, `test_gptq.py`) had their preset / layerwise
literals updated but were not run locally — relying on CI.
- `examples/llm_qat/ARGUMENTS.md` is hand-edited to match what the
`generate-arguments-md` hook emits; the generator could not run locally
(missing `transformers` package metadata in this environment).
### Before your PR is "*Ready for review*"
- Is this change backward compatible?: ❌ — that is the point of the PR:
it removes shims deprecated in 0.45/0.46. Callers must move to the
replacements in the table above. Additionally, a pre-0.45
`modelopt_state` carrying `use_sequential` or a top-level
`layerwise_checkpoint_dir` will now fail config validation.
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: ✅ — existing tests migrated to
the surviving APIs;
`TestLayerwiseNestedConfig::test_legacy_forms_rejected` pins that the
bool form, the `use_sequential` alias and the flat checkpoint-dir key
are all rejected. Tests covering the removed shims were deleted.
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
✅
- Did you get Claude approval on this PR?: ❌
### Additional Information
Follow-up: the transformers 4.x drop deprecated in 0.46 is still
outstanding and will need its own PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## New Features
- Added MXFP4 and NVFP4 weight-only quantization recipes for MLP and MoE
layers.
- Added shared layer exclusions for more accurate effective-bits
calculations.
## Improvements
- Updated PTQ, QAT, GPT-OSS, deployment, and quantization-format
examples with current recipe names and configuration formats.
- Standardized layerwise settings under nested configuration fields.
## Breaking Changes
- Removed deprecated AutoQuantize options, `quant_cfg` usage, format
aliases, legacy layerwise settings, and compatibility example paths.
- Recipe-based and nested configuration forms are now required.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Zhiyu
|
53ccec614f |
Pin GDPVal evaluator launcher to 0.2.6 (#2188)
### What does this PR do? Type of change: Bug fix Pins GDPVal evaluation commands to the currently validated `nemo-evaluator-launcher==0.2.6` through a dedicated `uvx` wrapper and updates the GDPVal skill, recipe, example, and reference documentation to use it. The reference also documents how to validate and adopt a future launcher release without mixing harness versions in baseline-versus-candidate comparisons. Older launchers generate a runtime re-export for `NEL_INVOCATION_ID` without first assigning the launcher-created invocation ID. The resulting Slurm script exits under `set -u` with `NEL_INVOCATION_ID: unbound variable` before the evaluation client starts. Launcher 0.2.6 writes the stable invocation ID into the script before the re-export, including for walltime-resume chains. ### Usage ```bash "$SKILL_DIR/scripts/nel-gdpval.sh" --version "$SKILL_DIR/scripts/nel-gdpval.sh" run --config gdpval.yaml --dry-run "$SKILL_DIR/scripts/nel-gdpval.sh" run --config gdpval.yaml ``` ### Testing - `bash -n plugins/modelopt/skills/evaluation/scripts/nel-gdpval.sh` - Verified the wrapper reports `nemo_evaluator_launcher: 0.2.6`. - Ran a real GDPVal dry-run and verified the generated script assigns `NEL_INVOCATION_ID="<invocation-id>"` before the runtime re-export. - Added a hermetic shell-wrapper test with a stub `uvx`; a conflicting ambient conflicting ambient version/spec values still invoke the validated 0.2.6 package. - `python -m pytest plugins/modelopt/skills/evaluation/ -o addopts="" -p no:cacheprovider -v` - Ran pre-commit on all six changed files. ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors). - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: ✅ — added a hermetic wrapper test that stubs `uvx` and verifies the exact package spec despite a conflicting environment value. - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A - Did you get Claude approval on this PR?: N/A ### Additional Information The fix preserves a stable GDPVal cache namespace across Slurm walltime resumes; using `SLURM_JOB_ID` would not. The failing run reused an existing base-environment `nel` 0.2.4 because the pre-existing prerequisite checked only whether `nel` was installed. The skill already identified 0.2.6 as the default launcher for Steps 1–9 and GDPVal. Version 0.2.6 is also the latest public `nemo-evaluator-launcher` release, so there is no newer public launcher version to validate. A dedicated `uvx` wrapper keeps the fix narrow and deterministic: - `nel-next.sh` manages the separate `nemo-evaluator` 0.4.x package, CLI, and schema; generalizing it would couple incompatible workflows. - A skill-level lock/project adds state for a single exact package invocation, while `uvx --from ...==0.2.6` already provides an isolated cached environment. - A version preflight would only reject the stale shared environment or require mutating it; the wrapper guarantees the known-good launcher on every GDPVal invocation without changing the existing `nel`. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added a GDPVal launcher wrapper that consistently uses the validated launcher version and forwards commands. - Added help output and checks for required runtime tooling. - **Documentation** - Clarified GDPVal launch, dry-run, reproducibility, resumed-run, and invocation ID guidance. - Documented version requirements and known sample limitations. - Updated evaluation examples and task instructions with the new launch procedure. - **Tests** - Added coverage verifying the pinned launcher version and argument forwarding. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com> |
||
|
ZhiyuandClaude Opus 5
|
6261f854aa |
docs: rebuild the unified HF deployment support matrix from the deploy test suite (NVBug 6550792) (#2087)
### What does this PR do? Type of change: documentation Fixes [NVBug 6550792](https://nvbugspro.nvidia.com/bug/6550792) / OMNIML-5693. The **Unified HF Checkpoint Deployment Model Support Matrix** listed 9 model families and **no VLMs**, while `tests/examples/hf_ptq/test_deploy.py` declares deployment cases for ~80 checkpoints across TRT-LLM, vLLM, and SGLang — including `Qwen2.5-VL`, `Qwen3-VL-235B`, and `Nemotron-3-Nano-Omni`. QA (the filer) could not use the doc to scope testing, and users could not tell what is actually covered. Filing also surfaced that the matrix lived in **three places that had drifted apart**: only the `.rst` listed Qwen3-VL, only the README listed Qwen3.5 MoE, and the skill reference had neither. #### Changes 1. **Rebuilt the matrix in `docs/source/deployment/3_unified_hf.rst`** from `test_deploy.py`, split into language models, vision-language/multimodal, speculative decoding drafters, and diffusion. 2. **Stated plainly what the matrix is and is not.** Review established that the original "CI-validated" framing claimed more than the suite substantiates, so a *What this matrix is based on* section now leads with two limits: - The suite is marked `release` and collects only under `--run-release`, which **no workflow passes** — these are declared cases, not PR-gated coverage. - Each case is a **load-and-generate smoke check on the text path**: no accuracy, no image/audio input, no diffusion output, no verification that speculative decoding engages. The legend follows from that: ✅ = declared in the suite, ⚠ = expected to work but not a suite entry (or an entry that does not exercise the feature the row names), `-` = not in the suite. Sections that would otherwise over-read carry their own qualifiers — VLM rows are labelled text-only smoke coverage, and Medusa and Wan 2.2 are ⚠ with the reason stated. 3. **Removed the two duplicate copies**, replacing them with links, so there is one table to maintain. 4. **Fixed stale prose**: the deployment tabs still claimed FP8-only support on vLLM v0.6.5 and a source build of SGLang main from Jan 2025, both contradicting the version table above them. The TRT-LLM floor moves to v1.2.0, qualified as the oldest version stated rather than the oldest that works. 5. **Dropped the Phi series** from the deployment matrix, following #2115 (NVBug 6563509) and confirmation that Phi-4 is being deprecated. ### Usage N/A — documentation only. ### Testing - `docutils` parse of the modified `.rst`: no warnings or errors from the new content; all 5 tables parse with every cell in the correct column. - Cell contents cross-checked against `test_deploy.py` by AST-parsing the `ModelDeployerList(...)` calls rather than by eye; the scope caveats were each verified against `tests/_test_utils/deploy_utils.py`. - `pre-commit run --files …` passes; `build-docs` green. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: N/A - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A — documentation only - Did you get Claude approval on this PR?: ❌ — not yet run ### Additional Information **Two known follow-ups, neither in scope here:** 1. **Nothing enforces that the doc matrix tracks `test_deploy.py`.** Consolidating to one copy removes the three-way drift but not the doc-vs-test drift; a generator plus a CI check would close it. 2. **The release deployment suite does not run in CI.** Wiring it into per-backend release CI is what would let ✅ mean "verified to pass" rather than "declared". That needs GPU capacity across three backends and should be tracked on its own. **For the filer (@Kenny Kang):** the ✅ cells are the scope the release deploy suite declares, and `test_deploy.py` carries the checkpoint, TP size, and minimum SM version per entry — but please read the legend first, since those cases are not currently executed by CI. --------- Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Chad Voegele
|
d3fe8117ff |
Add ModelOpt agent plugin marketplace (#2025)
### What does this PR do? Type of change: new feature Packages the existing ModelOpt agent skills as installable Codex and Claude plugins: - Adds a repo-scoped Codex marketplace and Claude-compatible marketplace. - Adds the canonical `plugins/modelopt/` plugin tree and manifests. - Moves the skill tree into the plugin and keeps `.agents/skills` as a compatibility symlink. - Adds a minimal `common` placeholder skill required by Codex validation. - Documents installation from this repository. ### Usage ```bash codex plugin marketplace add NVIDIA/Model-Optimizer ``` Then open `/plugins`, select the `modelopt` marketplace, and install `modelopt`. For Claude Code: ```bash claude plugin marketplace add https://github.com/NVIDIA/Model-Optimizer.git claude plugin install modelopt@modelopt ``` ### Testing - Codex plugin validator - `claude plugin validate . --strict` - `claude plugin validate plugins/modelopt --strict` 1. Install the marketplace plugin with Codex and Claude from an unrelated temporary workspace. 2. Exercise packaged evaluation helpers, a day-0 gate, and the shared remote helper from that workspace. 3. Run `uv run --frozen --extra dev python -m pytest -q plugins/modelopt/skills/day0-release/tests/test_gates.py plugins/modelopt/skills/benchmark-model-kernels/tests`. 4. Run pre-commit hooks for all changed files. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: ✅ — added a plugin-path validator; existing focused skill tests and installed-plugin smoke tests pass. - Did you update Changelog?: N/A — agent tooling and distribution only. - Did you get Claude approval on this PR?: N/A ### Additional Information Skills remain available through `.agents/skills`; bundled helpers are packaged under the plugin and resolved from `$SKILL_DIR` so installed workflows do not depend on the current workspace. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added installable ModelOpt plugins for Claude Code and Codex. * Added skills for PTQ, deployment, evaluation, monitoring, debugging, benchmarking, MLflow access, EAGLE3 workflows, and release management. * Added deployment helpers, evaluation recipes, checkpoint validation, and release-gating tools. * **Documentation** * Expanded setup, credential, SLURM, benchmarking, deployment, evaluation, troubleshooting, and workspace guidance. * Added installation instructions and updated agent-skill discovery guidance. * **Maintenance** * Updated skill references and compatibility links for reliable use across supported plugin environments. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Chad Voegele <cvoegele@nvidia.com> |
||
|
Keval Morabia
|
dc1a9d1b2d | nvidia-modelopt 0.17.0 examples release | ||
|
Chenjie Luo
|
06a155388e | nvidia-modelopt 0.11 examples release |