mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-26 06:54:04 +08:00
pull-request/2240
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chenjie LuoandClaude Opus 5
|
d32c2c2a56 |
docs(eval-skill): add MRCR (NeMo Gym) benchmark (#2192)
### What does this PR do?
Type of change: documentation (agent skill)
Adds **MRCR** — OpenAI's Multi-Round Co-reference Resolution, a
long-context
retrieval benchmark — to the `evaluation` skill as a standalone NeMo Gym
task,
derived from the reviewed `nemotron_nano_v35_nvfp4_mrcr_gym` golden;
regroups the
gym tasks/examples under one `gym/` dir; and fixes several latent bugs
in the
shared gym command block that were found by running the benchmark
end-to-end.
MRCR tasks are long multi-turn conversations containing N near-identical
"needle"
responses; the model must reproduce the Nth verbatim behind a random
prefix.
Grading is deterministic (`SequenceMatcher.ratio()`, gated on the
prefix). Unlike
GDPVal it uses the `simple_agent`: no SIF, no judge, no Tavily —
`HF_TOKEN` is the
only secret, and the cost is context length (up to 1M tokens), not agent
turns.
**It is not an AA benchmark** and is never generated for an "AA"
request.
### Layout
| File | |
| --- | --- |
| `recipes/tasks/gym/mrcr.md` | new recipe — variants, 1M serving
envelope, canary, score extraction |
| `recipes/examples/gym/example_mrcr.yaml` | new self-contained SLURM +
vLLM config |
| `SKILL.md` | MRCR branch + gym index table |
| `references/quantization-benchmarks.md` | table row + comparability
notes |
`examples/gym_{gdpval,mrcr}/` → `examples/gym/example_<task>.yaml` and
`tasks/aa_gym/gdpval.md` → `tasks/gym/gdpval.md`, all tracked as git
renames.
`aa_gym` encoded "in the AA suite" in the *path*; since GDPVal is AA and
MRCR is
not, membership is now stated explicitly in both recipe headers and an
"In AA
suite?" column in the SKILL.md index. `gym/` groups by harness, not
suite.
### Bugs fixed (found by running it, not by reading it)
The first three are in the **shared** gym command block, so they also
affect
`example_gdpval.yaml` — GDPVal was broken independently of MRCR.
1. **Invalid OmegaConf interpolation.** A literal `${...}` inside a
comment in the
gym `command:` block is parsed as an interpolation and rejected, so
*every*
`nel run --dry-run` of either gym template died with
`hydra.errors.ConfigCompositionException`.
2. **Hardcoded `ray==2.49.2`** injected into each sub-server's
requirements. Against
an image carrying `ray[default]==2.55.1` this makes `uv` unsatisfiable
and the
gym resources server exits at startup. Now derived from the image at
runtime.
3. **`--max-num-seqs` sized against the wrong topology.** The template
shipped DP1
→ 4 replicas at 64 concurrent while citing a golden that is TP2×DP2 → 8
replicas
at 32. Following it ran double the reference's per-replica load, which
on
1M-token prompts is what decides whether the KV cache fits.
4. **Score extraction pointed at an empty map.** `results.yml` →
`groups.nemo_gym.metrics` holds only `key_metrics/mean/*` telemetry; the
scores
are in `artifacts/evaluator_rollouts_aggregate_metrics.json` →
`[0].agent_metrics`.
Also documents that `pass@1/accuracy` is already 0-100 while
`mean/reward` is the
same number as a 0-1 fraction, and records `mean/prefix_matched ≈ 0.55`
as the
healthy calibration.
5. **The template shipped a container its own bootstrap rejects.** After
adding the
hard-fail on an unpinned Gym, `container:` still defaulted to public
`nemo-gym:26.05` — the image the docs say has a non-git `/opt/Gym`. Now
`???`,
so `--dry-run`'s mandatory-value check catches it instead of the job
dying at
startup.
### Review feedback
All items from @meenchen and CodeRabbit addressed or answered inline.
Two worth
surfacing here:
- **`process_reasoning_traces` vs `use_reasoning`** — verified against
`nemo_evaluator/adapters/adapter_config.py`: both exist, `use_reasoning`
is the
**deprecated** one and they are bidirectionally aliased. Documented
rather than
switched to the deprecated name.
- **`tiktoken` / `transformers` left unpinned** — deliberate. The n3
prepare path
uses `transformers.AutoTokenizer` to decide which samples exceed the
cap, so a
bump can shift dataset membership; but pinning would diverge from the
golden's
`pre_cmd` and therefore from the run that produced the reference number.
Risk is
now documented under "Deferred, know the risk" instead of being
implicit.
Topology, `gres` and TP/DP were aligned to the **existing** sibling
templates
rather than to a new convention: `gres` stays a comment (already the
convention in
`example_eval.yaml` and `example_gdpval.yaml`), TP is a concrete `1`
like both
siblings, and `num_nodes`/`num_instances`/`--max-num-seqs` are guidance
rather than
baked values. `--max-num-seqs` sizing follows **AA-LCR**, since MRCR is
the same
KV-bound problem at ~1M tokens vs LCR's ~120K: the formula gives a
ceiling, not a
target, because oversubscribing causes preemption and recomputing a
1M-token
prefill makes the run slower.
### Usage
```bash
cp plugins/modelopt/skills/evaluation/recipes/examples/gym/example_mrcr.yaml mrcr.yaml
# fill checkpoint_path / served_model_name / container / SLURM ??? values
export NEMO_EVALUATOR_TRUST_PRE_CMD=1 NEMO_EVALUATOR_TRUST_UNLISTED_TASKS=1
nel run --config mrcr.yaml --dry-run && nel run --config mrcr.yaml
```
### Testing
Docs/config-only; no library code touched. `pre-commit` clean on every
commit.
- Both gym YAMLs parse; asserted the folded-scalar rule (no `#` inside
`>-`), that
every string survives `OmegaConf.create` (bug 1), and that the variant
is
identical in `data_prep_params` and `collect_rollout_params`.
- After the rename: zero stale `aa_gym`/`gym_gdpval`/`gym_mrcr` refs and
every
`recipes/**` path referenced across both skill trees resolves on disk —
this
caught `env.example`, which an extension-filtered grep missed. The
`nemo_gym_gdpval_stirrup_agent` metric names contain the substring
`gym_gdpval`
and were deliberately left untouched.
- **Ran the benchmark end-to-end.** A full run on gcp-nrt (B200)
completed
2363/2363 rollouts and scored; that run produced bugs 2 and 4 and the
corrected
metric paths. A second attempt on aws-cmh was preempted and later
cancelled.
`--dry-run` now passes (it did not before bug 1 was fixed), and the bare
template
correctly fails validation on unresolved mandatory values.
- Regenerated the config from scratch with a fresh agent against the
fixed
templates as a regression check; its dry-run passed and its findings
drove the
topology/`gres`/TP-DP alignment above.
Model-specific scores are deliberately **not** recorded in the recipe —
the
reference there stays the golden's BF16 Nano 3.5 shape, since quoting a
different
model's number beside it invites exactly the false comparison the recipe
warns
about.
### Before your PR is "*Ready for review*"
- Backward compatible?: ✅ — additive plus a doc-tree rename; all
referencing files
updated and verified. The template fixes change only broken behaviour.
- Copied code / new PIP dependency: N/A.
- New tests: N/A — agent-skill documentation.
- Changelog: N/A — skill/docs change; consistent with prior skill-only
commits.
- Claude approval: ❌ not yet — will trigger `/claude review`.
### Additional Information
Upstream drift flagged to reviewers (**no change made to that repo**):
in
`nvidia-eval-factory-benchmarking`, `configs/benchmarks/mrcr/bench.yaml`
is still
old-style `ng_*` while `configs/models/nemotron_nano_v35/gym.yaml` moved
to the
`gym eval` CLI on 2026-07-29 — RULER migrated, MRCR did not, so the
layers no
longer compose. This template follows `ng_*`, which is what MRCR's own
bench.yaml
specifies and what the sign-off run executed.
NVIDIA-internal companion (container specifics kept out of this public
tree):
Model-Optimizer-Internal MR !116.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Zhiyu
|
53ccec614f |
Pin GDPVal evaluator launcher to 0.2.6 (#2188)
### What does this PR do? Type of change: Bug fix Pins GDPVal evaluation commands to the currently validated `nemo-evaluator-launcher==0.2.6` through a dedicated `uvx` wrapper and updates the GDPVal skill, recipe, example, and reference documentation to use it. The reference also documents how to validate and adopt a future launcher release without mixing harness versions in baseline-versus-candidate comparisons. Older launchers generate a runtime re-export for `NEL_INVOCATION_ID` without first assigning the launcher-created invocation ID. The resulting Slurm script exits under `set -u` with `NEL_INVOCATION_ID: unbound variable` before the evaluation client starts. Launcher 0.2.6 writes the stable invocation ID into the script before the re-export, including for walltime-resume chains. ### Usage ```bash "$SKILL_DIR/scripts/nel-gdpval.sh" --version "$SKILL_DIR/scripts/nel-gdpval.sh" run --config gdpval.yaml --dry-run "$SKILL_DIR/scripts/nel-gdpval.sh" run --config gdpval.yaml ``` ### Testing - `bash -n plugins/modelopt/skills/evaluation/scripts/nel-gdpval.sh` - Verified the wrapper reports `nemo_evaluator_launcher: 0.2.6`. - Ran a real GDPVal dry-run and verified the generated script assigns `NEL_INVOCATION_ID="<invocation-id>"` before the runtime re-export. - Added a hermetic shell-wrapper test with a stub `uvx`; a conflicting ambient conflicting ambient version/spec values still invoke the validated 0.2.6 package. - `python -m pytest plugins/modelopt/skills/evaluation/ -o addopts="" -p no:cacheprovider -v` - Ran pre-commit on all six changed files. ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors). - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: ✅ — added a hermetic wrapper test that stubs `uvx` and verifies the exact package spec despite a conflicting environment value. - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A - Did you get Claude approval on this PR?: N/A ### Additional Information The fix preserves a stable GDPVal cache namespace across Slurm walltime resumes; using `SLURM_JOB_ID` would not. The failing run reused an existing base-environment `nel` 0.2.4 because the pre-existing prerequisite checked only whether `nel` was installed. The skill already identified 0.2.6 as the default launcher for Steps 1–9 and GDPVal. Version 0.2.6 is also the latest public `nemo-evaluator-launcher` release, so there is no newer public launcher version to validate. A dedicated `uvx` wrapper keeps the fix narrow and deterministic: - `nel-next.sh` manages the separate `nemo-evaluator` 0.4.x package, CLI, and schema; generalizing it would couple incompatible workflows. - A skill-level lock/project adds state for a single exact package invocation, while `uvx --from ...==0.2.6` already provides an isolated cached environment. - A version preflight would only reject the stale shared environment or require mutating it; the wrapper guarantees the known-good launcher on every GDPVal invocation without changing the existing `nel`. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added a GDPVal launcher wrapper that consistently uses the validated launcher version and forwards commands. - Added help output and checks for required runtime tooling. - **Documentation** - Clarified GDPVal launch, dry-run, reproducibility, resumed-run, and invocation ID guidance. - Documented version requirements and known sample limitations. - Updated evaluation examples and task instructions with the new launch procedure. - **Tests** - Added coverage verifying the pinned launcher version and argument forwarding. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com> |
||
|
Chad Voegele
|
d3fe8117ff |
Add ModelOpt agent plugin marketplace (#2025)
### What does this PR do? Type of change: new feature Packages the existing ModelOpt agent skills as installable Codex and Claude plugins: - Adds a repo-scoped Codex marketplace and Claude-compatible marketplace. - Adds the canonical `plugins/modelopt/` plugin tree and manifests. - Moves the skill tree into the plugin and keeps `.agents/skills` as a compatibility symlink. - Adds a minimal `common` placeholder skill required by Codex validation. - Documents installation from this repository. ### Usage ```bash codex plugin marketplace add NVIDIA/Model-Optimizer ``` Then open `/plugins`, select the `modelopt` marketplace, and install `modelopt`. For Claude Code: ```bash claude plugin marketplace add https://github.com/NVIDIA/Model-Optimizer.git claude plugin install modelopt@modelopt ``` ### Testing - Codex plugin validator - `claude plugin validate . --strict` - `claude plugin validate plugins/modelopt --strict` 1. Install the marketplace plugin with Codex and Claude from an unrelated temporary workspace. 2. Exercise packaged evaluation helpers, a day-0 gate, and the shared remote helper from that workspace. 3. Run `uv run --frozen --extra dev python -m pytest -q plugins/modelopt/skills/day0-release/tests/test_gates.py plugins/modelopt/skills/benchmark-model-kernels/tests`. 4. Run pre-commit hooks for all changed files. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A - Did you write any new necessary tests?: ✅ — added a plugin-path validator; existing focused skill tests and installed-plugin smoke tests pass. - Did you update Changelog?: N/A — agent tooling and distribution only. - Did you get Claude approval on this PR?: N/A ### Additional Information Skills remain available through `.agents/skills`; bundled helpers are packaged under the plugin and resolved from `$SKILL_DIR` so installed workflows do not depend on the current workspace. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added installable ModelOpt plugins for Claude Code and Codex. * Added skills for PTQ, deployment, evaluation, monitoring, debugging, benchmarking, MLflow access, EAGLE3 workflows, and release management. * Added deployment helpers, evaluation recipes, checkpoint validation, and release-gating tools. * **Documentation** * Expanded setup, credential, SLURM, benchmarking, deployment, evaluation, troubleshooting, and workspace guidance. * Added installation instructions and updated agent-skill discovery guidance. * **Maintenance** * Updated skill references and compatibility links for reliable use across supported plugin environments. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Chad Voegele <cvoegele@nvidia.com> |