Files
Model-Optimizer/tox.ini
T
Chenhan D. YuandClaude Opus 4.6 3131195241 add: DFlash block diffusion speculative decoding (#1211)
DFlash (Block Diffusion for Flash Speculative Decoding) predicts an
entire block of tokens in a single forward pass using masked parallel
prediction with KV injection from the target model's hidden states.

Key features:
- Feature fusion (multi-layer hidden states -> FC + RMSNorm)
- KV injection (fused features as K/V in every draft layer with QK-norm)
- Random anchor sampling with bidirectional intra-block attention
- Logit distillation with exponential loss decay (gamma weighting)
- Multi-node DDP training with checkpoint resume
- Export to z-lab compatible HF format
- Online validation (context-dependent ground truth)

Training recipe:
modelopt_recipes/general/speculative_decoding/dflash.yaml
Results: examples/speculative_decoding/doc/dflash_results.md

### ModelOpt Eval (online validation, osl=512)

| Dataset | z-lab | ModelOpt (306K) | Diff |
|---------|-------|-----------------|------|
| gsm8k | 4.10 | **5.19** | **+1.09** |
| MT-Bench | 3.58 | **4.36** | **+0.78** |

### z-lab Official Eval (dflash.benchmark, osl=512)

| Dataset | z-lab | ModelOpt (306K) | Diff |
|---------|-------|-----------------|------|
| gsm8k | **5.00** | 4.08 | -0.92 |
| MT-Bench | **3.28** | 2.99 | -0.29 |

> z-lab model trained with block_size=16. ModelOpt trained with
block_size=8.

## Evaluation Method Impact (gsm8k)

| Eval Method | z-lab checkpoint | ModelOpt (306K) |
|-------------|-----------------|-----------------|
| Fixed GT (ModelOpt eval) | 2.95 | 4.23 |
| Online GT (ModelOpt eval) | 4.10 | **5.19** |
| z-lab official eval | **5.00** | 4.08 |

### What does this PR do?

Type of change: ? <!-- Use one of the following: Bug fix, new feature,
new example, new tests, documentation. -->

<!-- Details about the change. -->

### Usage

```python
# Add a code snippet demonstrating how to use this
```

### Testing
<!-- Mention how have you tested your change if applicable. -->

### Before your PR is "*Ready for review*"

Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).

Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).

- Is this change backward compatible?: ✅ / ❌ / N/A <!--- If ❌, explain
why. -->
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ / ❌ / N/A
<!--- Mandatory -->
- Did you write any new necessary tests?: ✅ / ❌ / N/A <!--- Mandatory
for new features or examples. -->
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
✅ / ❌ / N/A <!--- Only for new features, API changes, critical bug fixes
or backward incompatible changes. -->

### Additional Information
<!-- E.g. related issue. -->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added DFlash speculative decoding mode with parallel block prediction
support.
* Included training launchers and MT-Bench evaluation scripts for DFlash
models.
* Added online acceptance rate validation for improved inference
verification.

* **Documentation**
* DFlash quick start guide with configuration parameters and training
examples.
  * Performance results and benchmarks for DFlash-trained models.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Chenhan Yu <chenhany@nvidia.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 15:58:39 -07:00

143 lines
4.3 KiB
INI

[tox]
envlist=
pre-commit-all
py312-torch210-tf_latest-unit
cuda13-gpu
cuda13-gpu-regression
cuda13-gpu-megatron
skipsdist = True
toxworkdir = /tmp/{env:USER}-modelopt-tox
passenv =
SETUPTOOLS_SCM_PRETEND_VERSION
############################
# CPU Unit test environments
############################
[testenv:{py310,py311,py312,py313}-torch{28,29,210,211}-tf_{min,latest}-unit]
deps =
# torch version auto-selected based on torchvision version
torch28: torchvision~=0.23.0
torch29: torchvision~=0.24.0
torch210: torchvision~=0.25.0
torch211: torchvision~=0.26.0
-e .[all,dev-test]
# Should match pyproject.toml
tf_min: transformers~=4.56.0
commands =
python -m pytest tests/unit {env:COV_ARGS:}
#####################################################################
# Environment to run unit tests with subset of dependencies installed
#####################################################################
[testenv:{py310,py311,py312,py313}-partial-unit-{onnx,torch,torch_deploy}]
allowlist_externals =
bash, rm
deps =
# Make sure torch 2.10 is used
torchvision~=0.26.0
# ONNX unit tests heavily rely on torch / torchvision
onnx: .[onnx,dev-test]
onnx: torchvision
# Install megatron-core to test torch-only install can still import plugins
torch: megatron-core
torch: .[dev-test]
torch_deploy: .[onnx,torch,dev-test]
commands =
onnx: python -m pytest tests/unit/onnx
torch: python -m pytest tests/unit/torch --ignore tests/unit/torch/deploy
torch_deploy: python -m pytest tests/unit/torch/deploy
###########################################################
# GPU test environments (Should be used with --current-env)
###########################################################
[testenv:cuda13-gpu]
commands_pre =
# Install deps here so that it gets installed even in --current-env
pip install --no-build-isolation git+https://github.com/Dao-AILab/fast-hadamard-transform.git
pip install -e .[all,dev-test]
# Install cupy-cuda13x for INT4 ONNX quantization (default is cupy-cuda12x)
pip uninstall -y cupy-cuda12x
pip install cupy-cuda13x
commands =
python -m pytest tests/gpu {env:COV_ARGS:}
[testenv:cuda13-gpu-regression]
commands_pre =
pip install -e .[hf,dev-test]
commands =
python -m pytest tests/gpu_regression {env:COV_ARGS:}
[testenv:cuda13-gpu-megatron]
commands_pre =
# Install deps here so that it gets installed even in --current-env
pip install -U megatron-core
pip install --no-build-isolation git+https://github.com/state-spaces/mamba.git
pip install --no-build-isolation git+https://github.com/Dao-AILab/causal-conv1d.git
pip install -e .[hf,dev-test]
commands =
python -m pytest tests/gpu_megatron {env:COV_ARGS:}
[testenv:cuda13-gpu-trtllm]
# Expected to be run in TRT-LLM container
commands_pre =
# Install deps here so that it gets installed even in --current-env
pip install -e .[hf,dev-test]
commands =
python -m pytest tests/gpu_trtllm {env:COV_ARGS:}
#############################################
# Code quality checks on all files or on diff
#############################################
[testenv:{pre-commit}-{all,diff}]
deps =
-e .[all,dev-lint]
commands =
all: pre-commit run --all-files --show-diff-on-failure {posargs}
diff: pre-commit run --from-ref origin/main --to-ref HEAD {posargs}
#########################
# Run documentation build
#########################
[testenv:{build,debug}-docs]
allowlist_externals =
rm
deps =
-e .[all,dev-docs]
changedir = docs
commands_pre =
rm -rf build
rm -rf source/reference/generated
commands =
sphinx-build source build/html --fail-on-warning --show-traceback --keep-going
debug: sphinx-autobuild source build/html --host 0.0.0.0
#################
# Run wheel build
#################
[testenv:build-wheel]
allowlist_externals =
bash, cd, rm
deps =
twine
commands =
# Clean build directory to avoid any stale files getting into the wheel
rm -rf build
# Build and check wheel
pip wheel --no-deps --wheel-dir=dist .
twine check dist/*
# Install and test the wheel
bash -c "find dist -name 'nvidia_modelopt-*.whl' | xargs pip install -f dist"
bash -c "cd dist; python -c 'import modelopt; print(modelopt.__version__);'"