mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-02 03:14:52 +08:00
DFlash (Block Diffusion for Flash Speculative Decoding) predicts an entire block of tokens in a single forward pass using masked parallel prediction with KV injection from the target model's hidden states. Key features: - Feature fusion (multi-layer hidden states -> FC + RMSNorm) - KV injection (fused features as K/V in every draft layer with QK-norm) - Random anchor sampling with bidirectional intra-block attention - Logit distillation with exponential loss decay (gamma weighting) - Multi-node DDP training with checkpoint resume - Export to z-lab compatible HF format - Online validation (context-dependent ground truth) Training recipe: modelopt_recipes/general/speculative_decoding/dflash.yaml Results: examples/speculative_decoding/doc/dflash_results.md ### ModelOpt Eval (online validation, osl=512) | Dataset | z-lab | ModelOpt (306K) | Diff | |---------|-------|-----------------|------| | gsm8k | 4.10 | **5.19** | **+1.09** | | MT-Bench | 3.58 | **4.36** | **+0.78** | ### z-lab Official Eval (dflash.benchmark, osl=512) | Dataset | z-lab | ModelOpt (306K) | Diff | |---------|-------|-----------------|------| | gsm8k | **5.00** | 4.08 | -0.92 | | MT-Bench | **3.28** | 2.99 | -0.29 | > z-lab model trained with block_size=16. ModelOpt trained with block_size=8. ## Evaluation Method Impact (gsm8k) | Eval Method | z-lab checkpoint | ModelOpt (306K) | |-------------|-----------------|-----------------| | Fixed GT (ModelOpt eval) | 2.95 | 4.23 | | Online GT (ModelOpt eval) | 4.10 | **5.19** | | z-lab official eval | **5.00** | 4.08 | ### What does this PR do? Type of change: ? <!-- Use one of the following: Bug fix, new feature, new example, new tests, documentation. --> <!-- Details about the change. --> ### Usage ```python # Add a code snippet demonstrating how to use this ``` ### Testing <!-- Mention how have you tested your change if applicable. --> ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ / ❌ / N/A <!--- If ❌, explain why. --> - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ / ❌ / N/A <!--- Mandatory --> - Did you write any new necessary tests?: ✅ / ❌ / N/A <!--- Mandatory for new features or examples. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: ✅ / ❌ / N/A <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> ### Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added DFlash speculative decoding mode with parallel block prediction support. * Included training launchers and MT-Bench evaluation scripts for DFlash models. * Added online acceptance rate validation for improved inference verification. * **Documentation** * DFlash quick start guide with configuration parameters and training examples. * Performance results and benchmarks for DFlash-trained models. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Chenhan Yu <chenhany@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
143 lines
4.3 KiB
INI
143 lines
4.3 KiB
INI
[tox]
|
|
envlist=
|
|
pre-commit-all
|
|
py312-torch210-tf_latest-unit
|
|
cuda13-gpu
|
|
cuda13-gpu-regression
|
|
cuda13-gpu-megatron
|
|
skipsdist = True
|
|
toxworkdir = /tmp/{env:USER}-modelopt-tox
|
|
passenv =
|
|
SETUPTOOLS_SCM_PRETEND_VERSION
|
|
|
|
############################
|
|
# CPU Unit test environments
|
|
############################
|
|
[testenv:{py310,py311,py312,py313}-torch{28,29,210,211}-tf_{min,latest}-unit]
|
|
deps =
|
|
# torch version auto-selected based on torchvision version
|
|
torch28: torchvision~=0.23.0
|
|
torch29: torchvision~=0.24.0
|
|
torch210: torchvision~=0.25.0
|
|
torch211: torchvision~=0.26.0
|
|
|
|
-e .[all,dev-test]
|
|
|
|
# Should match pyproject.toml
|
|
tf_min: transformers~=4.56.0
|
|
commands =
|
|
python -m pytest tests/unit {env:COV_ARGS:}
|
|
|
|
|
|
#####################################################################
|
|
# Environment to run unit tests with subset of dependencies installed
|
|
#####################################################################
|
|
[testenv:{py310,py311,py312,py313}-partial-unit-{onnx,torch,torch_deploy}]
|
|
allowlist_externals =
|
|
bash, rm
|
|
deps =
|
|
# Make sure torch 2.10 is used
|
|
torchvision~=0.26.0
|
|
|
|
# ONNX unit tests heavily rely on torch / torchvision
|
|
onnx: .[onnx,dev-test]
|
|
onnx: torchvision
|
|
|
|
# Install megatron-core to test torch-only install can still import plugins
|
|
torch: megatron-core
|
|
torch: .[dev-test]
|
|
|
|
torch_deploy: .[onnx,torch,dev-test]
|
|
commands =
|
|
onnx: python -m pytest tests/unit/onnx
|
|
torch: python -m pytest tests/unit/torch --ignore tests/unit/torch/deploy
|
|
torch_deploy: python -m pytest tests/unit/torch/deploy
|
|
|
|
|
|
###########################################################
|
|
# GPU test environments (Should be used with --current-env)
|
|
###########################################################
|
|
[testenv:cuda13-gpu]
|
|
commands_pre =
|
|
# Install deps here so that it gets installed even in --current-env
|
|
pip install --no-build-isolation git+https://github.com/Dao-AILab/fast-hadamard-transform.git
|
|
pip install -e .[all,dev-test]
|
|
|
|
# Install cupy-cuda13x for INT4 ONNX quantization (default is cupy-cuda12x)
|
|
pip uninstall -y cupy-cuda12x
|
|
pip install cupy-cuda13x
|
|
commands =
|
|
python -m pytest tests/gpu {env:COV_ARGS:}
|
|
|
|
[testenv:cuda13-gpu-regression]
|
|
commands_pre =
|
|
pip install -e .[hf,dev-test]
|
|
commands =
|
|
python -m pytest tests/gpu_regression {env:COV_ARGS:}
|
|
|
|
[testenv:cuda13-gpu-megatron]
|
|
commands_pre =
|
|
# Install deps here so that it gets installed even in --current-env
|
|
pip install -U megatron-core
|
|
pip install --no-build-isolation git+https://github.com/state-spaces/mamba.git
|
|
pip install --no-build-isolation git+https://github.com/Dao-AILab/causal-conv1d.git
|
|
pip install -e .[hf,dev-test]
|
|
commands =
|
|
python -m pytest tests/gpu_megatron {env:COV_ARGS:}
|
|
|
|
[testenv:cuda13-gpu-trtllm]
|
|
# Expected to be run in TRT-LLM container
|
|
commands_pre =
|
|
# Install deps here so that it gets installed even in --current-env
|
|
pip install -e .[hf,dev-test]
|
|
commands =
|
|
python -m pytest tests/gpu_trtllm {env:COV_ARGS:}
|
|
|
|
#############################################
|
|
# Code quality checks on all files or on diff
|
|
#############################################
|
|
[testenv:{pre-commit}-{all,diff}]
|
|
deps =
|
|
-e .[all,dev-lint]
|
|
commands =
|
|
all: pre-commit run --all-files --show-diff-on-failure {posargs}
|
|
diff: pre-commit run --from-ref origin/main --to-ref HEAD {posargs}
|
|
|
|
|
|
#########################
|
|
# Run documentation build
|
|
#########################
|
|
[testenv:{build,debug}-docs]
|
|
allowlist_externals =
|
|
rm
|
|
deps =
|
|
-e .[all,dev-docs]
|
|
changedir = docs
|
|
commands_pre =
|
|
rm -rf build
|
|
rm -rf source/reference/generated
|
|
commands =
|
|
sphinx-build source build/html --fail-on-warning --show-traceback --keep-going
|
|
debug: sphinx-autobuild source build/html --host 0.0.0.0
|
|
|
|
|
|
#################
|
|
# Run wheel build
|
|
#################
|
|
[testenv:build-wheel]
|
|
allowlist_externals =
|
|
bash, cd, rm
|
|
deps =
|
|
twine
|
|
commands =
|
|
# Clean build directory to avoid any stale files getting into the wheel
|
|
rm -rf build
|
|
|
|
# Build and check wheel
|
|
pip wheel --no-deps --wheel-dir=dist .
|
|
twine check dist/*
|
|
|
|
# Install and test the wheel
|
|
bash -c "find dist -name 'nvidia_modelopt-*.whl' | xargs pip install -f dist"
|
|
bash -c "cd dist; python -c 'import modelopt; print(modelopt.__version__);'"
|