mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-02 03:14:52 +08:00
Keval Morabia
51cc5dbade
### What does this PR do? Type of change: Bug fix (CI) + test coverage **Fixes `onnx (torch_onnx)` and `onnx (diffusers)`**, which have failed on every branch since `torch 2.14.0` was published to PyPI today (2026-09-02 13:42 UTC), and **adds torch 2.14 to the unit test matrix as the new default** so the next torch release is caught there rather than in an example job. ### Root cause Every test in those two jobs failed with: ``` RuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed ptrDesc->finalize() cudnn_status: CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED ``` `nvcr.io/nvidia/tensorrt:26.05-py3` ships cuDNN **9.22** and has no preinstalled torch, so pip resolved the newest one — and torch 2.14 pins `nvidia-cudnn-cu13==9.24.0.43`. Loading 9.24 sublibraries against the image's 9.22 `libcudnn.so.9` is exactly what that status reports. | | last good run (08:55) | first failing run (13:34) | |---|---|---| | `torch` | 2.13.0 | **2.14.0** | | `nvidia-cudnn-cu13` | 9.20.0.48 | **9.24.0.43** | | image cuDNN | 9.22.0.52 | 9.22.0.52 | ### Why only these two jobs - The **nemo** and **pytorch** images have a preinstalled torch that already satisfies `torch>=2.8`, so pip never resolves a new one — confirmed from the megatron job log, where torch does not appear in `Successfully installed`. - **`tensorrt:26.05-py3` has no preinstalled torch**, so pip takes the newest from PyPI. - **`onnx (torch_trt)`** shares that image but passes throughout, because `torch-tensorrt<2.13` already holds torch below 2.14. ### The changes 1. **Constrain torch only where the incompatibility is.** `PIP_CONSTRAINT=torch<2.14` in the example runner, applied when the job's image is a `tensorrt` one. It also covers the `examples/*/requirements.txt` loop in the same shell, which matters because `nemo_automodel` pulls torch in too. Not pinned in `pyproject.toml`: torch 2.14 is fine anywhere its own bundled cuDNN is the one loaded, so that would constrain users to work around one pinned image. 2. **Test torch 2.14.** `torch_214` added to `TORCH_VERSIONS` (`torchvision~=0.29.0`) and promoted to the unit-test default across the supported Python versions, with 2.13 demoted to the back-compat row. `release.yml`'s basic unit test moves to the same default (it was still on 2.12). Nothing exercised 2.14 before — which is why a torch release reached us through an example job instead of a unit test. ### Testing - `actionlint` and YAML/TOML parse clean; pre-commit clean. - Verified by this PR's own jobs: `onnx (torch_onnx)` and `onnx (diffusers)` reproduce the failure on `main` right now, and the new `unit-3.12(torch_214, tf_latest)` job is the first run of ModelOpt against torch 2.14. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ — CI-only; no source or package metadata change - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A — no new dependency - Did you write any new necessary tests?: ✅ — torch 2.14 added to the unit test matrix - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A — internal CI, not user-facing - Did you get Claude approval on this PR?: ❌ — not yet requested --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
26 lines
737 B
YAML
26 lines
737 B
YAML
name: Wheel Release
|
|
|
|
on:
|
|
release:
|
|
types: [published]
|
|
|
|
permissions:
|
|
contents: write # Required to upload assets to the release
|
|
|
|
jobs:
|
|
build-upload:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
- uses: ./.github/actions/ubuntu-setup
|
|
- name: Run basic unit tests
|
|
run: pip install nox uv && nox -s "unit-3.12(torch_214, tf_latest)"
|
|
- name: Build Wheel
|
|
run: |
|
|
nox -s build_wheel
|
|
echo "WHEEL_PATH=$(find dist -name "*.whl" | head -n 1)" >> $GITHUB_ENV
|
|
- name: Upload GitHub Release Artifact
|
|
env:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
run: gh release upload ${{ github.event.release.tag_name }} ${{ env.WHEEL_PATH }}
|