mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-02 03:14:52 +08:00
Keval Morabia
51cc5dbade
### What does this PR do? Type of change: Bug fix (CI) + test coverage **Fixes `onnx (torch_onnx)` and `onnx (diffusers)`**, which have failed on every branch since `torch 2.14.0` was published to PyPI today (2026-09-02 13:42 UTC), and **adds torch 2.14 to the unit test matrix as the new default** so the next torch release is caught there rather than in an example job. ### Root cause Every test in those two jobs failed with: ``` RuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failed ptrDesc->finalize() cudnn_status: CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED ``` `nvcr.io/nvidia/tensorrt:26.05-py3` ships cuDNN **9.22** and has no preinstalled torch, so pip resolved the newest one — and torch 2.14 pins `nvidia-cudnn-cu13==9.24.0.43`. Loading 9.24 sublibraries against the image's 9.22 `libcudnn.so.9` is exactly what that status reports. | | last good run (08:55) | first failing run (13:34) | |---|---|---| | `torch` | 2.13.0 | **2.14.0** | | `nvidia-cudnn-cu13` | 9.20.0.48 | **9.24.0.43** | | image cuDNN | 9.22.0.52 | 9.22.0.52 | ### Why only these two jobs - The **nemo** and **pytorch** images have a preinstalled torch that already satisfies `torch>=2.8`, so pip never resolves a new one — confirmed from the megatron job log, where torch does not appear in `Successfully installed`. - **`tensorrt:26.05-py3` has no preinstalled torch**, so pip takes the newest from PyPI. - **`onnx (torch_trt)`** shares that image but passes throughout, because `torch-tensorrt<2.13` already holds torch below 2.14. ### The changes 1. **Constrain torch only where the incompatibility is.** `PIP_CONSTRAINT=torch<2.14` in the example runner, applied when the job's image is a `tensorrt` one. It also covers the `examples/*/requirements.txt` loop in the same shell, which matters because `nemo_automodel` pulls torch in too. Not pinned in `pyproject.toml`: torch 2.14 is fine anywhere its own bundled cuDNN is the one loaded, so that would constrain users to work around one pinned image. 2. **Test torch 2.14.** `torch_214` added to `TORCH_VERSIONS` (`torchvision~=0.29.0`) and promoted to the unit-test default across the supported Python versions, with 2.13 demoted to the back-compat row. `release.yml`'s basic unit test moves to the same default (it was still on 2.12). Nothing exercised 2.14 before — which is why a torch release reached us through an example job instead of a unit test. ### Testing - `actionlint` and YAML/TOML parse clean; pre-commit clean. - Verified by this PR's own jobs: `onnx (torch_onnx)` and `onnx (diffusers)` reproduce the failure on `main` right now, and the new `unit-3.12(torch_214, tf_latest)` job is the first run of ModelOpt against torch 2.14. ### Before your PR is "*Ready for review*" - Is this change backward compatible?: ✅ — CI-only; no source or package metadata change - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A — no new dependency - Did you write any new necessary tests?: ✅ — torch 2.14 added to the unit test matrix - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A — internal CI, not user-facing - Did you get Claude approval on this PR?: ❌ — not yet requested --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
39 lines
1.5 KiB
YAML
39 lines
1.5 KiB
YAML
name: Code Quality
|
|
|
|
on:
|
|
pull_request:
|
|
branches: [main, release/*, feature/*]
|
|
schedule:
|
|
- cron: "0 0 * * *" # Nightly
|
|
workflow_dispatch:
|
|
# On-demand
|
|
|
|
|
|
concurrency:
|
|
# Cancel previous runs if new commit is pushed to the same PR
|
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
code-quality:
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
- uses: ./.github/actions/ubuntu-setup
|
|
- name: Secret Scanning
|
|
uses: trufflesecurity/trufflehog@v3.90.5
|
|
with:
|
|
# Exclude lob: its `(live|test)_[a-zA-Z0-9_]{35}` pattern matches any pytest function
|
|
# whose name happens to be exactly 35 characters after `test_`, and it reports them as
|
|
# verified. Only nightly sees them, since it scans all history while PRs scan their diff.
|
|
extra_args: --results=verified,unknown --exclude-detectors=lob
|
|
- name: Run code quality checks
|
|
run: |
|
|
# torch 2.14 breaks this env: torchvision::nms fails to register, which disables
|
|
# modelopt's transformers plugin and takes the ARGUMENTS.md hook down with it.
|
|
echo "torch<2.14" > /tmp/pip-constraints.txt
|
|
# Both: nox builds the session venv with uv, which reads UV_CONSTRAINT, not PIP_CONSTRAINT.
|
|
export PIP_CONSTRAINT=/tmp/pip-constraints.txt UV_CONSTRAINT=/tmp/pip-constraints.txt
|
|
pip install nox uv && nox -s pre_commit_all
|