Commit Graph
1 Commits
Author SHA1 Message Date
ynankani-nv 4a70040ddd Ynankani/fvd benchmark (#1182)
### What does this PR do?

Type of change: ? New benchmark addition

<!-- Details about the change. -->
This PR adds the following to examples/windows/:
1) FVD (Fréchet Video Distance) evaluation tool — a standalone script
for computing FVD between two sets of videos using a pre-trained I3D
model (Kinetics-400, 1024-dim pooled features).
2) Directory reorganization — moved torch_onnx/ and qad_example/ under a
new diffusers/ folder for clearer grouping of diffusion model examples.
3) Minor fixes: Updated WikiText-2 dataset link in
perplexity_metrics/README.md (old wikitext → Salesforce/wikitext)
### Usage

```python
python compute_fvd.py \
  --ref-dir /path/to/reference/videos \
  --gen-dir /path/to/generated/videos \
  --output results.json \
  --device cuda
```

### Testing
Evaluated FVD tool on LTX-2.3 outputs comparing QAD vs PTQ checkpoints
(both NVFp4) against a BF16 baseline across 11 VBench dimensions.

### Before your PR is "*Ready for review*"

Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).

Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).

- Is this change backward compatible?: ✅ 
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A <!---
Mandatory -->
- Did you write any new necessary tests?: N/A <!--- Mandatory for new
features or examples. -->
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A <!--- Only for new features, API changes, critical bug fixes or
backward incompatible changes. -->

### Additional Information
<!-- E.g. related issue. -->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added FVD (Fréchet Video Distance) evaluation: command‑line tool,
I3D‑based feature extraction, PCA option, and quantized comparisons (PTQ
vs QAD) with per‑category and overall results showing QAD's lower
average FVD.

* **Documentation**
* Expanded benchmarks TOC with “Additional Metrics”, added FVD
evaluation guide, usage examples, troubleshooting, and updated
perplexity dataset reference.

* **Chores**
  * Added Python requirements for the FVD benchmark.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: ynankani <ynankani@nvidia.com>
Signed-off-by: Yash Nankani <ynankani@nvidia.com>
2026-04-08 19:19:13 +00:00