mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-20 05:59:32 +08:00
ynankani-nv
4a70040ddd
### What does this PR do? Type of change: ? New benchmark addition <!-- Details about the change. --> This PR adds the following to examples/windows/: 1) FVD (Fréchet Video Distance) evaluation tool — a standalone script for computing FVD between two sets of videos using a pre-trained I3D model (Kinetics-400, 1024-dim pooled features). 2) Directory reorganization — moved torch_onnx/ and qad_example/ under a new diffusers/ folder for clearer grouping of diffusion model examples. 3) Minor fixes: Updated WikiText-2 dataset link in perplexity_metrics/README.md (old wikitext → Salesforce/wikitext) ### Usage ```python python compute_fvd.py \ --ref-dir /path/to/reference/videos \ --gen-dir /path/to/generated/videos \ --output results.json \ --device cuda ``` ### Testing Evaluated FVD tool on LTX-2.3 outputs comparing QAD vs PTQ checkpoints (both NVFp4) against a BF16 baseline across 11 VBench dimensions. ### Before your PR is "*Ready for review*" Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md) and your commits are signed (`git commit -s -S`). Make sure you read and follow the [Security Best Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors) (e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(..., weights_only=False)`, `pickle`, etc.). - Is this change backward compatible?: ✅ - If you copied code from any other sources or added a new PIP dependency, did you follow guidance in `CONTRIBUTING.md`: N/A <!--- Mandatory --> - Did you write any new necessary tests?: N/A <!--- Mandatory for new features or examples. --> - Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?: N/A <!--- Only for new features, API changes, critical bug fixes or backward incompatible changes. --> ### Additional Information <!-- E.g. related issue. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added FVD (Fréchet Video Distance) evaluation: command‑line tool, I3D‑based feature extraction, PCA option, and quantized comparisons (PTQ vs QAD) with per‑category and overall results showing QAD's lower average FVD. * **Documentation** * Expanded benchmarks TOC with “Additional Metrics”, added FVD evaluation guide, usage examples, troubleshooting, and updated perplexity dataset reference. * **Chores** * Added Python requirements for the FVD benchmark. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: ynankani <ynankani@nvidia.com> Signed-off-by: Yash Nankani <ynankani@nvidia.com>
30 lines
792 B
YAML
30 lines
792 B
YAML
compute_environment: LOCAL_MACHINE
|
|
debug: false
|
|
distributed_type: FSDP
|
|
downcast_bf16: 'yes'
|
|
enable_cpu_affinity: false
|
|
fsdp_config:
|
|
fsdp_activation_checkpointing: false
|
|
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
|
|
fsdp_backward_prefetch: BACKWARD_PRE
|
|
fsdp_cpu_ram_efficient_loading: true
|
|
fsdp_forward_prefetch: false
|
|
fsdp_offload_params: false
|
|
fsdp_reshard_after_forward: FULL_SHARD
|
|
fsdp_state_dict_type: SHARDED_STATE_DICT
|
|
fsdp_sync_module_states: true
|
|
fsdp_transformer_layer_cls_to_wrap: BasicAVTransformerBlock
|
|
fsdp_use_orig_params: true
|
|
fsdp_version: 1
|
|
machine_rank: 0
|
|
main_training_function: main
|
|
mixed_precision: "no"
|
|
num_machines: 1
|
|
num_processes: 8
|
|
rdzv_backend: static
|
|
same_network: true
|
|
tpu_env: []
|
|
tpu_use_cluster: false
|
|
tpu_use_sudo: false
|
|
use_cpu: false
|