mirror of
https://github.com/NVIDIA/Model-Optimizer.git
synced 2026-10-26 06:54:04 +08:00
54 lines
1.4 KiB
Markdown
54 lines
1.4 KiB
Markdown
# Diffusion Image Evaluation
|
|
|
|
This simple code demonstrates how to evaluate images generated by diffusion (or other generative) models using popular metrics such as [imagereward](https://arxiv.org/abs/2304.05977), [clip-iqa](https://arxiv.org/abs/2207.12396), and [clip](https://arxiv.org/abs/2104.08718).
|
|
|
|
## Setup
|
|
|
|
### Install Requirements
|
|
|
|
```bash
|
|
pip install -r requirments.txt
|
|
```
|
|
|
|
### Data Format
|
|
|
|
Prepare a JSON file with your prompts and corresponding images in the structure below:
|
|
|
|
```json
|
|
[
|
|
{
|
|
"prompt": "YOUR_PROMPT",
|
|
"images": {
|
|
"MODEL_NAME": "PATH_TO_THE_IMAGE",
|
|
"MODEL_NAME": "PATH_TO_THE_IMAGE",
|
|
...
|
|
}
|
|
},
|
|
...
|
|
]
|
|
```
|
|
|
|
- `prompt`: The text prompt used to generate the images.
|
|
- `images`: Key-value pairs of model names and image file paths.
|
|
|
|
## Usage
|
|
|
|
Run the evaluation script with your JSON file:
|
|
|
|
```bash
|
|
python main.py --data-path {PATH_TO_THE_IMAGE_JSON_PATH} --metrics imagereward
|
|
```
|
|
|
|
- `--data-path`: Path to your JSON file.
|
|
- `--metrics`: One or more metrics to compute (e.g. imagereward, clip-iqa, clip).
|
|
|
|
### Sample results
|
|
|
|
Example metrics obtained with 30 sampling steps on a set of 1K prompts (values will vary based on data and model configurations):
|
|
|
|
| Model | Precision | ImageReward | CLIP-IQA | CLIP |
|
|
|:------------:|:------------:|:------------:|:------------:|:------------:|
|
|
| FLUX 1 Dev | BF16 | 1.118 | 0.927 | 30.15 |
|
|
| | FP4 PTQ | 1.096 | 0.923 | 29.86 |
|
|
| | FP4 QAT | 1.119 | 0.928 | 29.919 |
|