# Diffusion Image Evaluation

This simple code demonstrates how to evaluate images generated by diffusion (or other generative) models using popular metrics such as [imagereward](https://arxiv.org/abs/2304.05977), [clip-iqa](https://arxiv.org/abs/2207.12396), and [clip](https://arxiv.org/abs/2104.08718).

## Setup

### Install Requirements

```bash
pip install -r requirments.txt
```

### Data Format

Prepare a JSON file with your prompts and corresponding images in the structure below:

```json
[
  {
    "prompt": "YOUR_PROMPT",
    "images": {
      "MODEL_NAME": "PATH_TO_THE_IMAGE",
      "MODEL_NAME": "PATH_TO_THE_IMAGE",
      ...
    }
  },
  ...
]
```

- `prompt`: The text prompt used to generate the images.
- `images`: Key-value pairs of model names and image file paths.

## Usage

Run the evaluation script with your JSON file:

```bash
python main.py --data-path {PATH_TO_THE_IMAGE_JSON_PATH} --metrics imagereward
```

- `--data-path`: Path to your JSON file.
- `--metrics`: One or more metrics to compute (e.g. imagereward, clip-iqa, clip).

### Sample results

Example metrics obtained with 30 sampling steps on a set of 1K prompts (values will vary based on data and model configurations):

| Model | Precision | ImageReward | CLIP-IQA | CLIP |
|:------------:|:------------:|:------------:|:------------:|:------------:|
| FLUX 1 Dev | BF16 | 1.118 | 0.927 | 30.15 |
| | FP4 PTQ | 1.096 | 0.923 | 29.86 |
| | FP4 QAT | 1.119 | 0.928 | 29.919 |