This simple code demonstrates how to evaluate images generated by diffusion (or other generative) models using popular metrics such as imagereward, clip-iqa, and clip.

Setup

Install Requirements

pip install -r requirments.txt

Data Format

Prepare a JSON file with your prompts and corresponding images in the structure below:

[
  {
    "prompt": "YOUR_PROMPT",
    "images": {
      "MODEL_NAME": "PATH_TO_THE_IMAGE",
      "MODEL_NAME": "PATH_TO_THE_IMAGE",
      ...
    }
  },
  ...
]

prompt: The text prompt used to generate the images.
images: Key-value pairs of model names and image file paths.

Usage

Run the evaluation script with your JSON file:

python main.py --data-path {PATH_TO_THE_IMAGE_JSON_PATH} --metrics imagereward

--data-path: Path to your JSON file.
--metrics: One or more metrics to compute (e.g. imagereward, clip-iqa, clip).

Sample results

Example metrics obtained with 30 sampling steps on a set of 1K prompts (values will vary based on data and model configurations):

Model	Precision	ImageReward	CLIP-IQA	CLIP
FLUX 1 Dev	BF16	1.118	0.927	30.15
	FP4 PTQ	1.096	0.923	29.86
	FP4 QAT	1.119	0.928	29.919