[NeurIPS 2023] ImageReward: Learning and Evaluating Human Preferences for Text-to-image Generation
š Paper ā¢ š¼ Dataset ā¢ š äøęå客 ā¢ š¤ HF Repo ā¢ š¦ Twitter
ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
š„News! The paper of ImageReward is accepted by NeurIPS 2023!
ImageReward is the first general-purpose text-to-image human preference RM, which is trained on in total 137k pairs of expert comparisons, outperforming existing text-image scoring methods, such as CLIP (by 38.6%), Aesthetic (by 39.6%), and BLIP (by 31.6%), in terms of understanding human preference in text-to-image synthesis.
Additionally, we introduce Reward Feedback Learning (ReFL) for direct optimizing a text-to-image diffusion model using ImageReward. ReFL-tuned Stable Diffusion wins against untuned version by 58.4% in human evaluation.
Both ImageReward and ReFL are all packed up to Python image-reward
package now!
Try image-reward
package in only 3 lines of code for ImageReward scoring!
# pip install image-reward
import ImageReward as RM
model = RM.load("ImageReward-v1.0")
rewards = model.score("<prompt>", ["<img1_obj_or_path>", "<img2_obj_or_path>", ...])
Try image-reward
package in only 4 lines of code for ReFL fine-tuning!
# pip install image-reward
# pip install diffusers==0.16.0 accelerate==0.16.0 datasets==2.11.0
from ImageReward import ReFL
args = ReFL.parse_args()
trainer = ReFL.Trainer("CompVis/stable-diffusion-v1-4", "data/refl_data.json", args=args)
trainer.train(args=args)
If you find ImageReward
's open-source effort useful, please š us to encourage our following developement!
We have integrated the whole repository to a single python package image-reward
. Following the commands below to prepare the environment:
# Clone the ImageReward repository (containing data for testing)
git clone https://github.com/THUDM/ImageReward.git
cd ImageReward
# Install the integrated package `image-reward`
pip install image-reward
We provide example images in the assets/images
directory of this repo. The example prompt is:
a painting of an ocean with clouds and birds, day time, low depth field effect
Use the following code to get the human preference scores from ImageReward:
import os
import torch
import ImageReward as RM
if __name__ == "__main__":
prompt = "a painting of an ocean with clouds and birds, day time, low depth field effect"
img_prefix = "assets/images"
generations = [f"{pic_id}.webp" for pic_id in range(1, 5)]
img_list = [os.path.join(img_prefix, img) for img in generations]
model = RM.load("ImageReward-v1.0")
with torch.no_grad():
ranking, rewards = model.inference_rank(prompt, img_list)
# Print the result
print("\nPreference predictions:\n")
print(f"ranking = {ranking}")
print(f"rewards = {rewards}")
for index in range(len(img_list)):
score = model.score(prompt, img_list[index])
print(f"{generations[index]:>16s}: {score:.2f}")
The output should be like as follow (the exact numbers may be slightly different depending on the compute device):
Preference predictions:
ranking = [1, 2, 3, 4]
rewards = [[0.5811622738838196], [0.2745276093482971], [-1.4131819009780884], [-2.029569625854492]]
1.webp: 0.58
2.webp: 0.27
3.webp: -1.41
4.webp: -2.03
pip install diffusers==0.16.0 accelerate==0.16.0 datasets==2.11.0
We provide example dataset for ReFL in the data/refl_data.json
of this repo. Run ReFL as following:
bash scripts/train_refl.sh
Download data: š¼ Dataset.
Make dataset.
cd train
python src/make_dataset.py
Set training config: train/src/config/config.yaml
One command to train.
bash scripts/train_one_node.sh
We have developed a custom script to integrate ImageReward into SD Web UI for a convenient experience.
The script is located at sdwebui/image_reward.py
in this repository.
The usage of the script is described as follows:
stable-diffusion-webui/scripts/
directoryhttps://github.com/THUDM/ImageReward/assets/98524878/9d8a036d-1583-4978-aac7-4b758edf9b89
https://github.com/THUDM/ImageReward/assets/98524878/b9f01629-87d6-4c92-9990-fe065711b9c6
Note that SD Web UI has two ways to set up its Python environment:
python launch.py
, Web UI will use the Python environmentĀ found in your PATH
(in Linux, you can check its exact path with which python
).webui-user.bat
, Web UI creates a newĀ venv environmentĀ in theĀ directory stable-diffusion-webui\venv
.
stable-diffusion-webui\venv\Scripts
Ā directory, runĀ activate
Ā orĀ activate.bat
Ā (if you are usingĀ cmd) orĀ activate.ps1
Ā (if you are usingĀ PowerShell) from .(venv)
Ā appear at the far left of the command line, you have successfully activated venv created by the SD Web UI.After activating the right Python environment, just do what you want to do true to form.
Note: The experimental results are produced in an environment that satisfies:
torch
Version: 1.12.1+cu113
According to our own reproduction experience, reproducing this experiment in other environments may cause the last decimal place to fluctuate, typically within a range of Ā±0.1.Run the following script to automatically download data, baseline models, and run experiments:
bash ./scripts/test-benchmark.sh
Then you can check the results in benchmark/results/
or the terminal.
If you want to check the raw data files individually:
benchmark/benchmark-prompts.json
.<model_name>.zip
contains a directory of the same name, in which there are in total 1000 images generated from 100 prompts of 10 images each.<model_name>.zip
should be decompressed into benchmark/generations/
as directory <model_name>
that contains images.
Run the following script to automatically download data, baseline models, and run experiments:
bash ./scripts/test.sh
If you want to check the raw data files individually:
data/test.json
.data/test_images
.@misc{xu2023imagereward,
title={ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation},
author={Jiazheng Xu and Xiao Liu and Yuchen Wu and Yuxuan Tong and Qinkai Li and Ming Ding and Jie Tang and Yuxiao Dong},
year={2023},
eprint={2304.05977},
archivePrefix={arXiv},
primaryClass={cs.CV}
}