File size: 4,012 Bytes
d643072 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 |
# 💻 How to Inference & Test Metrics (FID, CLIP Score, GenEval, DPG-Bench, etc...)
This ToolKit will automatically inference your model and log the metrics results onto wandb as chart for better illustration. We curerntly support:
- \[x\] [FID](https://github.com/mseitzer/pytorch-fid) & [CLIP-Score](https://github.com/openai/CLIP)
- \[x\] [GenEval](https://github.com/djghosh13/geneval)
- \[x\] [DPG-Bench](https://github.com/TencentQQGYLab/ELLA)
- \[x\] [ImageReward](https://github.com/THUDM/ImageReward/tree/main)
### 0. Install corresponding env for GenEval and DPG-Bench
Make sure you can activate the following envs:
- `conda activate geneval`([GenEval](https://github.com/djghosh13/geneval))
- `conda activate dpg`([DGB-Bench](https://github.com/TencentQQGYLab/ELLA))
### 0.1 Prepare data.
Metirc FID & CLIP-Score on [MJHQ-30K](https://huggingface.co/datasets/playgroundai/MJHQ-30K)
```python
from huggingface_hub import hf_hub_download
hf_hub_download(
repo_id="playgroundai/MJHQ-30K",
filename="mjhq30k_imgs.zip",
local_dir="data/test/PG-eval-data/MJHQ-30K/",
repo_type="dataset"
)
```
Unzip mjhq30k_imgs.zip into its per-category folder structure.
```
data/test/PG-eval-data/MJHQ-30K/imgs/
├── animals
├── art
├── fashion
├── food
├── indoor
├── landscape
├── logo
├── people
├── plants
└── vehicles
```
### 0.2 Prepare checkpoints
```bash
huggingface-cli download Efficient-Large-Model/Sana_1600M_1024px --repo-type model --local-dir ./output/Sana_1600M_1024px --local-dir-use-symlinks False
```
### 1. directly \[Inference and Metric\] a .pth file
```bash
# We provide four scripts for evaluating metrics:
fid_clipscore_launch=scripts/bash_run_inference_metric.sh
geneval_launch=scripts/bash_run_inference_metric_geneval.sh
dpg_launch=scripts/bash_run_inference_metric_dpg.sh
image_reward_launch=scripts/bash_run_inference_metric_imagereward.sh
# Use following format to metric your models:
# bash $correspoinding_metric_launch $your_config_file_path $your_relative_pth_file_path
# example
bash $geneval_launch \
configs/sana_config/1024ms/Sana_1600M_img1024.yaml \
output/Sana_1600M_1024px/checkpoints/Sana_1600M_1024px.pth
```
### 2. \[Inference and Metric\] a list of .pth files using a txt file
You can also write all your pth files of a job in one txt file, eg. [model_paths.txt](../model_paths.txt)
```bash
# Use following format to metric your models, gathering in a txt file:
# bash $correspoinding_metric_launch $your_config_file_path $your_txt_file_path_containing_pth_path
# We suggest follow the file tree structure in our project for robust experiment
# example
bash scripts/bash_run_inference_metric.sh \
configs/sana_config/1024ms/Sana_1600M_img1024.yaml \
asset/model_paths.txt
```
### 3. You will get the following data tree.
```
output
├──your_job_name/ (everything will be saved here)
│ ├──config.yaml
│ ├──train_log.log
│ ├──checkpoints (all checkpoints)
│ │ ├──epoch_1_step_6666.pth
│ │ ├──epoch_1_step_8888.pth
│ │ ├──......
│ ├──vis (all visualization result dirs)
│ │ ├──visualization_file_name
│ │ │ ├──xxxxxxx.jpg
│ │ │ ├──......
│ │ ├──visualization_file_name2
│ │ │ ├──xxxxxxx.jpg
│ │ │ ├──......
│ ├──......
│ ├──metrics (all metrics testing related files)
│ │ ├──model_paths.txt Optional(👈)(relative path of testing ckpts)
│ │ │ ├──output/your_job_name/checkpoings/epoch_1_step_6666.pth
│ │ │ ├──output/your_job_name/checkpoings/epoch_1_step_8888.pth
│ │ ├──fid_img_paths.txt Optional(👈)(name of testing img_dir in vis)
│ │ │ ├──visualization_file_name
│ │ │ ├──visualization_file_name2
│ │ ├──cached_img_paths.txt Optional(👈)
│ │ ├──......
```
|