VBench/vbench/third_party/amt/docs/develop.md

Development for evaluation and training

• Datasets
• Pretrained Models
• Evaluation
• Training

Datasets

First, please prepare standard datasets for evaluation and training.

We present most of prevailing datasets in video frame interpolation, though some are not used in our project. Hope this collection could help your research.

Dataset	:link: Source	Train/Eval	Arbitrary/Fixed
Vimeo90k	ToFlow (IJCV 2019)	Both	Fixed
ATD-12K	AnimeInterp (CVPR 2021)	Both	Fixed
SNU-FILM	CAIN (AAAI 2021)	Eval	Fixed
UCF101	Google Driver	Eval	Fixed
HD	MEMC-Net (TPAMI 2018)/Google Driver	Eval	Fixed
Xiph-2k/-4k	SoftSplat (CVPR 2020)	Eval	Fixed
MiddleBury	MiddleBury	Eval	Fixed
GoPro	GoPro	Both	Arbitrary
Adobe240fps	DBN (CVPR 2017)	Both	Arbitrary
X4K1000FPS	XVFI (ICCV 2021)	Both	Arbitrary

Pretrained Models

Dataset	:link: Download Links	Config file	Trained on	Arbitrary/Fixed
AMT-S	[Google Driver][Baidu Cloud]	[cfgs/AMT-S]	Vimeo90k	Fixed
AMT-L	[Google Driver][Baidu Cloud]	[cfgs/AMT-L]	Vimeo90k	Fixed
AMT-G	[Google Driver][Baidu Cloud]	[cfgs/AMT-G]	Vimeo90k	Fixed
AMT-S	[Google Driver][Baidu Cloud]	[cfgs/AMT-S_gopro]	GoPro	Arbitrary

Evaluation

Before evaluation, you should:

1. Check the dataroot is organized as follows:

./data
├── Adobe240
│   ├── original_high_fps_videos
│   └── test_frames # using ffmpeg to extract 240 fps frames from `original_high_fps_videos`
├── GOPRO
│   ├── test
│   └── train
├── SNU_FILM
│   ├── GOPRO_test
│   ├── test-easy.txt
│   ├── test-extreme.txt
│   ├── test-hard.txt
│   ├── test-medium.txt
│   └── YouTube_test
├── ucf101_interp_ours
│   ├── 1
│   ├── 1001
│   └── ...
└── vimeo_triplet
    ├── readme.txt
    ├── sequences
    ├── tri_testlist.txt
    └── tri_trainlist.txt

2. Download the provided pretrained models.

Then, you can perform evaluation as follows:

• Run all benchmarks for fixed-time models.

  sh ./scripts/benchmark_fixed.sh [CFG] [CKPT_PATH]
  ## e.g.
  sh ./scripts/benchmark_fixed.sh cfgs/AMT-S.yaml pretrained/amt-s.pth
  

• Run all benchmarks for arbitrary-time models.

  sh ./scripts/benchmark_arbitrary.sh [CFG] [CKPT_PATH]
  ## e.g.
  sh ./scripts/benchmark_arbitrary.sh cfgs/AMT-S.yaml pretrained/gopro_amt-s.pth
  

• Run a single benchmark for fixed-time models. You can custom data paths in this case.

  python [BENCHMARK] -c [CFG] -p [CKPT_PATH] -r [DATAROOT]
  ## e.g.
  python benchmarks/vimeo90k.py -c cfgs/AMT-S.yaml -p pretrained/amt-s.pth -r data/vimeo_triplet
  

• Run the inference speed & model size comparisons using:

  python speed_parameters.py -c [CFG]
  ## e.g.
  python speed_parameters.py -c cfgs/AMT-S.yaml
  

Training

Before training, please first prepare the optical flows (which are used for supervision).

We need to install cupy first before flow generation:

conda activate amt # satisfying `requirement.txt`
conda install -c conda-forge cupy

After installing cupy, we can generate optical flows by the following command:

python flow_generation/gen_flow.py -r [DATA_ROOT]
## e.g.
python flow_generation/gen_flow.py -r data/vimeo_triplet

After obtaining the optical flow of the training data, run the following commands for training (DDP mode):

 sh ./scripts/train.sh [NUM_GPU] [CFG] [MASTER_PORT]
 ## e.g.
 sh ./scripts/train.sh 2 cfgs/AMT-S.yaml 14514

Our training configuration files are provided in cfgs. Please carefully check the dataset_dir is suitable for you.

Note:

• If you intend to turn off DDP training, you can switch the key distributed from true to false in the config file.

• If you do not use wandb, you can switch the key logger.use_wandb from true to false in the config file.