Hugging Face's logo

viddle
/
viddle-pix2pix-animatediff

Model card Files Community
1
viddle-pix2pix-animatediff
File size: 912 Bytes 
506646f
 
 
 
 
 
 
 
 
 
 
 
c533bf6
506646f
 
 
c533bf6
506646f
 
 
c533bf6
506646f
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
 
# InstructPix2Pix for AnimateDiff

This model is a fine-tuned version of the AnimateDiff V2 model using the InstructPix2Pix as the T2I model. 

## Inference 

You need use a pipeline built for InstructPix2Pix and AnimateDiff inference. And example is [here](https://github.com/viddle-app/viddle-pix2pix-animatediff)

Using the pipeline, InstructPix2Pix T2I model and this motion module, you can generate animations that use an init image faithfully.

### Init Image

![Init Image](images/mona-lisa.jpg)

_Make her smile_

![Smiling](images/mona-lisa-smiles.mp4)

_Make her on fire_

![On Fire](images/mona-lisa-on-fire.mp4)

## Training
The training procedure was modified to pass the first frame of all videos to the extra channels of the InstructPix2Pix model. The model was trained 4500 steps with a learning rate of 5e-7 and 5500 steps at 1e-7, with a batch size of 1. 

## LICENSE 
Creative OpenRAIL License