刘虹雨 commited on
Commit
c2fc1d1
·
1 Parent(s): 56a37c5

update readme

Browse files
Files changed (1) hide show
  1. README.md +10 -193
README.md CHANGED
@@ -1,193 +1,10 @@
1
- <div align="center">
2
- <img src="docs/AvatarArtist.ico" width="250px">
3
- </div>
4
-
5
- <h2 align="center">
6
- <a href="https://arxiv.org/abs/2503.19906">
7
- [CVPR 2025] AvatarArtist: Open-Domain 4D Avatarization
8
- </a>
9
- </h2>
10
-
11
- <p align="center">
12
- <img alt="avatarrtist" src="docs/teaser.gif" width="80%">
13
- </p>
14
-
15
- [//]: # (<div align="center">)
16
-
17
- [//]: # ( <img src="docs/teaser.gif" width="350px">)
18
-
19
- [//]: # (</div>)
20
-
21
- <h5 align="center"> If you like our project, please give us a star ⭐ on GitHub for the latest update. </h5>
22
-
23
- <h5 align="center">
24
- <a href='https://kumapowerliu.github.io/AvatarArtist'>
25
- <img src='https://img.shields.io/badge/Project-Page-green'>
26
- </a>
27
- <a href='https://arxiv.org/abs/2503.19906'>
28
- <img src='https://img.shields.io/badge/Technique-Report-red'>
29
- </a>
30
- <a href='https://huggingface.co/KUMAPOWER/AvatarArtist'>
31
- <img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue'>
32
- </a>
33
- <a href="https://github.com/ant-research/AvatarArtist">
34
- <img src="https://img.shields.io/github/stars/ant-research/AvatarArtist?style=social" alt="GitHub stars">
35
- </a>
36
- </h5>
37
-
38
- <div align="center">
39
- This repository contains the official implementation of AvatarArtist, a method for generating 4D avatars from a single image.
40
- </div>
41
-
42
- <br>
43
-
44
- <details open>
45
- <summary>💡 We also have other avatar projects that may interest you ✨.</summary>
46
- <p>
47
-
48
- > **[HeadArtist: Text-conditioned 3D Head Generation with Self Score Distillation, SIGGRAPH 2024](https://arxiv.org/abs/2312.07539)**
49
- > Hongyu Liu, Xuan Wang, Ziyu Wan, etc.
50
- > <span>
51
- > <a href='https://github.com/ant-research/HeadArtist'><img src='https://img.shields.io/badge/-Github-black?logo=github'></a>
52
- > <a href='https://kumapowerliu.github.io/HeadArtist'><img src='https://img.shields.io/badge/Project-Page-green'></a>
53
- > <a href='https://arxiv.org/abs/2312.07539'><img src='https://img.shields.io/badge/Arxiv-2312.07539-b31b1b.svg?logo=arXiv'></a>
54
- > </span>
55
-
56
- > **[Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation, SIGGRAPH Asia 2024](https://arxiv.org/abs/2406.01900)**
57
- > Yue Ma, Hongyu Liu, Hongfa Wang, etc.
58
- > <span><a href='https://github.com/mayuelala/FollowYourEmoji'><img src='https://img.shields.io/badge/-Github-black?logo=github'></a>
59
- > <a href='https://follow-your-emoji.github.io/'><img src='https://img.shields.io/badge/Project-Page-green'></a>
60
- > <a href='https://arxiv.org/abs/2406.01900'><img src='https://img.shields.io/badge/Arxiv-2406.01900-b31b1b.svg?logo=arXiv'></a></span>
61
-
62
- </details>
63
-
64
- ## 🚨 News
65
- - [03/26/2025] Inference Code and pretrained models released!
66
-
67
- ## ⚙️ Setup
68
-
69
- ### Environment
70
-
71
- ```bash
72
- git clone --depth=1 https://github.com/ant-research/AvatarArtist
73
- cd AvatarArtist
74
- conda create -n avatarartist python=3.9.0
75
- conda activate avatarartist
76
- pip install -r requirements.txt
77
- ```
78
-
79
- ### Download Weights
80
-
81
- The weights are available at [🤗HuggingFace](https://huggingface.co/KumaPower/AvatarArtist), you can download it with the following commands. Please move the required files into the `pretrained_model` directory:
82
-
83
- ```bash
84
- # if you are in china mainland, run this first: export HF_ENDPOINT=https://hf-mirror.com
85
- huggingface-cli download --repo-type model \
86
- KUMAPOWER/AvatarArtist \
87
- --local-dir pretrained_model
88
- ```
89
-
90
-
91
- ## 🤗 Usage
92
-
93
- ### Inference
94
- <div align="center">
95
- <img src="docs/avatarartist_infer_overview.gif" width="850px" />
96
- <p>
97
- <em>
98
- Our approach consists of two steps during the inference process. First, the DiT model generates a 4D representation based on the input image. Then, our Motion-Aware Cross-Domain Renderer takes this 4D representation as input and, guided by both the input image and driving signals, renders it into the final target image.
99
- </em>
100
- </p>
101
- </div>
102
-
103
- This is an example of inference using the demo data. The images used in this example are sourced from https://civitai.com/.
104
- ```python
105
- python3 inference.py \
106
- --img_file './demo_data/source_img/img_from_web/images512x512/final_ipimgs' \
107
- --input_img_fvid './demo_data/source_img/img_from_web/coeffs/final_ipimgs' \
108
- --input_img_motion './demo_data/source_img/img_from_web/motions/final_ipimgs' \
109
- --video_name 'Obama' \
110
- --target_path './demo_data/target_video/data_obama'
111
- # --use_demo_cam (create a video like the teaser using predefined camera parameters)
112
- ```
113
-
114
- This is an example of performing inference using the model. The images used in this example are diverse-domain images generated by a diffusion model, as described in our paper. You can use the --select_img option to specify a particular input image.
115
- ```python
116
- python3 inference.py \
117
- --img_file './demo_data/source_img/img_generate_different_domain/images512x512/demo_imgs' \
118
- --input_img_fvid './demo_data/img_generate_different_domain/coeffs/demo_imgs' \
119
- --input_img_motion './demo_data/source_img/img_generate_different_domain/motions/demo_imgs' \
120
- --video_name "Obama" \
121
- --target_path './demo_data/target_video/data_obama' \
122
- --select_img 'your_selected_image.png in img_file'
123
- ```
124
-
125
-
126
- ### Custom Data Processing
127
-
128
- We provide a set of scripts to process input images and videos for use with our model. These scripts ensure that the data is properly formatted and preprocessed, making it compatible with our inference pipeline. You can use them to prepare your own custom data for generating results with our model.
129
-
130
- Please refer to [this guide](https://github.com/ant-research/AvatarArtist/tree/main/data_process) to learn how to obtain the inference data. You can also check the [demo data](https://github.com/ant-research/HeadArtist/tree/main/demo_data) for reference. The data structure is shown below.
131
-
132
- The files in the `"dataset"` folder serve as the final input to the model, while the other files are intermediate outputs from the data processing pipeline:
133
-
134
- ```
135
- 📦 datasets/
136
- ├── 📂 dataset/
137
- │ ├── 📂 coeffs/
138
- │ ├── 📂 images512x512/
139
- │ ├── 📂 uvRender256x256/
140
- │ ├── 📂 orthRender256x256_face_eye/
141
- │ ├── 📂 motions/
142
- ├── 📂 crop_fv_tracking/
143
- ├── 📂 realign_detections/
144
- ├── 📂 realign_detections/
145
- ├── 📂 realign/
146
- ├── 📂 raw_detection/
147
- ├── 📂 align_3d_landmark/
148
- ├── 📂 raw_frames/
149
- ```
150
-
151
- ### Different domain's input images generation
152
-
153
- We provide a set of scripts to transfer the realistic domain's portrait to the other domain. Please refer to [this guide](https://github.com/ant-research/AvatarArtist/tree/main/different_domain_imge_gen).
154
-
155
-
156
-
157
- ## **📋 To-Do List**
158
- ### **Pending Tasks**
159
- - [ ] Gradio demo
160
- - [ ] Release training code
161
-
162
- ---
163
-
164
- ### **✅ Completed Tasks**
165
- - [x] Release inference code
166
- - [x] Release data processing tools
167
- - [x] Release the pipeline to generate input for different domains
168
-
169
- ## 👍 Credits
170
-
171
- We sincerely appreciate the contributions of the following open-source projects, which have significantly influenced our work:
172
-
173
- - **DiT** builds upon [PixArt-alpha](https://github.com/PixArt-alpha/PixArt-alpha).
174
- - **VAE** is based on [LVDM](https://github.com/YingqingHe/LVDM).
175
- - **Motion-aware rendering** is inspired by [Portrait4D](https://github.com/YuDeng/Portrait-4D).
176
- - **4D representation** in our paper is proposed in [Next3D](https://github.com/MrTornado24/Next3D) and [Next3D++](https://github.com/XChenZ/invertAvatar).
177
- - We referenced [DATID3D](https://github.com/gwang-kim/DATID-3D) for domain-specific prompts.
178
-
179
- ## 🔒 License
180
-
181
- * The majority of this project is released under the Apache 2.0 license as found in the [LICENSE](LICENSE) file.
182
-
183
- ## ✏️ Citation
184
- If you make use of our work, please cite our paper.
185
- ```bibtex
186
- @article{liu2025avatarartist,
187
- title={AvatarArtist: Open-Domain 4D Avatarization},
188
- author={Hongyu Liu, Xuan Wang, Ziyu Wan, Yue Ma, Jingye Chen, Yanbo Fan, Yujun Shen, Yibing Song, Qifeng Chen },
189
- booktitle={CVPR},
190
- year={2025}
191
- }
192
- ```
193
-
 
1
+ title: AvatarArtist
2
+ emoji:
3
+ colorFrom: red
4
+ colorTo: indigo
5
+ sdk: gradio
6
+ sdk_version: 5.20.1
7
+ app_file: app.py
8
+ pinned: false
9
+ license: apache-2.0
10
+ short_description: Open-Domain 4D Avatarization