ID-Animator: Zero-Shot Identity-Preserving Human Video Generation
Xuanhua He Quande Liu Shengju Qian Xin Wang Tao Hu Ke Cao Keyu Yan Man Zhou Jie Zhang



SMPLer: Taming Transformers for Monocular 3D Human Shape and Pose Estimation
    Xiangyu Xu,  
		Lijuan Liu,  Shuicheng Yan
		
			X. Xu is with Xi'an Jiaotong University, Xi'an, China. E-mail: xuxiangyu2014@gmail.com.
			L. Liu is with Sea AI Lab, Singapore. E-mail: liulj@sea.com.
			S. Yan is with Skywork AI, Singapore. E-mail: shuicheng.yan@gmail.com.
			
		
		
	
    Received xx; accepted xx
==================================================================================================================================================================================================================================================================================================



^* Intern in Tencent  ^† Corresponding authors




Generating high fidelity human video with specified identities has attracted significant attention in the content generation community. However, existing techniques struggle to strike a balance between training efficiency and identity preservation, either requiring tedious case-by-case finetuning or usually missing the identity details in video generation process. 
In this study, we present ID-Animator, a zero-shot human-video generation approach that can perform personalized video generation given single reference facial image without further training. ID-Animator inherits existing diffusion-based video generation backbones with a face adapter to encode the ID-relevant embeddings from learnable facial latent queries. To facilitate the extraction of identity information in video generation, we introduce an ID-oriented dataset construction pipeline, which incorporates decoupled human attribute and action captioning technique from a constructed facial image pool. Based on this pipeline, a random face reference training method is further devised to precisely capture the ID-relevant embeddings from reference images, thus improving the fidelity and generalization capacity of our model for ID-specific video generation. Extensive experiments demonstrate the superiority of ID-Animator to generate personalized human videos over previous models. Moreover, our method is highly compatible with popular pre-trained T2V models like animatediff and various community backbone models, showing high extendability in real-world applications for video generation where identity preservation is highly desired. Our codes and checkpoints will be released at <https://github.com/ID-Animator/ID-Animator>.










§ INTRODUCTION

Personalized or customized generation is to create images consistent in style, subject, or character ID based on one or more reference images.
In the realm of image generation, considerable strides have been made in crafting this identity-specific content, particularly in the domain of human image synthesis <cit.>. Recently, text-driven video generation <cit.> has gathered substantial interest within the research community. These methods enable the creation of videos based on user-specified textual prompts. However, the quest for generating high-fidelity, identity-specific human videos remains an area to explore. The generation of identity-specific human videos holds profound significance, particularly within the film industry, where characters must authentically execute actions. Previous approaches to customization primarily emphasized specified postures <cit.>, styles <cit.>, and action sequences <cit.>, often employing additional control to ensure generated videos met user requirements. However, these methods largely overlook specific identity control. Some techniques involved model fine-tuning through methods like LoRA <cit.> and textural inversion <cit.> to achieve ID-specific control <cit.>, but at the expense of substantial training costs and necessitating separate training weights for each ID. Others relied on image prompts to guide the model in generating videos featuring particular subjects, yet encountered challenges such as intricate dataset pipeline construction and limited ID variations <cit.>.

Furthermore, the direct integration of image customization modules <cit.> into the video generation model resulted in poor quality, such as static motion and ineffective instruction following. As shown in Figure <ref>, the use of successful image customization methods, IP-Adapter, led to clear failures in following textual descriptions and identity preservation, as well as subtle motion dynamics. 


The field of ID-specified video generation currently confronts several notable challenges:

    
  * High training costs: Many ID-specified methods need large training costs, often due to the customization modules with large parameter counts and lack of prior knowledge, consequently imposing significant training overheads. These training costs hinder the widespread adoption and scalability of ID-specified video generation techniques.
    
  * Scarcity of high-quality text-video paired datasets: Unlike the image generation community, where datasets like LAION-face <cit.> are readily available, the video generation community lacks sufficient high-quality text-video paired datasets. Existing datasets, such as CelebV-text <cit.>, feature captions annotated with fixed templates that concentrate on emotion changes while ignoring human attributes and actions, making them unsuitable for ID-preserving video generation tasks. This scarcity hampers research progress, forcing many endeavors to resort to collecting private datasets.
    
  * Influence of ID-irrelevant features from reference images on video generation quality: The presence of ID-irrelevant features in reference images can adversely affect the quality of generated videos. Reducing the influence of such features poses a challenge, demanding novel solutions to ensure fidelity in ID-specified video generation.












Solutions
To tackle the first issue, we propose an efficient ID-specific video generation framework, named ID-Animator, which is composed of a pre-trained text-to-video diffusion model and a lightweight face adapter module. With this design, our module can complete training within a day on a single A100 GPU and can generate 21 frames of video on a single 3090 GPU.

To address the second issue, we build an ID-oriented dataset construction pipeline. By leveraging existing publicly available datasets, we introduce the concept of decoupled captions, which involves generating captions for human actions, human attributes, and a unified human description. Additionally, we utilize facial recognition, cropping, and other techniques to create corresponding reference images. Trained with the re-written captions, 
our ID-Animator significantly enhances its effectiveness in following instructions.
In response to the third issue, we devise a novel training method for using random face images as references. By randomly sampling faces from the face pool, we decouple ID-independent image content from ID-related facial features, allowing the adapter to focus on ID-related characteristics.





Through the aforementioned designs, our model can achieve ID-specific video generation in a lightweight manner. It seamlessly integrates into existing community models <cit.>, showcasing robust generalization and ID-preserving capabilities.

Our contribution can be summarized as follows:

    
  * We propose ID-Animator, a novel framework that can generate identity-specific videos given any reference ficial image without model tuning. It inherits pre-trained video diffusion models with a lightweight face adapter to encode the ID-relevant embeddings from learnable facial latent queries. To the best of our knowledge, this is the first endeavor towards achieving zero-shot ID-specific human video generation.
    
  * We develop an ID-oriented dataset construction pipeline to mitigate the missing of training dataset in presonalized video generation. Over publicly available data sources, we present decoupled captioning of human videos, which extracts textual descriptions for human attributes and action respectively to attain comprehensive human captions. Besides, a facial image pool is constructed over this dataset to facilitate the extraction of facial embeddings. 
    
    
  * Over this pipeline, we further devise a random reference training strategy for ID-Animator to precisely extract the identity-relevant features and diminish the influence of ID-irrelevant information inherent in the reference facial image, therefore improving the identity fidelity and generation ability in real-world applications for personalized human video generation. 




§ RELATED WORK



 §.§ Video Generation

Video generation has been a key area of interest in research for a long time. Early endeavors in the task utilized models like generative adversarial networks <cit.> and vector quantized variational autoencoder generate video <cit.>. However, due to the inherent model ability, this video lacks motion and details and is unable to achieve good results.
With the rise of the diffusion model <cit.>, notably the latent diffusion model <cit.> and its success in image generation, researchers have extended the diffusion model's applicability to video generation <cit.>. 
This technique can be classified into two parts: image-to-video and text-to-video generation.
The former essentially transforms a given image into a dynamic video, whereas the latter generates video only following text instructions, without any image as input.
Leading-edge methods, exemplified by these works, include Animate Diffusion <cit.>, Dynamicrafter <cit.>, Modelscope <cit.>, AnimateAnything <cit.>, and Stable Video <cit.>, among others. These techniques generally exploit pre-trained text-to-image models and intersperse them with diverse forms of temporal mixing layers. Although these techniques are pushing the boundaries of producing visually appealing videos, there is still a gap in providing user-specific video creation using reference images, like portraits.



 §.§ ID Preserving Image Generation

The impressive generative abilities of diffusion models have attracted recent research endeavors investigating their personalized generation potential. Current methods within this domain can be divided into two categories, based on the necessity of fine-tuning during the testing phase. A subset of these methods requires the fine-tuning of the diffusion model leveraging ID-specific datasets during the testing phase, representative techniques such as DreamBooth <cit.>, textual inversion <cit.>, and LoRA <cit.>. 
While these methods exhibit acceptable ID preservation abilities, they necessitate individual model training for each unique ID, thus posing a significant challenge related to training costs and dataset collection, subsequently hindering their practical applicability. 
The latest focus of research in this domain has shifted towards training-free methods that bypass additional fine-tuning or inversion processes in testing phase. During the inference phase, it is possible to create a high-quality ID-preserving image with just a reference image as the condition. Methods like Face0 <cit.> replace the final three tokens of text embedding with face embedding within CLIP's feature space, utilizing this new embedding as conditional for image generation. PhotoMaker <cit.>, on the other hand, takes a similar approach by stacking multiple images to reduce the influence of ID-irrelevant features. Similarly, IP-Adapter <cit.> decoupled reference image features and text features to facilitate cross attention, resulting in better instruction following. 
Concurrently, InstantID <cit.> combined the features of IP-Adapter and ControlNet <cit.>, utilizing both global structural attributes and the fine-grained features of reference images for the generation of ID-preserving images. Although these methods have yielded promising results, the domain of video generation still remains relatively underexplored.


 §.§ Subject Driven Video Generation

Research on subject-driven video generation is still in its early stages, with two notable works being VideoBooth <cit.> and MagicMe <cit.>. VideoBooth <cit.> strives to generate videos that maintain high consistency with the input subject by utilizing the subject's clip feature and latent embedding obtained through a VAE encoder. This approach offers more fine-grained information than ID-preserving generation methods; however, its limitation remains as the subjects required to be present in the training data, such as cats, dogs, and vehicles, which results in a restricted range of applicable subjects. MagicMe <cit.>, on the other hand, is more closely related to the ID-preserving generation task. It learns ID-related representations by generating unique prompt tokens for each ID. However, this method requires separate training for each ID, making it unable to achieve zero-shot training-free capabilities. This limitation poses a challenge for its practical application. Our proposed method distinguishes itself from these two approaches by being applicable to any human image without necessitating retraining during inference. 





§ METHOD





 §.§ Overview

Given a reference ID image, ID-Animator endeavors to produce high-fidelity ID-specific human videos. Figure <ref> demonstrates our methods, featuring three pivotal constituents: a dataset reconstruction pipeline, the ID-Animator framework, and the random reference strategy employed during the training process of ID-Animator.


 §.§ ID-Animator

As depicted at the bottom of Figure <ref>, our ID-Animator framework comprises two components: the backbone text-to-video model, which is compatible with diverse T2V models, and the face adapter, which is subject to training for efficiency.

Pretrained Text to Video Diffusion Model The pre-trained text-to-video diffusion model exhibits strong video generation prowess, yet it lacks efficacy in the realm of ID-specific human video generation. Thus, our objective is to harness the existing capabilities of the T2V model and tailor it to the ID-specific human video generation domain. Specifically, we employ AnimateDiff <cit.> as our foundational T2V model.

Face Adapter
The advent of image prompting has substantially bolstered the generative ability of diffusion models, particularly when the desired content is challenging to describe precisely in text. IP-Adapter <cit.> proposed a novel method, enabling image prompting capabilities on par with text prompts, without necessitating any modification to the original diffusion model. Our approach mirrors the decoupling of image and text features in cross-attention. 
This procedure can be mathematically expressed as:

    Z_new = Attention(Q,K^t,V^t)+λ· Attention(Q,K^i,V^i)

where Q, K^t, and V^t denote the query, key, and value matrices for text cross-attention, respectively, while K^i and V^i correspond to image cross-attention. Provided the query features Z and the image features c_i, Q = ZW_q, K^i = c_iW^i_k, and V^i =c_iW^i_v. Only W^i_k and W^i_v are trainable weights.

Inspired by IP-Adapter, we limit our modifications to the cross-attention layer in the video generation model, leaving the temporal attention layer unchanged to preserve the original generative capacity of the model. 
A lightweight face adapter module is designed, encompassing a handful of simple query-based image encoder and the cross-attention module with trainable cross-attention projection weights, as shown in Figure <ref>.
The image feature c_i is derived from the clip feature of the reference image, and is further refined by the query-based image encoder. The other weights in cross attention module are initialized from the original diffusion model, of which the projection weights W_K^i and W_V^i are initialized using the weights of the IP-Adapter, facilitating the acquisition of preliminary image prompting capabilities and reducing the overall training costs.
Through a simplified and rapid training process, we can attain a video generation model with identity preservation capability.



 §.§ ID-Oriented Human Dataset Reconstruction

Contrary to identity-preservation image generation tasks, video generation tasks currently suffer from a lack of identity-oriented datasets. The dataset most relevant to our work is the CelebV-HQ <cit.> dataset, comprising 35,666 video clips that encompass 15,653 identities and 83 manually labeled facial attributes covering appearance, action, and emotion. However, their captions are derived from manually set templates, primarily focusing on facial appearance and human emotion while neglecting the comprehensive environment, human action, and detailed attributes of video. Additionally, its style significantly deviates from the user instructions, rendering it unsuitable for contemporary video generation models, and it lacks facial labels, such as masks and bounding boxes. Consequently, we find it necessary to reconstruct this dataset into an identity-oriented human dataset. Our pipeline incorporates caption rewriting and face detection, coupled with cropping.



  §.§.§ Decoupled Human Video Caption Generation

To enhance the instruction following ability of ID-Animator, we design a comprehensive restructuring of the captions within the CelebV-HQ dataset. 
To produce high-quality human videos, it is crucial for the caption to comprehensively encapsulate the semantic information and intricate details present within the video. Consequently, the caption must incorporate detailed attributes of the individual as well as the actions they are performing in the video. 
In light of this, we employ a novel rewriting technique that decouples the caption into two distinct components: human attributes and human actions. Subsequently, we leverage a language model to amalgamate these elements into a cohesive and comprehensive caption, as illustrated at the top of Figure <ref>.

Human Attribute Caption
As a preliminary step, we focus on crafting an attribute caption that aims to vividly depict the individual's preferences and the surrounding context. To achieve this, we employ the ShareGPT4V <cit.> model for caption generation. Recognized as a leading tool within the image captioning domain, ShareGPT4V is trained using a dataset generated by GPT4, enabling it to provide detailed descriptions of images.
We choose the median frame of the video as the input for ShareGPT4V. This approach allows us to generate detailed character descriptions that incorporate a wealth of attribute information.

Human Action Caption


Our objective is to create human videos with accurate and rich motions, where a mere human attribute caption is insufficient for our needs. We require a caption that emphasizes the overall dynamism and actions inherent in the video. To address this requirement, we introduce the concept of a human action caption, which strives to depict the action present within the video. These captions are specifically designed to concentrate on the semantic content across the entire video, facilitating a comprehensive understanding of the individual's actions captured therein. To achieve this goal, we leverage the Video-LLava <cit.> model, which has been trained on video data and excels at focusing on the overall dynamism. By employing Video-LLava, we ensure that our captions effectively convey the dynamic nature of the actions taking place in the video.

Unified Human Caption
The limitations of relying solely on human attribute captions and human action captions are demonstrated in Figure <ref>. Human attribute caption fails to encompass the overall action of the individual, while human action caption neglects the detailed characteristics of the subject. To address this, we designed a unified human caption that amalgamates the benefits of both caption types, using this comprehensive caption to train our model.

We employ a Large Language Model to facilitate this integration, capitalizing on its capacity for human-like expression and its prowess in generating high-quality captions. The GPT-3.5 API is utilized in this process. As depicted in Figure <ref>, the rewritten caption effectively encapsulates the video scene, aligning more closely with human instructions.

Figure <ref> also illustrates the shortcomings of the CelebV-caption, which deviates from the human instruction distribution and even includes incorrect information (e.g., a young boy is erroneously annotated as a woman). Our method disentangles the video content into attributes and actions, yielding more comprehensive results.



  §.§.§ Random Face Extraction for Face Pool Construction

In contrast to previous methods <cit.>, our approach does not directly utilize a frame from the video as a reference image. Instead, we opt to extract the facial region from the video, using this as the identity reference image. This strategy effectively reduces the influence of ID-irrelevant features on video generation.
Simultaneously, our technique differs from the image reconstruction training strategy
 employed in the ID preservation image generation works <cit.>, which typically reconstructs a reference image using the same image as condition. Hence, we adopt a more stochastic approach for training with randomrized face extraction.

As depicted at the bottom of Figure <ref>, we employ shuffling on video sqeuences and extract facial region from five randomly selected frames. In instances where a frame contains more than one face, it is discarded and additional frames are selected for re-extraction. The extracted facial images are subsequently stored in the face pool. This stochastic approach of facial extraction enables us to disentangle identity information from the semantic content of the video.















 §.§ Random Reference Training For Diminishing ID-Irrelevant Features

Prior to presenting our approach, we begin by revising the training methods of existing identity preservation image generation models, thereby highlighting the distinctions between our proposed method and previous research.

In the training phase of the Diffusion model, the objective is to estimate the noise ϵ at the current time step t from a noisy latent representation z_t. This noisy latent z_t is derived from the clean latent z combined with the noise component associated with the current time step t, i.e., z_t=f (z, t). This optimization procedure can be expressed by the following function:

    ℒ = E_z_t,t,ϵ 𝒩(0,1)[||ϵ-ϵ_θ(z_t,t)||_2^2]

Generally speaking, the clean latent z can be either the image itself or its embedding within the feature space. Specifically, within the context of the latent diffusion model, z originates from the encoding of image I obtained via the VAE encoder. Consequently, this process can be viewed as the reconstruction of z from a given noisy encoding z_t. Incorporating conditions such as a text condition C and an image condition C_i, this process can be mathematically expressed as:

    ℒ = E_z_t,t,C,C_i,ϵ 𝒩(0,1)[||ϵ-ϵ_θ(z_t,t,C,C_i)||_2^2]


In current identity preservation image generation models, the image condition C_i and the reconstruction target Z typically originate from the same image I. For instance, Face0 <cit.>, InstantID <cit.>, and FaceStudio <cit.> utilize image I as the target latent Z, with the facial region of I serving as C_i. Conversely, PhotoMaker <cit.>, Anydoor <cit.>, and IP-Adapter directly employ the feature of image I as C_i. In the learning phase of image reconstruction, this approach provides overly strong conditions for the diffusion model, which not only concentrates on facial features but also encompasses extraneous features such as the background, characters, and adornments (e.g., hats, jewelry, glasses). This may result in the neglect of domain-invariant identity features.
When directly applying this technique to videos, we are essentially using the first frame as a guide to recreate the video sequence.

This strong conditioning can cause the model to devolve into an image-to-video model, where the video content becomes heavily dependent on the semantic information of the reference image, rather than focusing on its facial embedding.

However, character identity should exhibit domain invariance, implying that given images of the same individual from various angles and attire, video generation outcomes should be similar. Therefore, drawing inspiration from the Monte Carlo concept, we designed a random reference training methodology. This approach employs weakly correlated images with the current video sequence as the condition C_j, effectively decoupling the generated content from the reference images. Specifically, during training, we randomly select a reference image from the previously extracted face pool, as depicted in Figure <ref>. By employing this Monte Carlo technique, the features from diverse reference images are averaged, reducing the influence of identity-invariant features. This transformation of the mapping from (C,C_i)->Z to (C,C_j)->Z not only diminishes the impact of extraneous features but also boosts the model's capacity to follow user instructions.




















§ EXPERIMENT



 §.§ Implementation details

We employe the open-source AnimateDiff <cit.> as our text-to-video generation model. Our training dataset is processed by clipping to 16 frames, center cropping, and resizing to 512x512 pixels. During training, only the parameters of the face adapter are updated, while the pre-trained text-to-video model remains frozen. Our experiments are carried out on a single NVIDIA A100 GPU (80GB) with a batch size of 2. We load the pretrained weights of IP-Adapter and set the learning rate to 1e-4 for our trainable adapter. Furthermore, to enhance the generation performance using classifier-free guidance, we applied a 20% probability of utilizing null-text embeddings to replace the original updated text embedding.

We utilize a subset of the CelebV dataset as our primary dataset, comprising 15k videos, and construct our identity-oriented dataset based on this foundation. Following the filtering of videos containing multiple faces, the final dataset employed for training contains 13k videos. 
































 §.§ Qualitative Comparison

We offer a qualitative comparison between our approach and the well-known methods in the domain of ID preserving image generation, specifically, the IP-Adapter-Plus-Face <cit.> and IP-Adapter-FaceID-Portrait <cit.>. We choose three images of celebrities and three of ordinary individuals as test cases, with the images of the latter being sourced from unused data in the CelebV dataset. We randomly generated six prompts from LLM, maintaining consistency with human language style, thus allowing us to assess the model's ability to follow instructions.

As depicted in Figure <ref>, it is evident that our approach yields the most desirable outcomes. The face generated by IP-Adapter-Plus-Face demonstrates a certain level of deformation, whereas the IP-Adapter-FaceID-Portrait model is deficient in facial structural information, resulting in a diminished similarity between the generated outputs and the reference image.

The results presented in Figure <ref> further underscore the superiority of our approach, showcasing the most pronounced motion, the highest facial similarity, and the capability of instruction following.








 §.§ Application

In this section, We showcase the potential applications of our model, encompassing recontextualization, alteration of age or gender, ID mixing, and integration with ControlNet or community models <cit.> to generate highly customized videos.


  §.§.§ Recontextualization

Given a reference image, our model is capable of generating ID fidelity videos and changing contextual information. The contextual information of characters can be tailored through text, encompassing attributes such as features, hair, clothing, creating novel character backgrounds, and enabling them to execute specific actions. As illustrated in Figure <ref>, we supply reference images and text, and the outcomes exhibit the robust editing and instruction-following capacities of our model.

As depicted in the figure <ref>, from top to bottom, we exhibit the model's proficiency in altering character hair, clothes, background, executing particular actions, and changing age or gender.



  §.§.§ Identity Mixing

The potential of our model to amalgamate different IDs is showcased in the figure <ref>. Through the blending of embeddings from two distinct IDs in varying proportions, we have effectively combined features from both IDs in the generated video. This experiment substantiates the proficiency of our face adapter in learning facial representations.





  §.§.§ Combination with ControlNet

Furthermore, our model demonstrates excellent compatibility with existing fine-grained condition modules, such as ControlNet <cit.>. We opted for SparseControlNet <cit.>, trained for AnimateDiff, as an additional condition to integrate with our model. As illustrated in Figure <ref>, we can supply either single frame control images or multi-frame control images. When a single frame control image is provided, the generated result adeptly fuses the control image with the face reference image. In cases where multiple control images are presented, the generated video sequence closely adheres to the sequence provided by the multiple images. This experiment highlights the robust generalization capabilities of our method, which can be seamlessly integrated with existing models.



  §.§.§ Inference with Community Models

We assessed the performance of our model using the https://civitai.com/ Civitai community model, and our model continues to function effectively with these weights, despite never having been trained on them. The selected models include https://civitai.com/models/22922/lyrielLyriel and https://civitai.com/models/113362/raemumixRaemumxi. As depicted in Figure <ref>, the first row presents the results obtained with the Lyriel model, while the second row showcases the outcomes achieved using the Raemuxi model. Our method consistently exhibits reliable facial preservation and motion generation capabilities.


§ CONCLUSION

In this research, our primary goal is to achieve ID-specific content generation in text-to-video (T2V) models. To this end, we introduce a ID-Animator framework to drive T2V models in generating ID-specific human videos using ID images. We facilitate the training of our ID-Animator by constructing an ID-oriented dataset based on publicly available resources, incorporating decoupled caption generation and face pool construction. Moreover, we develop a random face reference training method to minimize ID-irrelevant content in reference images, thereby directing the adapter's focus towards ID-related features. Our extensive experiments demonstrate that our ID-Animator generates stable videos with superior ID fidelity compared to previous models. 








































































































































































































































abbrvnat

