MuseumMaker: Continual Style Customization without Catastrophic Forgetting
Chenxi Liu Gan Sun Wenqi Liang Jiahua Dong Can Qin Yang Cong









Journal of  Class Files, Vol. 14, No. 8, August 2021
Shell et al.: A Sample Article Using IEEEtran.cls for IEEE Journals





MuseumMaker: Continual Style Customization without Catastrophic Forgetting
    Chenxi Liu^†, Gan Sun^* Member, IEEE, Wenqi Liang^†, Jiahua Dong, Can Qin, Yang Cong
        



Chenxi Liu and Wenqi Liang are with the State Key Laboratory of Robotics, the Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110169, China, and also
with the University of Chinese Academy of Sciences, Beijing 100049, China
.
Gan Sun and Yang Cong are with the School of Automation Science and Engineering, South China University of Technology, Guangzhou, 510640, China. 
^†These authors contributed equally to this work.
^*The corresponding author is Prof. Gan Sun.
    April 29, 2024
====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================




Pre-trained large text-to-image (T2I) models with an appropriate text prompt has attracted growing interests in customized images generation field. However, catastrophic forgetting issue make it hard to continually synthesize new user-provided styles while retaining the satisfying results amongst learned styles. In this paper, we propose MuseumMaker, a method that enables the synthesis of images by following a set of customized styles in a never-end manner, and gradually accumulate these creative artistic works as a Museum. When facing with a new customization style, we develop a style distillation loss module to transfer the style of the whole dataset into generation of images. It can minimize the learning biases caused by content of images, and address the catastrophic overfitting issue induced by few-shot images. To deal with catastrophic forgetting amongst past learned styles, we devise a dual regularization for shared-LoRA module to optimize the direction of model update, which could regularize the diffusion model from both weight and feature aspects, respectively. Meanwhile, a unique token embedding corresponding to this new style is learned by a task-wise token learning module, which could preserve historical knowledge from past styles with the limitation of LoRA parameter quantity. As any new user-provided style come, our MuseumMaker can capture the nuances of the new styles while maintaining the details of learned styles. Experimental results on diverse style datasets validate the effectiveness of our proposed MuseumMaker method, showcasing its robustness and versatility across various scenarios.              








Text-to-Image Model, Image Generation, Style Customization, Continual learning.





§ INTRODUCTION

Text-to-image (T2I) models have emerged in recent studies. Generative models based on diffusion model<cit.><cit.><cit.> have demonstrated remarkable efficacy and flexibility in the field of text-to-image generation. Amongst these methods, Stable Diffusion <cit.> stands out for its ability to generate high quality images through simple language descriptions, which advances the application of generative models to new heights. Based on Stable Diffusion, the area of image generation has attracted widespread attentions, which also promotes the research of following areas, such as image super-resolution<cit.><cit.> and image restoration<cit.><cit.>.

Although most recent models<cit.><cit.> focus on boosting their generative capabilities with natural languages, these models fail to control the exact texture and color palettes of generation images, when adding popular or customized styles (e.g., “impressionist”) to the input text prompt. A naive way is to fine-tune the T2I diffusion model using few images of the given “impressionist” style, whereas this manner cannot synthesize various images of different specific styles in a never-ending manner. For example, as shown in Fig. <ref>, when the user inputs a prompt such as “a cat wearing sunglasses in the style of impressionism" to generate an impressionist image of the cat, a naive way is to fine-tune the diffusion model with images in impressionist style. Subsequently, if the user tries to augment this diffusion model with a new style, e.g., “cubism”, a simple method is to fine-tune the diffusion model with both impressionistic and cubist images. As users continuously seek to incorporate new styles, the large computational requirements and ever-increasing training times swiftly become impractical. Moreover, there is a high probability for diffusion model to overfit to the new user-provided styles after continual fine-tuning operator, resulting in unsatisfactory artistic generation performance.   






Inspired by the aforementioned practical scenarios, we assume that the pre-trained large T2I diffusion model receives data of different customized styles in a streaming manner, as shown in Fig. <ref>. In the scenario of continuous style stream, we aim to establish a continual style customization method with T2I diffusion model, and maintain original generative capacity even after extensive fine-tuning. To achieve this, the T2I diffusion models need to extract pure style features rather than intricate image content, and solve the following key challenges:



  * “Catastrophic Overfitting”, i.e., the ability to learn pure features stylistic representation from the images provided from users, instead of influencing by the complex and intricate content of images. When learning a new style from a limited set of images, how to overcome the problem of overfitting to the specific contents of user-provided images is a common thought of style learning. For instance, if the diffusion model has learned an “impressionist” style, which contains a significant number of pictures with mountains in training data. When we input a prompt like, “a smiling man in impressionism”, the model may neglect or remove the concept of “man” and instead output a picture of mountains in impressionism. 


  * “Catastrophic Forgetting”, i.e., the knowledge of various styles obtained by diffusion model tends to be forgotten when incorporating with new styles. For this aspect, the personalized style knowledge forgetting learned from prior styles should be concerned. Boosting the ability to continually generate images of various styles without revisiting old data is a crucial consideration in the realm of continual style customization. For instance, the user is able to generate images of “impressionism” without accessing the training data, even after long periods of fine-tuning for new styles.












Inspired by the aforementioned considerations, we present a continual style learning approach based on T2I diffusion model in this work. Our proposed method aims to enable the continuous learning of new styles for generation purposes without accessing past datasets, which is designed to reduce memory consumption and compute efficiently. To address the two aforementioned challenges, we develop a continual style customization for diffusion model i.e, MuseumMaker, which could continuously adapt upcoming new styles and purely learn stylistic features from user-provided data. To tackle the catastrophic overfitting problem, we introduce a Style Distillation Loss (SDL) module, which distills the noise images generated from all images with the noise images from individual image to decouple the style and content of images. The SDL module enables the diffusion model to concentrate on acquiring the stylistic attributes of images while reducing the influence of specific content of images. To mitigate catastrophic forgetting, we propose a Dual Rregularization for shared-LoRA (DR-LoRA) module, which is designed to facilitate the smooth transfer of knowledge from old styles. A LoRA weight regularization and a noise graph regularization are devised to maintain the knowledge learned from old styles, which fully consider both aspects of weights and features. All the customization tasks share a single set of LoRA parameters with DR-LoRA module to minimize memory consumption. Additionally, to address the limitation posed by the limited parameters of LoRA, a Task-wise Token Learning (TTL) module is develops to learn a distinct token for each style, which enables generation of different styles. The tokens corresponding to past styles are stored to further mitigate the catastrophic forgetting problem. Finally, we demonstrate the validation of our MuseumMaker through extensive experiments, and conduct ablation studies to emphasize the contribution of each module in our MuseumMaker.

To summarise, the main contributions of this paper are as follows:


  * We take an earlier attempt to propose continual style customization with pre-trained large T2I diffusion model, i.e., MuseumMaker, which enables continual learning of various styles and effectively mitigates the catastrophic forgetting issue amongst past styles. 

  * To deal with the issue of catastrophic overfitting, we introduce a style distillation loss module to distill the style representation from the entire dataset into the latent representation generated from each image, which could overcome the problem of learning bias to the content of the images.

  * To address the catastrophic forgetting issue, we devise a dual regularization for shared-LoRA module and a task-wise token learning module, which could maintain the style knowledge from previous learned styles. Extensive experiments have confirmed the significant performance improvements and effectiveness of our proposed MuseumMaker.



We organize the rest of the paper as follows: the first section briefly introduces some related work. The second section revisits the text-to-image diffusion model and defines the problem of continual style learning for diffusion model. Then, we describe our methods in detail in third section. To the end, we conduct various experiments to evaluate our proposed method, followed by the conclusion.



§ RELATED WORK



 §.§ Continual Learning

The field of continual learning has attracted significant attention in recent years. Early approaches to continual learning focus on regularization-based methods, such as EWC<cit.> and SI<cit.>. These methods introduce additional regularization terms to penalize the changes of parameters, which are important for preserving previously acquired knowledge. Subsequent works have explored extensions and variations of these techniques, including strategies for better approximating parameter importance <cit.><cit.> and methods for considering the heterogeneous forgetting across different classes<cit.> <cit.>.

Another notable direction is experience replay, where a subset of past data is stored and replayed during training on new tasks to mitigate forgetting <cit.>. Variations of this approach include strategies for constructing and exploiting the memory buffer<cit.><cit.>, and leveraging generated data from generative models to augment or replace the replay buffer<cit.><cit.>.
Optimization-based approaches have also been explored, such as gradient projection methods<cit.><cit.> and meta-learning strategies<cit.><cit.>. These techniques aim to directly manipulate the optimization process or learn inductive biases that facilitate continual learning.
Architectural innovations also play a role in continual learning, with methods such as parameter isolation<cit.><cit.> , dynamic architectures<cit.><cit.>, and modular networks<cit.>.
More recently, there has been a growing interest in representation-based approaches that leverage category-guided learning<cit.> and large-scale pre-training<cit.><cit.> to obtain robust and transferable representation for continual learning. These methods seek to exploit the inherent advantages of self-supervised and pre-trained representation, such as improved generalization and robustness to catastrophic forgetting.

Despite significant progress, continual learning remains an active area of research. In this area, continual style learning for diffusion model is still a matter of concern, where various stylistic feature representation are hard to merge into a special diffusion model. Continual generation for different styles of images remains a challenge for diffusion model.




 §.§ Text to Image Generation

The emergence of diffusion models has ushered in a new era of text-to-image (T2I) generation, with state-of-the-art methods achieving unprecedented levels of photorealism and caption-image alignment. GLIDE<cit.> pioneers the application of diffusion models to this task, which adopts classifier-free guidance by replacing class labels with text prompts. Imagen <cit.> further improves upon this approach by leveraging pre-trained language models as text encoders, enabling the use of rich textual representation learned from large-scale corpora.

Other advanced works explore diffusion models in latent space, rather than operating directly on pixel space. Stable Diffusion<cit.> is trained with a large amount of data based on latent diffusion model (LDM)<cit.>, which employs a VQ-GAN<cit.> for latent representation and adds text as conditional information to the denoising process. DALL-E 2<cit.> inverts the CLIP<cit.> image encoder with a diffusion model, learning a prior to connect the text and the features of images in the latent space. This text-image latent prior is found to be crucial for performance.

Building upon these pioneering efforts, subsequent studies have sought to enhance T2I diffusion models in various ways. These improvements encompass model architectures<cit.>, spatial control via sketches or spatio-textual representation<cit.><cit.>, textual inversion for controlling novel concepts<cit.>, and retrieval mechanisms<cit.> to handle out-of-distribution scenarios. To further tackle the issue of continual learning for diffusion, <cit.><cit.> achieve the continuous generation of personalized concepts. However, the methods for continuous style learning and image generation are still lack of research.



 §.§ Image Style Learning

Capturing expressive representation of image styles is pivotal for achieving high-fidelity style learning. Early techniques like texture synthesis<cit.><cit.> and non-photorealistic rendering (NPR)<cit.><cit.> explore artistic expression through computational methods. However, these methods still face a constraint in transfer quality, universality, and feature extraction capabilities.





Generative Adversarial Networks (GANs) have emerged as a transformative paradigm for style representation learning. <cit.> proposes the DCGAN model to integrate a convolutional neural network into the original GAN framework, and adopts an encoder-decoder structure to bolster stability and generation quality. <cit.> introduces Wasserstein GAN (WGAN), leveraging the Earth Mover (EM) distance as a loss function to measure distributional discrepancies. With the recent rapid advancement of diffusion models, extensive methods for image style learning based on diffusion models have emerged. <cit.> treats the style information of images as trainable textual descriptions for model learning. <cit.> fine-tunes the U-Net architecture to enable the model to learn the stylistic information from images, and generates images in specific styles. However, these exciting style learning methods are unable to tackle a never-ending streaming manner of styles, which constrains the further application of diffusion model to generate images of various styles.




§ PRELIMINARIES



 §.§ Revisiting Text-to-Image Diffusion Model
 
The diffusion model <cit.> represents a probabilistic approach utilized for image generation, which generates a image by gradually denoising a noise graph from a Gaussian distribution. As exemplified by the Stable Diffusion (SD) technique <cit.>, the primary objective of the diffusion model is to align semantic information between natural language and images throughout the training phase. Specifically, SD relies on a pre-trained text encoder ψ from CLIP<cit.>, a latent encoder ℱ and a denoising U-Net ϵ_θ. Given a text prompt p, a timestep number T and a random Gaussian noise graph ε∈𝒩(0,𝐈), the text embedding can be computed as c = ψ(p). Formally, the training process of SD can be defined as follow:


    ℒ_SD=𝔼_z,c,ε,t[ε-ϵ_θ(z_t|c,t)_2^2],

where timestep t ∈Uniform(1,T), z_t is the image latents encoded by ℰ at timestep t, and θ represents the paremeters of denosing model.



 §.§ Problem Definition
 



Suppose that a series of continuous tasks for Text-to-Image (T2I) diffusion model are denoted as 𝐃^K={𝒟^1_n_k,𝒟^2_n_k,⋯,𝒟^K_n_k}, where K represents the number of consecutive tasks and n_k denotes the number of samples per task. Consequently, we can define all the tasks requiring generation as 𝒯 = {𝒯^k}_k=1^K. For the k-th learning task, 𝒯^k corresponds to a dataset 𝒟^k_n_k = {x^k_i,p_i^k}^n_k_i=1, where x^k_i ∈𝒳^k and p_i^k ∈𝒫^k denote the i-th image and its corresponding prompt. Different from existing works<cit.><cit.>, we consider a scenario that users provide consecutive images of different specific styles for T2I diffusion model, namely task 𝒯^k. For each task, the number of pairs of image x^k_i and prompt p^k_i with n_k are imposed into the continual style customization diffusion model, MuseumMaker. Considering the problem of privacy and memory limitation, the dataset of past tasks is unavailable. In this scenario, the model we proposed is able to incorporate the new style while ensuring the memory of past encountered styles. In details, the problem of continual style customization can be formulated as follows:

    ℒ^k_SD =1/ℬ∑_i=1^ℬ   𝔼_z_i^k,c_i^k,ε,t[ε-ϵ_θ^k(z_i,t^k|c_i^k)_2^2],
    ℒ_CSA   =∑_k=1^K(ℒ^k_SD+λℒ_c^k),

where ϵ denotes the denoising U-Net, θ represents its trainable parameters, and ℬ is the batch size during train process. (z_i,t^k,c_i^k) respectively represent the corresponding conditional image latents at timestep t and text embeddings on the k-th task, where c_i^k = ψ(p_i^k). ℒ^k_c is a loss proposed under the constraint of past prior knowledge, aiming at facilitating continuous learning of the model. Further explanation on ℒ^k_c will be provided in the subsequent sections.




§ OUR PROPOSED METHOD

In this section, we firstly present the overview of our proposed MuseumMaker in Sec. <ref>, which is illustrated in Fig. <ref>. In Sec. <ref>, we fristly present a style distillation loss to overcome the problem of catastrophic overfitting when encountering with a new style customization task. To tackle the catastrophic forgetting issue, we introduce a dual regularization for shared-LoRA module in Sec. <ref>, and a task-wise token learning module in Sec. <ref>. 



 §.§ Overview
 
As illustrated in Fig. <ref>, we develop a progressive continual style customization method with T2I diffusion model (i.e., MuseumMaker), which aims to retain knowledge of past learned styles while incorporating with new customized styles. To address the issue of catastrophic overfitting to image content, we devise a Style Distillation Loss (SDL) module for each new style, which distills the mean latent features across all images with individual image latent features to reduce the influence of image content during training. The direction of optimization is corrected to style of images by SDL, ensuring that the model focuses on style of images and alleviates overfitting to content. Considering the issue of catastrophic forgetting in continual style learning, we develop a Dual Rregularization for shared-LoRA (DR-LoRA) module and a Task-wise Token Learning (TTL) module. These two modules intend to transfer the knowledge from old model to current model and attain a unique token embedding for each specific style. Overall, MuseumMaker could offer a comprehensive solution for continual style customization.










 §.§ Style Distillation Loss
 
Given a reference style customization task with 8∼10 images, continual customized style learning aims to extract its artistic style information while explicitly removing the content information, which also emerges as a key area of focus in image style transfer <cit.><cit.>. One of feasible solutions is to explicitly defines the high-level features as content and the feature correlations (i.e., Gram matrix) as style <cit.>. However, this method relies on additional networks, which is not feasible enough to deploy to diffusion model and not available for continual style customization. Therefore, we introduce an easily deployable style distillation loss module, which addresses catastrophic overfitting and captures the pure style representations to prevent the disruption of original concepts in diffusion model.



To capture pure style representation from user-provided images and guide the model towards learning style features, we propose a straightforward yet highly implementable method in the continual style customization setting, i.e., Style Distillation Loss (SDL) module. Specifically, given a set of input images 𝒳^k={x^k_1,x^k_2,...x^k_n_k} on the k-th style customization task. We encode all images using an VAE <cit.> image encoder ℱ to obtain their feature representation 𝒵^k={z_i^k=ℱ(x_i^k)|x_i^k∈𝒳^k}, and then compute the mean of all feature representation as the representation of the image style of specific dataset. The latent features of images in the k-th task can be represented as follows:

    z̅_t^k=1/n_k∑_i=1^n_kℱ(x_i^k).

By utilizing both the mean feature and individual image features to train unique styles, we then obtain a distillation loss during the denoising process:

    ℒ_SDL = 1/ℬ∑_i=1^ℬρ(ℱ(x_i^k)/τ)log(ρ(ℱ(x_i^k)/τ)/ρ(z̅_t^k/τ)),

where τ is a temperature hyperparameter, and ρ represents the softmax function. Considering the Kullback-Leibler (KL) divergence loss between the mean latent features of all images and the latent features of individual images, SDL demonstrates its efficacy in reducing the probability of model overfitting to specific image content while learning image styles. This mechanism ensures the capacity of model to effectively concentrate on the diverse styles inherent in images.







 §.§ DR-LoRA
 
To rapid learn with extracted user-provided style from a small number of images, one of the popular strategies is fine-tuning T2I diffusion model with low-rank adaptation(LoRA) <cit.>, which has demonstrated its effectiveness in various text-to-image diffusion models <cit.><cit.>. Inspired by LoRA, we adopt a shared-LoRA strategy for continual customized styles learning in this work, and intend to significantly reduce the computational consumption usage. However, a wide and disparate distribution existing in style data stream poses a formidable hurdle in the continual style customization setting, when the diffusion model tries to preserve the style knowledge among pre-trained model. To tackle this issue, we devise a Dual Rregularization for this LoRA module (DR-LoRA) to the balance the learning of different styles, which simultaneously considers the knowledge transfer of style distribution from both weight manifold and feature representation manifold. 


Specifically, LoRA achieves fine-tuning of pre-trained large T2I models by freezing original weight while learning a pair of rank-decomposition matrices. Formally, this can be expressed as: 𝐖' = 𝐖 + Δ𝐖, where Δ𝐖 = 𝐀𝐁^T. LoRA effectively reduces the number of parameters required during training by fine-tuning matrices 𝐀 and 𝐁, and makes a swift adaptation to new styles. However, without constraining the optimization of diffusion model, the model above tends to learn incoming style data and further causes the catastrophic forgetting. Therefore, we consider the weight regularization for LoRA weight, in which we compute the offset of LoRA between adjacent tasks as a loss function to optimize the direction of model updating. Mathematically, we denote the weight manifold regularization of the k-th task as follows:

    0.47!
        ℒ_w   = 1/L∑_l=1^L[1 - Sim(Flatten(Δ𝐖_l^k-1), Flatten(Δ𝐖_l^k))],

where l represents the number of cross-attention layer of U-net, and Sim(.) denotes the cosine similarity calculation. To better calculate the similarity between past LoRA weight and current one, we flatten out the LoRA weight. We slow down the updating on key weights through ℒ_w, which is definitively crucial for T2I diffusion model to maintain the knowledge learned from previous styles.

While ℒ_w regularizes the diffusion model from the perspective of weight, it overlooks the forgetting of representation from past styles, which is an indispensable aspect to overcome catastrophic forgetting. Moreover, although shared-LoRA explores the task-shared global representation of styles, the task-specific representation for each customized style is neglected. We thus consider the unique knowledge of representation from each style, and introduce a feature representation regularization in shared-LoRA. Specifically, to transfer the style knowledge from past denoising model ϵ^k-1_θ to the current one ϵ^k_θ and well align feature representation, we input a set of past style prompts 𝒫^k-1 = {p_i^j}_j=1^k-1 into both past and current model. Subsequently, we can generate two noise graphs ẑ_i^j and ẑ_i^k from past and current denoising models, respectively. We consider noise graph ẑ_i^j generated from ϵ^k-1_θ as a pseudo noise graph for past encountered styles, which implicitly provides strong semantic guidance for current model to maintain prior knowledge. Therefore, we devise a loss function to transfer the knowledge of pseudo noise graph ẑ_i^k-1 to current denoising model ϵ^k_θ. Formally, the loss on the k-th style generation task can be computed as the follows:

    ℒ_f=1/K1/ℬ∑_j=1^K∑_i=1^ℬρ(ẑ_i^j/τ)log(ρ(ẑ_i^j/τ)/ρ(ẑ_i^k/τ)).
 

The knowledge from old styles can be transferred by ℒ_f, which regularizes the T2I diffusion model from the aspect of features. 

Overall, the dual regularization loss for shared-LoRA is:

    ℒ_DR=λ_1ℒ_w + λ_2ℒ_f,
 
where λ_1 and λ_2 are the hyperparameters to trade-off the regularization between LoRA weights and features. 




















 	







 §.§ Task-wise Token Learning
 
Deploying the above DR-LoRA module for continual style customization presents a promising method for sharing knowledge across diverse stylistic domains. However, when the T2I diffusion model encounters a massive continuous data stream with various styles, the DR-LoRA is limited to capture the distinct feature of each style. Additionally, as the diffusion model acquires different stylistic features, it may generate images intertwined with multiple styles, resulting in unsatisfactory generative performance. To address these challenges, we expend the model parameters of text embeddings minimally to enable the continual style learning. Furthermore, we devise unique token embeddings for each style to distinguish stylistic features from different style customization tasks. To be specific, prior works <cit.><cit.> have extensively explored token training, where a trainable token embedding can efficiently inject a special style to diffusion model for further images generation. Therefore, we develop a Task-wise Token Learning (TTL) module based on the textual inversion method. TTL module learns an independent token for each style customization task, and only stores the fine-tuned token parameters to maintain the prior knowledge, which take up extremely little memory. With a unique token corresponding to a distinctive style, the T2I diffusion model is able to capture unique features in a variety of styles. Formally, this module for the k-th task can be represented as:

    v^k_*=1/ℬ∑_i=1^ℬmin_v^*𝔼_z_i^k,p^k_i,c_i^k,ε,t[ε-ϵ_θ(z^k_t,t,c_θ(p_i^k))_2^2],

where v_*^k is the learnable token embedding, and p_i^k ∈𝒫^k denotes the user-provided conditioned style images of the k-th task. We combine the text description v^k_* with the conditioned stylistic images by Eq. (<ref>). The T2I diffusion model fuses the features of conditioned model, which can be triggered to generate specific style images when the corresponding v^k_* is input.

However, Textual Inversion <cit.> overlooks the independence of cross-attention layers at different resolutions during denoising process. Solely training the input token may not sufficiently unfold the embedding space. Inspired by this observation, we extend the input token embedding by introducing multiple independent tokens, which are learned across different cross-attention layers of the U-Net. We denote the extended v_*^k as 𝒱_*^k = {v^k_1, v^k_2, ..., v^k_L}, where k represents the corresponding task index, L represents the number of layers of cross-attention. The trainable tokens set of K tasks can be written as V_*^K = {𝒱_*^1, 𝒱_*^2, ... 𝒱_*^K }. Consequently, the Eq. (<ref>) for K tasks can be rewritten as:

    0.47!
        V_*^K=1/ℬ∑_k=1^K∑_i=1^ℬmin_v𝔼_z_i^k,p^k_i,c_i^k,ε,t[ε-ϵ_θ(z^k_i,t,t,c_θ(p_i^k))_2^2],

where each task obtains a extended token. We effectively extend the dimension of the token embedding space by training an independent token embedding for each cross-attention layer, which enhances the final generation quality. After training, we store a special token embedding for each task to further mitigate catastrophic forgetting during continual style customization. 



In summary, the overall pipeline optimization for our proposed MuseumMaker method is illustrated in Algorithm 1. The loss function constrained by prior knowledge to enable continual learning can be formulated as:

    ℒ_c=αℒ_SDL + βℒ_DR,

where α and β are manually set hyperparameters. The total optimization objective can be formulated as follows:

    ℒ_Overall=∑_k=1^K(ℒ^k_SD+ αℒ_SDL + βℒ_DR).


Regarding ℒ_Overall, our MuseumMaker systematically aggregates a continual collection of styles, empowering users to draw artworks of diverse styles and build their own customized museum.








§ EXPERIMENT



 §.§ Datasets and Evaluation

We present comparison experiments and ablation studies on Wikiart<cit.> to illustrate the efficacy of our MuseumMaker. Due to the particularity of imges generation, we utilize three kinds of metrics (i.e., style loss<cit.>, FID<cit.> and CLIP score<cit.>) to measure the superior performance of our proposed MuseumMaker.


Datasets: WikiArt consists of a diverse and rich artwork dataset of diferent styles, which encompasses artworks from 61 genres and localizes in 8 languages. The artworks from WikiArt is sourced from museums, universities, city halls, and other municipal buildings spanning over 100 countries.
For evaluation, we select 10 datasets representing different artistic styles in continual learning manner, as follow: (1) impressionism, (2) cubism, (3) realism, (4) ukiyo, (5) baroque, (6) rococo, (7) expressionism, (8) pop art, (9) renaissance, (10) pointillism. Each dataset comprises 8-10 images, showcasing distinct distributions of artistic styles. For ease of reference, we abbreviate each style to the first three letters.


















Evaluation:  To fairly evaluate the generative performance of models in a continual learning scenario, we leverage hundreds of images generated from 20 prompts for quantitative evaluation, with each prompt generating 50 sets of images. In order to evaluate the similarity of style between the generated images and the original style, we introduce style loss, FID and CLIP score. The details of these three metrics are as follow:

1) Style loss is employed in the field of image generation to assess the stylistic similarity between two images. We take the images from the training dataset as references and calculate the style similarity between these references and all generated images, subsequently averaging the results; 2) FID assesses the quality of generated images by quantifying the distance between source images and generated, which considers the distribution between both source data; 3) CLIP score utilizes the pre-trained ViT-B/32 model from CLIP to measure the property of diffusion model. By encoding both the generated images and reference images using CLIP, their similarity in the latent space can be computed. A higher similarity score indicates a better correspondence between the generated images and reference images.





 §.§ Implementation Details

In this work, we conduct continual style customization experiments on our proposed MuseumMaker, LoRA<cit.> and Specialist Diffusion (SPD)<cit.>. For the continual style customization experiment, we conduct the experiment on the 10 style datasets, ensuring that the previous style is not available during the training process. 
To assess the effectiveness of our continual style customization method in contrast to existing approaches, we deploy Fine-tuning (FT), as well as two classic continual learning methods, namely EWC <cit.> and LWF <cit.>, for LoRA and SPD. These methods serve as benchmarks for comparison with our approach.
Furthermore, we store a learnable token embedding and LoRA weight for each style as our upper bound. During the learning phase for each style, we employ 1000 training steps for LoRA baselines method, 100 training epoch for SPD baselines methods, 1000 training steps for ours. All the methods are implied with a batch size of 1. To achieve desirable generation performance, we set the learning rate to 1 × 10^-5 for LoRA baselines method, 1 × 10^-6 for SPD baselines methods and 1 × 10^-5 for our method. Additionally, the hyperparameters λ_1 and λ_2 are set 0.8 and 1.0 in Eq. (<ref>),  α and β in Eq. (<ref>) are set to 0.8 and 1.5. During the inference stage, we apply a DDPM sampler with 50 steps for all methods. All experiments are conducted on a single NVIDIA RTX A6000 GPU, with all generated images having a resolution of 512 × 512 pixels and based on Stable Diffusion v1.5.














 §.§ Comparison Experiments

As show in Table <ref> and Fig. <ref>, we present a comprehensive comparison of experiments conducted between our method and other competing approaches. To assess the efficacy of the continual style customization method proposed in this study, we first continually learn all 10 styles available in the Wikiart dataset. Subsequently, we demonstrate the results by generating each style using the same prompt. In order to comprehensively evaluate the performance of the generation, we consider the results of continual style customization from two aspects: qualitative evaluation and quantitative evaluation.

For qualitative evaluation, the results of experiments are shown in Fig. <ref>. The most competitive methods often suffer from catastrophic forgetting of previously encountered styles. For instance, approaches like LoRA combined with fine-tuning (LoRA+FT) and SPD combined with Fine-tuning (SPD+FT) struggle to preserve the ability to generate the textures and color schemes of previous styles. Simply applying the fine-tuning method with LoRA and SPD restricts the T2I diffusion model from retaining prior knowledge. We combine classical continual learning methods EWC and LWF with existing personalized T2I diffusion model to further compare with our method. The combinations such as LoRA+EWC, LoRA+LwF, SPD+EWC, and SPD+LwF vary degrees of forgetting regarding previously learned styles. For instance, the generated results of the pop art style and expressionism style markedly differ from the samples in the dataset. Moreover, the lack of distinct token embeddings for each style in these methods frequently results in confusion among similar styles. Specifically, combinations such as SPD+EWC and SPD+LwF tend to misclassify the baroque style as rococo style, while our method successfully generates accurate results. These outcomes successfully demonstrate that our method adeptly preserves prior knowledge, enabling the generation of a diverse range of styles while effectively distinguishing between similar styles. Our approach showcases a robust capacity to mitigate catastrophic forgetting and differentiate similar stylistic features. This demonstrates the effectiveness of our dual regularization approach, which operates across both the shared-LoRA module and the task-wise token learning module.



For quantitative evaluation, we conduct three metrics (i.e., style loss, FID, CLIP score) to measure the quality of generated images. 

As the result shown in Table. <ref>, Table. <ref> and Fig. <ref>, we compare the performance of all the competing methods in terms of style loss, FID and CLIP score, respectively. Our method achieves the best performance, exhibiting improvements ranging from 20.7 to 67.2 in terms of FID and from 0.96 to 11.77 in terms of CLIP score.  Such improvements demonstrate that our method reduce the catastrophic forgetting to old styles and catastrophic overfitting to new styles. In the realm of style loss, SPD+FT attains the highest performance, as it is directly tailored to optimize style loss. However, when it comes to FID and CLIP scores, it fails to demonstrate satisfactory performance. Our approach ranks second in terms of style loss performance. Compared to LoRA-based methods, SPD+LWF and SPD+EWC, our approach exhibits an improvement ranging from 0.002 to 0.058 in style loss. According to the above, our method attains the strongest ability to overcome the catastrophic forgetting and overfitting under the setting of continual style learning for text-to-image diffusion model.












 §.§ Ablation Studies

In the subsection, we conduct ablation studies on the continual learning of 10 styles, which encompasses five experimental settings: LoRA+FT, Ours w/o TTL & SDL, Ours w/o SDL, our complete proposed method and upper bound. For the upper bound setting, we store a unique token embedding and a special LoRA weight for each style. To ensure a fair comparison, all five settings employ the same hyperparameters and number of training steps. Furthermore, we define the same setup as in the comparison experiments to minimize the influence of randomness. Consistent with the comparison experiments, we perform both qualitative and quantitative analyses.




For the qualitative comparison, we evaluate the contribution of each module we proposed, which is depicted in Fig <ref>. We propose a dual regularization for shared-LoRA (DR-LoRA) module and a task-wise token learning (TTL) module to tackle the catastrophic forgetting. We eliminate these two modules one by one to evaluate their effect, and compare the result generated by ours w/o SDL, ours w/o TTL & SDL and LoRA based fine-tuning (LoRA+FT). As the first three rows of the output section in Fig. <ref>, the images generated by ours w/o SDL shows the best performance on images generation of past styles, which demonstrates the effectiveness of DR-LoRA and TTL module. Additionally, we devise a style distillation loss for addressing catastrophic overfitting. As shown in the last three rows of Fig. <ref>, the lack of the SDL module results in overfitting to the image content compared to our proposed method, adversely affecting the generation of pre-trained concepts. The results generated by our MuseumMaker exhibit a similar performance to the upper bound. To further measure the influence of SDL module under different weights, we design a experiments with varying weights, as shown in Fig. <ref>. When the diffusion model has a higher weight of SDL module, the model more effectively reduces the risk of overfitting the content of images. 






Regarding the quantitative comparison for ablation studies, we evaluate hundreds of generated images with  style loss, FID and CLIP score metrics. As shown in Table. <ref>,Table. <ref>  and Fig. <ref>, we compare ours, ours w/o SDL, our w/o TTL & SDL and LoRA+FT with three metrics to evaluate the effectiveness of each proposed module. Ours method achieves the best performance on two metrics (i.e., style loss and FID), which improve 0.007∼0.058 and 9.0∼66.9 in terms of styles loss and FID, respectively. As we propose DR-LoRA and TTL to tackle catastrophic forgetting, the setting of ours w/o SDL and our w/o TTL & SDL increase 0.015∼0.051 in terms of style loss, 1.8∼57.9 in terms of FID and 0.01∼13.61 in terms of CLIP score. The improvement across these three metrics highlights the effectiveness of the DR-LoRA module and TTL module in addressing catastrophic forgetting. However, the setting of ours w/o SDL achieves the best performance on CLIP score, and our method falls short by 1.84. Lower scores of ours w/o SDL in terms of style loss and FID reveal that ours w/o SDL slightly overfits to the content of images and learns impure represents of styles, which leads to a better adaptability to CLIP score. This phenomenon justifies the importance of SDL module to overcome catastrophic overfitting.





 §.§ Model Efficiency Study

To thoroughly evaluate the efficiency and effectiveness of our proposed MuseumMaker against other competitive methods, we detailedly analyze the training parameters and training time of comparison methods, as presented in Table.<ref>. While LoRA-based methods require a substantial number of parameters (39.06M) and training times ranging from 105 to 145 minutes, these methods show unsatisfactory generation results for continual style customization. Moreover, SPD-based methods utilize a much larger number of parameters (859.52M) and a relatively long training time, ranging from 170 to 290 minutes. Due to such a large amount of computing consumption, SPD-based methods are not efficient enough for continual style customization task. Our approach achieves comparable performance with significantly minimal parameters (39.06M) and a relatively short training time of 155 minutes. Considering the results of image generation, our approach stands out for achieving performance closest to the upper bound with minimal training parameters. These outcomes demonstrate the remarkable effectiveness of our DR-LoRA, task-wise token learning and style distillation loss techniques in the context of continuous style customization tasks, showcasing its potential for practical applications in real-world scenarios.




 §.§ Style Transfer Study

To evaluate the efficacy of our proposed continuous style customization approach in capturing the intricate nuances and diverse characteristics of styles continually provided by users, we devise its application in style transfer studies. This expansion not only broadens the scope of our method but also demonstrates its adaptability to diverse tasks in the field of computer vision. The exploration leverages the accumulated knowledge captured from the continuous stream of style data to facilitate the transfer of style from input images. As depicted in Fig.<ref>,  we observe that MuseumMaker adeptly captures the subtle textures, color themes, and structural elements after obtaining knowledge from 10 distinct styles. These results demonstrate the remarkable effectiveness of MuseumMaker in seamlessly learning a multitude of styles for both image generation and image style transfer, showcasing its versatility beyond the realm of text-to-image tasks.




§ CONCLUSION

This paper explores the challenge of continually learning new user-provided styles and generating a variety of stylistic images with a pre-trained diffusion model. Our proposed MuseumMaker efficiently embeds styles into diffusion model without overfitting to the content of images. Specially, we consider two critical aspects in developing our MuseumMaker, i.e., “catastrophic forgetting” and “catastrophic overfitting”. We devise a dual regularization for shared-LoRA module and a task-wise token learning module to address the “catasrtophic forgetting”. These modules help the model retain knowledge of previously encountered styles while adapting to new ones. Furthermore, we adopt a style distillation loss to tackle the “catastrophic overfitting”. This loss function ensures that the model learns pure representation of styles without being overly influenced by the content of images. Comprehensive experiments are conducted to show the effectiveness of our method. Results show that our method outperforms existing approaches in terms of style loss, FID and CLIP score, highlighting its ability to generate diverse and high-quality stylistic images.


















 
IEEEtran






