SynCellFactory: Generative Data Augmentation for Cell Tracking
Moritz Sturm Lorenzo Cerrone Fred A. Hamprecht

Robust Fine-tuning for Pre-trained 3D Point Cloud Models
    Cheng Jin
    
========================================================




 Cell tracking remains a pivotal yet challenging task in biomedical research. The full potential of deep learning for this purpose is often untapped due to the limited availability of comprehensive and varied training data sets. In this paper, we present SynCellFactory, a generative cell video augmentation.

At the heart of SynCellFactory lies the ControlNet architecture, which has been fine-tuned to synthesize cell imagery with photorealistic accuracy in style and motion patterns. This technique enables the creation of synthetic yet realistic cell videos that mirror the complexity of authentic microscopy time-lapses.

Our experiments demonstrate that SynCellFactory boosts the performance of well-established deep learning models for cell tracking, particularly when original training data is sparse.





§ INTRODUCTION


Digital time-lapse microscopy allows for large-scale observations of cells over time, providing a deeper understanding of cellular processes <cit.>.
However, to fully harness the potential of time-lapse imaging,  automated cell tracking approaches are needed, which can provide a quantitative analysis of cell behavior for vast amounts of data. 

Cell tracking is characterized by challenges such as variable image contrast, intricate behaviors such as cell division and, in some examples, indistinguishability of cells. Recent advancements in computer vision have shown that neural networks are highly effective in multi-object tracking tasks <cit.>. However, their application in cell tracking remains limited and exploratory, primarily due to the scarcity of annotated cell tracking data. This shortage hinders the development of robust neural network models for cell tracking <cit.>.


In recent years, although medium-scale annotated tracking data sets in the order of several thousand timeframes have become available <cit.>, they are limited to specific cell styles. However, given the vast diversity of cell types and imaging techniques, it is difficult for a single data set to cover all these variations comprehensively.
The reason for the lack of a broad range of annotated training cell tracking data is the costly expert annotation combined with cell tracking's niche status in the broader field of machine learning.

To address these challenges, this paper introduces SynCellFactory, a generative data augmentation strategy specifically designed for cell tracking. 
We embrace the power of conditioned 2D diffusion models <cit.> to generate high-quality synthetic cell videos that mimic the appearance and behavior of real cell data sets. 


Utilizing as little as one annotated cell video for training, SynCellFactory can generate an extensive library of annotated videos in a consistent style, effectively augmenting the available training data. This advancement holds the potential to transform cell tracking by enabling the application of sophisticated deep learning models previously hindered by data scarcity.
Our research focuses on evaluating the impact of this data augmentation on the performance of state-of-the-art cell tracking models. The results indicate that this novel approach addresses the data scarcity issue and enhances tracking accuracy. We aim to open the door for large-scale deep learning architectures in cell tracking. 



To summarize, our contributions are: 

    
  * SynCellFactory, a generative data augmentation pipeline for cell tracking. 
    
  * The proposed pipeline demonstrates out-of-the-box robust results on a wide variety of data sets without complex hyperparameter tuning or domain-specific knowledge. 
    
  * Empirical proof that the proposed data augmentation strategy can further enhance accuracy of an already leading deep learning cell tracking method. 










 §.§ Related Work


Generative Data Augmentation: Data augmentation is ubiquitous in deep learning and is known to help enhance the robustness and generalization of deep learning models <cit.>. 
Most data augmentation strategies in computer vision are based on simple transformations of the input data, including rotations, non-rigid deformations, color alteration, and cropping of images or videos. Recently, approaches based on generative machine learning are gaining traction <cit.>. The idea is to use a generative model to sample a new data point drawn from the estimated real data distribution. 
Until recently, Generative Adversarial Networks (GANs) were the primary choice for this task <cit.>. However, diffusion models advanced and eventually surpassed GANs in image synthesis quality <cit.>, making diffusion models the new state of the art for generating synthetic image data <cit.>.
Refs. <cit.> demonstrate the effectiveness of generative data augmentation using diffusion models in terms of achievable image classification accuracy. 
These advances also start to impact the medical imaging domain, where they show great promise  <cit.> and have already proven effective when used to train deep learning models <cit.>. The main motivation behind using generative data augmentation in medical computer vision is the scarcity of ground truth as well as privacy concerns.


Cell Synthesis: Several approaches to synthesizing artificial cell images have been proposed since the late 90s <cit.>. 
Early methods relied heavily on handcrafted parametric models. These models, informed by extensive biological knowledge, simulated the appearance and temporal evolution of cells <cit.>.

However, with the evolution of deep learning, an effort has been made to reduce the domain-specific knowledge required to generate realistic cell video.
While <cit.> combines statistical shape and evolution models with conditional GANs to generate 2D+t data, <cit.> employs video diffusion models to create full cell videos directly from the training set.
Most existing models focus on generating a single cell in either 2D or 3D <cit.>. To simulate a cell population, some methods first model single cell appearance and temporal development and then incorporate simultaneous evolution of multiple cells to model a population of cells <cit.>. Our proposed method is based on a similar premise to model cell appearance and temporal development separately, but distinguishes itself by modeling the image of an entire cell population jointly.

Closely related to our work, <cit.> generates an entire video at once using a 3D diffusion model guided by optical flow; crucially, their approach does not produce pseudo ground truth labels, while SynCellFactory does. Ref. <cit.> proposes a CycleGAN <cit.> based model which can also generate raw data as well as lineage and segmentation pesudo ground truth. An important difference is that said work relies on simulated ground truth segmentation masks to condition the data generation; this is a much harder problem to solve than the simple trajectory modeling employed here. 




§ METHOD

SynCellFactory operates on the principle of decoupling cell dynamics from their appearance. By controlling these two aspects independently, our model can generate highly heterogeneous annotated cell videos.
Our model comprises two principal components:


  * A simple 2D motion model that simulates the spatial distribution and dynamics of cell cultures. It uses statistical parameters derived from real data, enabling SynCellFactory to produce coherent and physically plausible timelapses, irrespective of frame count. This simulation approach enhances the diversity and realism of the generated data.

  * Two distinct ControlNets, each with a specific function, are trained for photorealistic rendering. The first, CN-Pos, is adept at inpainting cells with lifelike appearance at accurate spatial positions. The second, CN-Mov, focuses on the temporal displacement of individual cells across consecutive frames, ensuring temporal consistency. Particularly crucial for our use case in data augmentation, these ControlNets demonstrate efficient training capabilities, even with limited data such as a single time-lapse sequence, making them well suited for augmenting data sets in cell tracking applications.


In the following sections, we will detail the methodology of our model and introduce an automated protocol for training SynCellFactory on new data sets.



 §.§ Motion Model


Cell movement under a microscope is a process influenced by various factors like cell type, environmental conditions, and cell-to-cell interactions. Biologically accurate modeling of the cell behavior is outside the scope of this manuscript. Therefore, we are not implementing intra-cellular forces and complex behaviors such as amoeboid, ciliary, and flagellar motions. The goal of our motion model is to generate plausible spatial cell configuration and displacement, leaving the heavy lifting of visually representing cell interactions to the generative neural network. 

Our engine represents cells as 2D disks, capable of moving, splitting, and interacting with one another.
The motion model initializes a population of cells with randomly sampled positions and sizes, drawn from the distribution observed in real cells from our training data. This population dynamically evolves following a stochastic Brownian motion, guided by statistics extracted from annotated real cell videos. These key parameters include the mean and standard deviation of cell area (A_c and σ_A) and the cell displacement distribution, which we model using a gamma distribution G(α_d,Θ_d). These statistical models are derived from available segmentation and tracking ground truth data.
A significant advantage of our approach is its flexibility in adjusting the tracking difficulty of the generated data sets. By manipulating variables such as the number of cells, their movement speed, and the frequency of cell splitting events, we can tailor the complexity of the tracking task. This allows us to create data sets that either exceed or are comparable in complexity to the original data, accommodating a range of research and application needs.
Details on the statistical methods, the specific equations governing cell movement, and the pseudocode for the simulation are provided in supplementary Sec. A. 



 §.§ ControlNet

ControlNet <cit.> is a popular architecture for enabling the conditioning of text-to-image generative models like stable diffusion <cit.>.

Our ControlNet uses the standard pre-trained Stable Diffusion v1.5 backbone trained on natural images. In contrast to the original ControlNet model that maintains the stable diffusion decoder locked, we fine-tune it to the appearance of biological images. 

Morever, training a ControlNet in the context of latent text-to-image diffusion models involves triplets of the form (c_txt,c_img, i_tgt), where c_txt is a conditioning text, c_img is the  conditioning image and i_tgt is the target image.
The text conditioning has been kept fixed for all our experiments to the prompt “cell, microscopy, image” to provide a good initialization during the early stages of training. 

In the following sections, we describe the specific details and  c_img conditioning the CN-Pos and CN-Move models. 





  §.§.§ Positional ControlNet


The positional ControlNet CN-Pos is tasked with drawing realistic looking cells conditioned on a given position (see <ref>). 
This model plays two fundamental roles in SynCellFactory: it is used to generate the last frame of our synthetic videos and is the pre-trained backbone of the CN-Move model.
At train time, the position map is constructed by computing the center coordinates for each cell given the corresponding detection ground truth (see <ref>). Each detection is represented in the conditioning image c_img as a disk with a fixed radius of r=√(A_c/π)/4, where A_c is the data set average cell area in pixels. 
Fixing the radii of the disks instead of varying them proportionally to the cell size empirically improved the diversity of cell appearances.
Moreover, the disks are colored according to the stage in the cell cycle. The disk is green while the cell is in the interphase, linearly changing between green and blue during the Mitosis phases (Prophase, Metaphase, Anaphase, and Telophase) and linearly reverting from blue to green after the cytokinesis phase and the split event is complete. The cycle length is a hyperparameter that needs to be set according to the specific cell type for each data set.
At inference time, the conditioning image c_img is derived by the state simulated by the motion model, where the center of the simulated cells are converted into color-coded disks. 



  §.§.§ Movement ControlNet

CN-Mov is analogous to CN-Pos in its implementation details; the sole difference lies in its conditioning. 

CN-Mov is tasked with predicting frame t-1 conditioned on the frame at time t, the position map at time t-1, and the displacement vectors.
Frame t is encoded in the red channel of c_img, while the position map at time t-1 is encoded in the blue and green channels as described in <ref>. 
Displacement vectors describe the movement of the cells between frames and are encoded as lines connecting the center position of cells. A more detailed description of the conditionings can be found in supplementary Sec. B.




 §.§ SynCellFactory Inference

During the sampling process, we apply the trained ControlNets on the output of our motion module to generate realistic-looking videos (see Fig. <ref>).  
The choice to sample our video in reverse time is arbitrary, but simplifies network learning: in backward time, cell merge events require the network to predict only a single position from two merging cells. Conversely, in forward time, the network would face the more complex task of generating two new cells from one during a split event.

The inference process is initiated with CN-Pos generating frame t. Subsequently, CN-Mov iteratively takes the generated frame t and, in conjunction with the motion model, samples a new frame t-1. This iterative process continues until the desired video length is reached.
In all our experiments, we generated a fixed length of 12 frames.



 §.§ Segmentation Pseudo Ground Truth


SynCellFactory produces only raw video, detection, and lineage ground truth, but critically is not able to produce instance segmentation ground truth. This is a significant challenge, as several deep learning methods <cit.> require full segmentation for effective training. To address this, we rely on Cellpose <cit.>, a renowned deep learning framework for cell segmentation, to create pseudo-ground truth segmentation and thereby mitigate this limitation.
Cellpose is particularly effective due to its range of pretrained models that cater to various cell image styles, boasting high segmentation accuracy right out of the box. For our cell data sets, we chose the pretrained model recommended by Cellpose, and further finetuned it for 100 epochs using ground truth segmentation masks from our training data. This finetuning allows the model to more closely adapt to the unique characteristics of a particular data set.
While the segmentation labels produced by Cellpose are mostly accurate, they occasionally miss or hallucinate cells. To improve upon this, we integrated the motion model into the segmentation process. In cases where a segmentation mask does not overlap with any part of the generated detection ground truth, it is removed. Conversely, when a ground truth detection is present without a corresponding segmentation mask, we generate one by drawing a circle with a radius r=√(A_c/π). This step is crucial to ensure that small inaccuracies or hallucination artifacts produced by the ControlNet do not receive segmentation masks, while cells that are challenging to segment are still provided with an approximate mask. 
See supplementary Sec. C for an illustrative example of the correction step.




 §.§ Automated Training

To reduce the domain-specific expertise required to train SynCellFactory, we propose a fully automated pipeline for training the ControlNets and the sampling from the motion module. 

The only required hyperparameters that have to be manually specified to produce our synthetic videos are: the number of videos to be generated, the number of frames per video, and the characteristic length of the mitosis cycle. 

Other parameters,such as movement statistics for the motion model, are automatically inferred from the raw data and ground truth annotations. One of the main challenges with training the ControlNet is defining a good stopping criterion. We found that in our applications, the number of required steps to convergence is correlated with the average cell number in each frame. Thus, we can automatically set an appropriate number of training iterations for our ControlNets (see supplementary Sec. D)




§ EXPERIMENTS AND RESULTS



 §.§ Data sets

To validate the proposed SynCellFactory we use the publicly available Cell Tracking Challenge (CTC) <cit.> [<http://celltrackingchallenge.net/2d-datasets/>] data sets.
The CTC is an open benchmark to test new cell tracking and segmentation advancements on data sets with a large variety of cell types, microscopy modalities, and different styles, as well as inhomogeneous spatial and temporal resolution.

For our experiments, we focus on the seven 2D data sets enumerated in <ref>.

Each data set consists of two timelapses with full tracking annotations of ground truth and only partial hand-curated segmentation. The tracking ground truth includes detection and identity masks for each frame and the corresponding cell lineages.
The number of frames per video ranges from 30 (Fluo-C2DL-Huh7) to 300 (PhC-C2DL-PSC).

Our experiments used a single time-lapse for the training and validation and one for testing tracking accuracy. 
Sample still frames from the mentioned data sets can be found in <ref>.
We have elided from the pool of data sets for our experiments Fluo-N2DH-SIM+ because it is an already simulated cell video <cit.>. We excluded BF-C2DL-HSC and BF-C2DL-MuSC because the boundary conditions of the petri dish could not be reproduced by our motion model and because these data sets already provide several thousand annotated frames, rendering augmentation less relevant.  




 §.§ Experimental Setup

The training of CN-Pos and CN-Move follows the standard procedure as presented in <cit.>; the only major difference is that we also finetune the stable diffusion UNet decoder block. In the original ControlNet architecture, this module is fixed at train time. 
A detailed diagram of the training computation is shown in <ref>.

Since we could only access a single annotated timelaps for training our ControlNet, we relied on data augmentation. In particular, we found random cropping of the images and random 90 degree rotations beneficial. The training process is highly resource-intensive, roughly 20 hours on a Nvidia A100 with 40GB of VRAM. A more detailed explanation of the ControlNet training protocol is reported in supplementary Sec. D.

To evaluate the usefulness of SynCellFactory as a data augmentation strategy, we used it in conjunction with the state-of-the-art deep learning model EmbedTrack <cit.>.
EmbedTrack uses CNNs to predict cell segmentation and tracking jointly. 
For training EmbedTrack, we use the default settings suggested by the authors; in particular, for each experiment, we trained the model for 15 epochs using the CyclingAdam optimizer <cit.>. For testing, we used the best-performing model with respect to the validation intersection over union.



  §.§.§ Tracking Metric

To evaluate the tracking performance of our trained models, we use the official tracking accuracy measure (TRA) provided by the Cell Tracking Challenge <cit.>.

The TRA score is based on the concept of Acyclic Oriented Graph Matching AOGM. Suppose we represent the tracking predictions as an acyclic-oriented graph. In that case, we can measure the accuracy of a predicted graph by counting the number of operations required to transform the prediction into the ground truth graph.
The operations considered are node insertion, node deletion, edge insertion, and edge deletion. 
Formally, the TRA score is defined as:

    TRA = 1 - min( AOGM, AOGM_0 )/AOGM_0,

where AOGM represents the weighted sum of operations required to build the prediction graph and AOGM_0 represents the weighted sum of operations required to build the ground truth graph. Higher TRA scores indicate better tracking performance.













 §.§ Quantitative results

In all but one of the test data sets, our SynCellFactory data augmentation improved the tracking quality as measured by the TRA score; the results can be found in <ref>.
As shown in previous work using generative models for data augmentation <cit.>, the ratio between synthetic and real data

    α = #syn-frames/#syn-frames + #real-frames

in the training set is crucial.
This is also true for the proposed methods; in  <ref>, we show the tracking accuracy achieved using different mixing ratios α.
Our experiments showed two behaviors between fluorescence microscopy data sets (denominated as Fluo-*) and all other microscopy modalities. 
In fluorescence datasets, TRA scores typically increased with the mixing ratio up to an optimal 0.5 < α < 0.7, then dropped, except in one dataset where improvement continued steadily. In Phase Contrast (PhC-*) and Differential Interference Contrast (DIC-*) experiments, most datasets initially showed a performance drop at low α, followed by consistent improvement at high α∼ 0.8, with one exhibiting an initial increase, a subsequent drop, and then improvement.
In addition to the TRA score, we also report additional performance metrics in  <ref> based on the number of correct tracking predictions.



  §.§.§ CTC Results

In addition to our standard experimental setup, we tested our strategy on the official cell tracking challenge evaluation data set. The CTC organizers evaluate the submissions on a private ground truth. We submitted the results for three data sets. Here, we trained SynCellFactory and the EmbedTrack model on all available training data and using the optimal mixing ratio α. The results are presented in <ref> and show an improvement on all three data sets compared to those in <cit.>. 



 §.§ Qualitative Results

Overall, the results for all tested data sets look realistic from the human perspective. We now present a summary of the qualitative characteristics we observed in our generated images.

Mitosis: One of the most challenging aspects of automatizing cell tracking is correctly identifying cell splitting. The process by which cells replicate is called mitosis, and most cell types exhibit a drastic change in their appearance during this stage. This process happens in subsequent stages (typically classified into Prophase, Metaphase, Anaphase, and Telophase), each with its distinct visual appearance.
It is, therefore, important to correctly mimic the appearance of cells during mitosis in case visual deep learning models are trained on the generated data sets.
For our data sets, DIC-C2DH-HeLa is the data set with the strongest visual indication of a splitting event. In <ref>, we compare generated and real HeLa cells undergoing mitosis from the DIC-C2DH data set.


Hallucinations: Hallucinations of cells and other artifacts are a weak point of the proposed pipeline (see <ref> for an example). We observe two main types of hallucinations in the generated data sets.
The first mode of hallucination is the appearance of additional cells near image borders. These cells are usually inpainted from the beginning of the video and remain stationary throughout the video. 
We hypothesize that these hallucinations are derived from the lack of cell detections at the border in the CTC data sets. Indeed, cells partially cut out of the field of view are often not included in the ground truth.
The second type of hallucination occurs more sporadically and consists of artifacts and debris in the generated frames seldomly promoted to cells by our ControlNet. We observed such hallucinations more prevalently in data sets with much background noise, like Fluo-C2DL-MSC and PhC-C2DH-U373. 





§ CONCLUSION

Despite the positive results, SynCellFactory is not without limitations, which also serve as directions for future work.

The current motion module in SynCellFactory is simplistic, focusing on basic cell movements. This limits its applicability in capturing the full spectrum of biological diversity and complexity. Future iterations of SynCellFactory could benefit from a more sophisticated motion module that can accurately model a broader range of biological behaviors and interactions.
Moreover, the model's performance declines in scenarios with high cell density or dramatic changes in cell count. Addressing this limitation would enhance the model's utility in more complex biological environments.
While SynCellFactory does not produce segmentation ground truth, the availability of advanced segmentation tools partially mitigates this issue. However, it remains a potential source of inaccuracies and could be an area for future integration.
SynCellFactory can sample videos of arbitrary length, but there is a noticeable drop in quality for extended sequences; in our experiments, we found that after 30 frames, the quality is reduced to the point of losing most resemblance with the original data. We hypothesize that this is due to the training procedure, in which the CN-Mov is only trained on frames pair and real data conditioning. Future developments could focus on maintaining consistent quality across varying video lengths.

In conclusion, SynCellFactory represents a significant step forward in the field of biological data augmentation. Its ability to generate realistic and diverse data sets holds great potential for advancing deep learning-based cell tracking pipelines. While there are several areas for improvement, the foundation laid by SynCellFactory offers a solid base for the concrete application of generative AI in biomedical computer vision.



    ieeenat_fullname
    









§ MOTION MODEL



 §.§ Statistics

In our study, we derive the motion module's key statistics from both the tracking ground truth and a limited set of segmentation ground truth data.

We use all segmentation masks available within a single data set to calculate the mean A_c and standard deviation σ_A of pixel counts per segmentation mask.

We estimate our cell displacement distribution given the tracking ground truth by computing the displacements of individual cells between consecutive timeframes. The distribution of all calculated displacements is then fitted using a gamma distribution G with respect to its parameters α_d, Θ_d:

    G(x; α_d, Θ_d) = x^α_d-1· e^-x/Θ_d/Θ_d^α_d·Γ(α_d)

Here, α_d represents the shape parameter, Θ_d the scale parameter, and Γ(α_d) denotes the gamma function. The choice of a gamma distribution is empirically driven by its adaptability in modeling data with skewed distributions.

We can generate new, plausible samples of cell population timelapses with these statistical parameters determined. These samples take into account variations in both cell area and motion characteristics.



 §.§ Simulation

Initialisation: 
In the first timeframe we sample n cell centers at random positions (x_i,y_i) ∈ (0,1) for i ∈{1, 2, …, n} with each cell having an area A_i ∼𝒩(A_c,σ_A/10). We model each cell as a 2D disk with radius r_i=√(A_i/π).

Movement of cells:
In our simulation, cell movements are modeled as a random walk. 

Let 𝐩_i, t = (x_i, t, y_i, t) represent the position vector of cell i at timeframe t. The position at the next timeframe t + 1 is updated using:

    ϕ   ∼Uniform(0,2π) 
    
    m_v    ∼ G(α_d, Θ_d) 
    𝐩_i, t+1   = 𝐩_i, t + m_v 
    [ cos(ϕ); sin(ϕ) ]



Each cell has a splitting probability p_split at each timeframe. Upon splitting, a cell divides into two daughter cells along a random axis. The position vectors of the daughter cells, denoted as 𝐩_i,1 and 𝐩_i,2, are given by:

    𝐩_i,1/2 = 𝐩_i ± d · r_i 
    [ cos(ϕ); sin(ϕ) ]




Here, d ∼Uniform(0.7,0.8) introduces a slight randomness in the splitting process, and r_i is a scaling factor.

Resolving overlaps:
The resolution of overlaps in our model is repeated in successive iterations, until no overlaps remain.

To resolve the overlap of two circles, we first define the repulsion vector 𝐯_repulsion between two overlapping circles c_i and c_j with center coordinates (x_i,y_i) and (x_j,y_j) as follows:

    𝐯_repulsion = [ x_i - x_j; y_i - y_j ]


The angle of repulsion θ is calculated using the arctan function:

    θ = arctan((𝐯_repulsion)_y, (𝐯_repulsion)_x) + θ_ϵ,

where θ_ϵ∼Uniform(-0.1, 0.1) is a small perturbation to the angle to prevent infinite loops during the overlap resolution process.     

Next, we calculate the repulsion displacement F_i,j, based on the overlap distance between the circles, scaled by the average of their radii to account for potentially larger overlap areas in bigger circles. The formula is given as:

    F_i,j = (( r_i + r_j - 𝐯_repulsion) ·r_i + r_j/2) ·(1 + ϵ)

where ϵ∼Uniform(0,0.2) is introduced in each iteration with the same reasoning as for θ_ϵ.

The positions of the cells are then updated as follows:

    𝐩_i   = 𝐩_i + F_i,j·[ cos(θ); sin(θ) ]
    𝐩_j   = 𝐩_j - F_i,j·[ cos(θ); sin(θ) ]













§ CONTROLNET CONDITIONING

In <ref> we showcase an example of the construction of position and movement maps for training. 

 




§ SEGMENTATION PSEUDO GROUND TRUTH

The segmentation correction step is illustrated in <ref>.



§ CONTROLNET TRAINING

To address the challenges of limited training data we found it beneficial to choose a sequential training procedure, training in total four consecutive ControlNets per dataset.


We start our training procedure by training a base model (BM) of CN-pos on randomly selected patches of c_img and i_tgt measuring h/2 × w/2. The initial phase focuses on enabling ControlNet to learn the correlation between position maps and cell appearance.

After a fixed number of iteration steps (see <ref>) the training procedure of CN-pos is stopped and we start training an initial version of CN-mov. This phase involves using cropped movement maps and corresponding target images of size h/2 × w/2, similarly to the base model of CN-pos.

CN-mov is initialized with the pre-trained weights from the CN-pos base model, ensuring continuity in the learning process.

The final step in our training methodology involves fine-tuning both base model ControlNets. This is achieved through limited optimization steps conducted on the full-sized h × w training timeframes. After fine-tuning, the full model (FM) ControlNets are optimized and ready to integrate into the sampling process.


In <ref> is a list of training parameters used to train our ControlNets. Numbers for one A100 40 GB GPU.  









§ ADDITIONAL GENERATED TIME-LAPSE

In <ref> - <ref>, we showcase more examples of generated time-lapses and the corresponding tracking ground truth as simulated by the motion module. Generated time-frames are shown in the top row, and tracking ground truth is shown in the bottom row. 

























