{'fraction_non_alphanumeric': 0.03250350890152914, 'fraction_numerical': 0.012964467755041738, 'mean_word_length': 4.775383959044369, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Dataset Condensation with Latent Space Knowledge Factorization and Sharing

In this paper, we introduce a novel approach for systematically solving dataset condensation problem in an efficient manner by exploiting the regularity in a given dataset. Instead of condensing the dataset directly in the original input space, we assume a generative process of the dataset with a set of learnable codes defined in a compact latent space followed by a set of tiny decoders which maps them differently to the original input space. By combining different codes and decoders interchangeably, we can dramatically increase the number of synthetic examples with essentially the same parameter count, because the latent space is much lower dimensional and since we can assume as many decoders as necessary to capture different styles represented in the dataset with negligible cost. Such knowledge factorization allows efficient sharing of information between synthetic examples in a systematic way, providing far better trade-off between compression ratio and quality of the generated examples. We experimentally show that our method achieves new state-of-the-art records by significant margins on various benchmark datasets such as SVHN, CIFAR10, CIFAR100, and TinyImageNet.


INTRODUCTION
Deep learning has been successful in numerous machine learning problems thanks to the recent progress in parallel processing and the huge amount of real-world data collected from various sources. However, in some machine learning applications it is required to rehearse the training process repeatedly, such as hyperparameter optimization (Bengio, 2000;Franceschi et al., 2017), neural architecture search (Liu et al., 2018) and continual learning (Lopez-Paz & Ranzato, 2017). In those cases, it is prohibitive to keep and rehearse all the examples in huge datasets, giving rise to the need for compressing each dataset into a small set of representative examples. The conventional approaches resort to selecting a coreset (Phillips, 2016;Toneva et al., 2018;Borsos et al., 2020), but their assumption is limited as there may not exist strongly representative examples in the original dataset  and also suffer from the difficulties of combinatorial optimizations (Borsos et al., 2020). Recently, more popular approaches focus on dataset condensation (Wang et al., 2018; which directly parameterize and optimize the synthetic dataset with gradient descent. They have shown to perform well compared to coreset selection approaches and generate plausibly looking examples as well. However, despite its great potential to other adjacent fields, most of the existing dataset condensation methodologies do not focus on exploiting the regularity in the dataset, such as what the underlying data generating process would be. They usually parameterize a set of synthetic examples directly in input space and minimize a distance between real and synthetic dataset in either parameter  or feature space . Such lack of assumptions on the underlying data generating process prevents them from efficiently sharing knowledge among the synthetic examples. Therefore, those approaches may require much larger space consumption to capture the same amount of information compared to when we can properly organize the data generating process. This intuition gives rise to questions about whether we can further improve the efficiency in solving the problem by exploiting the regularity in the given dataset. To this end, we naturally assume that each datapoint is generated from a code in a compact latent space followed by a sharable mapping function from that space to the original input space (i.e. decoder), both of which are learned in an end-to-end manner. In this way, we can dramatically increase the number of generated synthetic examples with the same parameter count as the latent space can be much lower dimensional than the original input space, and also since we can assume as many decoders as appropriate to capturing various styles represented in the dataset. See Figure 1 for the concept. Note that the additional per-class parameter count introduced by the decoders is negligible because those decoders are very tiny and can be shared across all the synthetic examples and classes. Furthermore, the knowledge is shared among the generated synthetic examples through the shared latent space and the set of shared decoders applicable to any latent codes interchangeably, dramatically improving the trade-off between compression ratio and quality of the synthetic dataset. To our knowledge, this paper is first to successfully employ such latent space approach to systematically solving the dataset condensation problem.
In algorithmic side, we show that the previous approaches that subsample synthetic examples for computational efficiency (e.g. gradient matching  or distribution matching ) actually produce biased gradients, preventing the synthetic examples from being diversified. We further experimentally observe that subsampling of the real examples produce highvariance gradients, degrading the quality of the generated synthetic examples. Those observations indicate that we have to perform full batch training for both real and synthetic dataset. We thus adopt distribution matching  as it does not require expensive second order computations unlike gradient matching . We also prestore and reuse some of the information needed to perform distribution matching for further improving the training efficiency.
We experimentally show that our method, which we name as Knowledge Factorization and Sharing (KFS), achieves new state-of-the-art records on various benchmark datasets such as SVHN, CI-FAR10, and CIFAR100 and TinyImageNet with different amount of parameter count allowed (e.g. 1, 10, or 50 images per class). We also show that the synthetic examples generated from KFS produce decent performances on other network architectures than what they have been trained on (e.g. from ConvNet-3 to DenseNet-121), and it provides superior performance over various amount of training budget at evaluation time without overfitting. We summarize the contribution of this paper as follows: • We introduce KFS, a novel approach for solving dataset condensation problem that factorizes the synthetic data generating process into a set of learnable latent codes and decoders, dramatically increasing the number of synthetic examples with a limited parameter count. • We show that the assumed generative process effectively allows knowledge sharing between the synthetic examples such that they achieve far better trade-off between the compression ratio versus the quality of the generated examples. • We further show that the existing dataset condensation approaches suffer from the issue of biased and high-variance gradient, and experimentally verify that full batch training significantly improves the quality of the synthetic dataset. • The proposed KFS achieves new state-of-the-art records in all datasets we considered, outperforming the previous state-of-the-art by significant margins.

RELATED WORK
Dataset condensation. Wang et al. (2018) firstly introduced a differentiable approach to dataset condensation problem through a bilevel optimization formulation. However, in their work only a few inner-gradient steps are assumed for computational efficiency and additionally a shared initialization is learned and entangled with the learned data.  overcome the limitation by trying to match the network weights and approximating the expensive bi-level optimization with shorthorizon approximation, resulting in gradient matching between real and synthetic examples for each step.  further develop the idea by learning a differentiable siamese networks for augmenting the synthetic examples. Cazenavette et al. (2022) recently propose to match much longer segments of learning trajectories, yielding competitive performances by guiding networks to a more similar state. On the other hand, instead of matching the network weights,  match distributions of real and synthetic dataset in a feature space. Although they sometimes show infereior performance to gradient matching, their approach is computationally more efficient as they do not require to compute expensive second-order derivatives such as vector-Jacobian products (VJPs) required by gradient matching. Simiilarly,  propose to layer-wisely align features with auxiliary discriminative loss and bi-level formulation for adjusting parameter updates.
On the other hand, Nguyen et al. (2020; propose to make use of the connection between infinitely wide neural networks and kernel ridge regression, but their approach requires hundreds of GPUs for training. Perhaps the most relevant to our work is recently proposed IDC (Kim et al., 2022). They generate various synthetic data from a single set of condensed examples by segmenting and upscaling the condensed examples. Their approach can be seen as exploiting the regularity in the dataset, for instance, spatially nearby pixels look similar to each other. Although they achieve strong performances on many datasets, it is questionable whether the proposed generative process is sophisticated enough to fully exploit the regularity in the dataset. In this work, we make a more natural and general assumption on the data generating process, yielding new state-of-the-art records on all the datasets we consider.
Generative models. Our work is closely related to generative models such as VAE (Kingma & Welling, 2013) and GAN (Goodfellow et al., 2014). They maximize the likelihood of data by learning a compact latent space and a shared decoder, similarly to our work. However, it is questionable whether maximizing the data likelihood would be sufficient for the target task such as classification. Also, whereas it has been shown that modality is crucial for the performance of condensed dataset (Kim et al., 2022), they are known to suffer from the posterior collapse (or mode collapse) problem (Razavi et al., 2019a). Van Den Oord et al. (2017) and Razavi et al. (2019b) overcome the issue by proposing VQ-VAE that models discrete representations in the latent space. The discreteness of the approach makes them closely related to the dataset condensation, but it is questionable whether such generative models can outperform the dataset condensation approaches recently introduced. For instance,  combine GAN with meta-learning to generate synthetic examples, but they only slightly outperform Wang et al. (2018).
Object files and schemata. Our approach is also closely related to object files and schemata discussed in Goyal & Bengio (2020); Goyal et al. (2021b;a). Object files typically refer to the state of each object (e.g. states in RL or an object in an image) and schemata determine how they behave and update the object files correspondingly (e.g. actions in RL or styles in images). By carefully factorizing those two different types of knowledge, the knowledge structure becomes more efficient as we can apply the same schema to various objects, and more effectively generalizable to out-ofdistribution because the architecture has better systematicity. Also, the interactions between object files and schemata are inherently sparse, i.e. in real-world scenarios, explaining a phenomenon usually requires only a few rules given a set of objects. In our case, a single pair of object file and schema is assumed to be sufficient to generate a single synthetic image.

APPROACH
We next present our approach, Knowledge Factorization and Sharing (KFS), that efficiently factorize the data generating process into latent codes and decoders so that knowledge is effectively shared among the synthetic examples generated.

KNOWLEDGE FACTORIZATION AND SHARING IN LATENT SPACE
It was observed by Kim et al. (2022) that what determines the performance of condensed dataset is its modality rather than resolution. We follow the same principle but we want to find even better trade-off between increasing the modality of the dataset and preserving or even improving the quality of generated images.
Latent code -decoder factorization. Our approach starts from assuming that the datapoints lie in a compact latent space whose dimension can be much lower than that of the actual input space. Thanks to its lower dimensionality, we can dramatically increase the number of datapoints given a limited parameter count (e.g. the number of parameters used by synthetic images per each class). We can also preserve the quality of the synthetic images to the extent to which the assumption holds, which has been proven to be generally the case. The only space overhead is additional parameters for a tiny decoder that maps those latent codes into the input space. Note that this decoder can be shared across all the datapoints and classes, and even if the decoder is relatively big we can flexibly adjust the number of latent codes so that the total parameter counts are comparable.
Let us take an illustrative example. Suppose we want to condense a dataset consisting of images of shape 3×32×32 = 3072 (e.g. SVHN or CIFAR10). Instead of learning those 3072 parameters in the pixel space, we instead learn much smaller codes in, say 12 × 4 × 4 = 192 dimensional latent space and decode them back to the original pixel space with a shared decoder. Given the same parameter count, this allows us to generate 16 times more number of images than the existing approaches (192 × 16 = 3072). Note that the per-class parameter count of shared decoder is negligible. In the above example, we could use a tiny 3-layer deconvolutional architecture with channels decreasing as 12 → 9 → 6 → 3 and 2 × 2 kernel, resulting in only 738 parameters in total. Since this decoder can be shared across all classes, dividing it with the number of classes (e.g. 10) results in a negligible amount of per-class parameter count (e.g. 73.8) compared to the actual image size.
Sharing various styles with multiple decoders. The above factorization means that now we decouple the compressed expression of the instances from how to decode them back. This decoding process can be thought of as adding details to the latent codes so that the decoded instances have sufficient information in the original input space. For instance, given a compressed expression of a car, the decoder may add in or supplement background details or colors to complete an image.
This insight implies that the number of decoders can be as many as how various styles exists in the actual dataset. For instance, given the same compressed expression of an airplane, the background may be ocean, sky, forest, or sunset. By assuming various decoders, we could expect each of the decoders to explain each of those distinct styles in the dataset respectively. Furthermore, since the knowledge about how to add styles to the latent codes can be shared across all instances and classes, we can dramatically increase the number of synthetic images without increasing parameter count too much, similarly to cartesian product between a pair of sets. For instance, if there are 8 shared decoders and 16 latent codes for each of 10 classes, we can generate total 8 × 16 = 128 synthetic images with only around 3131 parameter count for each class, which is less than 2% increase from the parameter count of a single image 3 × 32 × 32 = 3072.

TRAINING WITH UNBIASED AND LOW-VARIANCE DISTRIBUTION MATCHING
Basically, our approach is compatible with any learning-based dataset condensation approaches that allows to differentiate w.r.t latent codes and decoder parameters by backpropagating through the generated examples, such as gradient matching  or distribution matching . In this paper, we propose to use distribution matching as it does not require expensive vector-Jacobian products (VJPs) computations whereas gradient matching requires to compute VJP for every training step.
Distribution matching. Here we introduce notations for further discussion. We denote a set of N real examples for each class c = 1, . . . , C as X c = {x c,1 , . . . , x c,N }. Similarly, a set of M latent codes for each class c is Θ c = {θ c,1 , . . . , θ c,M } and we collect them into Θ = {Θ 1 , . . . , Θ C }. Note that the dimension of θ is much lower than that of x. g(x) is a shared feature extractor taking x as an input. f (θ; φ d ) is d-th decoder parameterized by φ d that takes a latent code θ as an input. We assume there are D decoders and collect their parameters into φ = {φ 1 , . . . , φ D }. With these notations, we consider the following empirical MMD loss (Gretton et al., 2012;. Following , we sample g(·) from random initializations rather than from a set of pretrained networks, which is computationally more efficient and works well in practice.
Bias due to subsampling of synthetic examples.  further approximate the first term (i.e. mean of g(x) over n) and the second term (i.e. mean of g(f (θ; φ)) over d and m) by random subsampling for computational efficiency. However, whereas the gradient ∇ Θ,φ L(Θ, φ) is unbiased w.r.t the random subsampling of indices of real data {n} N n=1 or classes {c} C c=1 , the gradient is actually biased w.r.t the random subsampling of indices of latent codes {m} M m=1 or decoders {d} D d=1 . Specifically, suppose for simplicity we randomly sample a single pair of m ∼ Uniform(1, M ) and d ∼ Uniform(1, D) and denote the corresponding loss asL. Then we can easily compute the bias of gradient (2) See Appendix B for the derivation. The above result suggests that the bias comes from ignoring the interactions within the latent codes and within the decoders, respectively. The biased gradient fails to minimize the inner product of the embeddings computed from different codes and decoders, such as g(f (θ c,m ; φ)) T g(f (θ c,m ; φ)) and g(f (θ c,m ; φ d )) T g(f (θ c,m ; φ d )), preventing the codes and decoders from being diversified. Intuitively, if we sample θ, then each individual θ ∈ Θ c will try to explain the whole class set X c at a time without considering the existence of other codes θ ∈ Θ c \{θ} due to the sampling. The same intuition holds for sampling the decoder f (·; φ d ). Therefore, we should not sample m and d. However, despite of its importance, most of the recent works do not consider correcting the bias Cazenavette et al., 2022;Kim et al., 2022).
Variance due to subsampling of real examples. We also found that random subsampling of real examples is detrimental to performance due to large variance. Again, suppose for simplicity we randomly sample a single index n ∼ Uniform(1, N ) independently for each class c and denote the corresponding loss asL. Then the variance of the gradient Var n (∇ Θ,φL (Θ, φ)) is . See Appendix C for the derivation. The middle term in the big braces is nothing but the variance of embeddings of real examples g(x c,n ) over n = 1, . . . , N . The problem is that since we sample g(·) from random initializations according to the original distribution matching algorithm , the variance of g(x c,n ) tends to be very high compared to the discriminative power of the representation g(·) across the classes. We empirically observe that such relatively high variance makes the gradient ∇ Θ,φL (Θ, φ) too noisy for the training process to find a good solution.
Full batch training. Based on the above analysis, we propose to perform full batch training for both real and synthetic dataset for unbiased and low-variance training. In case of real dataset, we basically need to compute the embedding mean of real examples 1 N N n=1 g(x c,n ) across the classes and random weights w 1 , . . . , w K , with K denoting the total training steps. Instead of repeatedly computing them for every experiment, we speed up our experiments by precomputing those means once and storing them in an external device with the corresponding random weights, so that we can quickly restore them for the later experiments. In case of synthetic dataset, we simply distribute the classes over multiple GPUs and accumulate the gradients over all the processes. When the size of each class is too large to fit in a single GPU, we compute the embedding mean over the whole synthetic examples but backpropagate through only a subset of them for handling memory issue, while keeping the gradient unbiased.
Discussion. Although the distribution matching objective in Eq. (1) works well in practice, it is not clear why it should work with repeated sampling of g(·) from random initializations. A simple explanation may be that for each class, a well-condensed synthetic dataset should lead to a similar embedding mean to that of real dataset regardless of those random initializations. However, further justification and understanding of the objective is required, which we leave as a future work.

EXPERIMENTS
We next experimentally validate the efficacy of KFS on various datasets and learning scenarios.
Datasets. We consider the following benchmark datasets such as SVHN (32×32) (Netzer et al., 2011), CIFAR10 (32×32) (Krizhevsky et al., 2009), CIFAR100 (32×32) (Krizhevsky et al., 2009), and TinyImageNet (64×64) (Le & Yang, 2015). Baselines. We consider coreset selection approaches such as Random and Herding. For dataset condensation approaches, we compare with gradient (or trajectory) matching such as DC , DSA , Trajectory Matcing (Cazenavette et al., 2022), and IDC (Kim et al., 2022), as well as distribution matching such as DM  and CAFE . We also compare against KIP (Nguyen et al., 2021). Experimental setup. For fair comparison, we use essentially the same parameter count to the baselines. For instance, if there are 10 classes with 10 "Images / Class" (in Table 1 and  Table 2) and the image shape is 3×32×32, then the total per-class parameter count is 30720. In this case our KFS assumes 16 codes of shape 12×4×4 with the 8 decoders requiring only 73.8 additional parameters per class (see section 3.1). See Appendix A for further details about experimental setup.
Quantitative analysis. Table 1 shows the performance of the baselines and our KFS from each dataset with varying number of images (or parameter count) per class. We can see that KFS outperforms all the previous methods by significant margins in all datasets and settings. KFS especially performs well when the given parameter count per class is small. For instance, when "Images / Class" = 1, KFS shows 14.8% and 9.8% performance improvements over IDC on SVHN and CI-FAR10 respectively, the previous state-of-the-art recently introduced by Kim et al. (2022). This reconfirms the observation from Kim et al. (2022) that increasing the modality in the synthetic dataset is more important than polishing a few high-quality examples when the parameter count is very limited. However, KFS provides much higher modality (i.e. more number of synthetic examples) than IDS with essentially the same parameter count because KFS can compress each instance far more tightly by systematically factorizing and sharing the knowledge across the synthetic examples.
Cross-architecture generalization. Table 2 shows how the baselines and KFS perform when the network architecture becomes different at the evaluation time (ResNet-10 and DenseNet-121) from the one that used for condensation (ConvNet-3). We can see that the performance of KFS is more robust to the change of network architectures. For instance, when "Images/Class" = 1 in SVHN   dataset, the performance drop of KFS is only 7.2% and 1.9% for ResNet-10 and DenseNet-121 respectively, whereas the drop is 28.5% and 28.2% for IDC. Such robustness means that the learned synthetic examples from KFS are more natural as they are less specifically tailored to a specific network architecture. Similarly, when "Images/Class" = 50 in CIFAR10 dataset, the performance of KFS even improves as the architecture changes, while the performance of other baselines decreases.
Training budgets at evaluation time. One may wonder whether the good performance of KFS is solely due to the larger size of the synthetic dataset rather than the quality of each example, such that KFS inherently requires much longer evaluation time than the baselines. To address this question, we consider various amount of budget at evaluation time defined as total training steps allowed. In Figure 2 we can clearly see that KFS provides significantly better performances than the baselines across all range of budgets: from the limited budgets (100-500 training steps) to the longer training steps (1K-20K steps), demonstrating that the quality of synthetic examples generated from KFS is also comparable to or even better than that of the baselines. Interestingly, whereas the baselines often suffer from overfitting (e.g., SVHN dataset), our KFS has no such issue hence can provide more diverse options to users about how much longer to train their models for better performance.
Qualitative analysis. We next visualize the synthetic images generated from KFS in Figure 3. We can see from the top row that KFS can generate various modes of the data distribution by varing the learned latent codes while fixing a decoder. For instance, in SVHN dataset the digits show various styles with different adjacent numbers, and in CIFAR10 dataset the orientations and shapes of objects are diverse, demonstrating the ability of the latent codes to capture multi-modality of each data distribution. We also fix a code and vary the decoders in the bottom row. The synthetic examples show various backgrounds and styles of the objects such as strokes and colors. Overall, the visualization clearly shows how the factorization between latent codes and decoders allows KFS to efficiently share knowledge between the synthetic examples.

CONCLUSION
In this paper, we introduced a novel method for solving dataset condensation problem in a systematic and efficient way. Instead of parameterizing synthetic examples directly in input space, we proposed to fully exploit the regularity in a given dataset, such that we assume a generative process of synthetic examples based on the factorization between latent codes and decoders. Based on it, we showed how to increase the number of synthetic examples with essentially the same parameter count by combining the codes and decoders interchangeably. We further demonstrated how the knowledge is efficiently shared between synthetic examples with the framework, achieving superior trade-off between compression ratio and quality of the synthetic examples. Possible future directions include scaling up the method to achieve or even surpass the performances of the full dataset, and further developing the idea to multi-task or meta-learning setting where the goal is to share knowledge about how to compress a dataset between multiple tasks.

A EXPERIMENTAL SETUP
We provide some additional information about the experimental setup for reproducing our results.   "I/C" denotes "Image / Class". "Low-R" refers to the decoder in Table 3, and "High-R" refers to the decoder in Table.  where f c,m,d := f (θ c,m ; φ d ). We first compute ∇ Θ,φ L c (Θ, φ).