Question: Represent the scientific article to find a short TLDR

Question: Collaborative personalization, such as through learned user representations (embeddings), can improve the prediction accuracy of neural-network-based models significantly. We propose Federated User Representation Learning (FURL), a simple, scalable, privacy-preserving and resource-efficient way to utilize existing neural personalization techniques in the Federated Learning (FL) setting. FURL divides model parameters into federated and private parameters. Private parameters, such as private user embeddings, are trained locally, but unlike federated parameters, they are not transferred to or averaged on the server. We show theoretically that this parameter split does not affect training for most model personalization approaches. Storing user embeddings locally not only preserves user privacy, but also improves memory locality of personalization compared to on-server training. We evaluate FURL on two datasets, demonstrating a significant improvement in model quality with 8% and 51% performance increases, and approximately the same level of performance as centralized training with only 0% and 4% reductions. Furthermore, we show that user embeddings learned in FL and the centralized setting have a very similar structure, indicating that FURL can learn collaboratively through the shared parameters while preserving user privacy. Collaborative personalization, like learning user embeddings jointly with the task, is a powerful way to improve accuracy of neural-network-based models by adapting the model to each user's behavior (; ; ; ; ;). However, model personalization usually assumes the availability of user data on a centralized server. To protect user privacy, it is desirable to train personalized models in a privacy-preserving way, for example, using Federated Learning (; b). Personalization in FL poses many challenges due to its distributed nature, high communication costs, and privacy constraints (a; ; b; 2018; ; a). To overcome these difficulties, we propose a simple, communication-efficient, scalable, privacypreserving scheme, called FURL, to extend existing neural-network personalization to FL. FURL can personalize models in FL by learning task-specific user representations (i.e., embeddings) (; ; ; ;) or by personalizing model weights . Research on collaborative personalization in FL (; ;) has generally focused on the development of new techniques tailored to the FL setting. We show that most existing neural-network personalization techniques, which satisfy the split-personalization constraint, can be used directly in FL, with only a small change to Federated Averaging , the most common FL training algorithm. Existing techniques do not efficiently train user embeddings in FL since the standard Federated Averaging algorithm transfers and averages all parameters on a central server. Conventional training assumes that all user embeddings are part of the same model. Transferring all user embeddings to devices during FL training is prohibitively resource-expensive (in terms of communication and storage on user devices) and does not preserve user privacy. FURL defines the concepts of federated and private parameters: the latter remain on the user device instead of being transferred to the server. Specifically, we use a private user embedding vector on each device and train it jointly with the global model. These embeddings are never transferred back to the server. We show theoretically and empirically that splitting model parameters as in FURL affects neither model performance nor the inherent structure in learned user embeddings. While global model aggregation time in FURL increases linearly in the number of users, this is a significant reduction compared with other approaches whose global aggregation time increases quadratically in the number of users. FURL has advantages over conventional on-server training since it exploits the fact that models are already distributed across users. There is little resource overhead in distributing the embedding table across users as well. Using a distributed embeddings table improves the memory locality of both training embeddings and using them for inference, compared to on-server training with a centralized and potentially very large user embedding table. Our evaluation of document classification tasks on two real-world datasets shows that FURL has similar performance to the server-only approach while preserving user privacy. Learning user embeddings improves the performance significantly in both server training and FL. Moreover, user representations learned in FL have a similar structure to those learned in a central server, indicating that embeddings are learned independently yet collaboratively in FL. In this paper, we make the following contributions: • We propose FURL, a simple, scalable, resource-efficient, and privacy preserving method that enables existing collaborative personalization techniques to work in the FL setting with only minimal changes by splitting the model into federated and private parameters. • We provide formal constraints under which the parameter splitting does not affect model performance. Most model personalization approaches satisfy these constraints when trained using Federated Averaging , the most popular FL algorithm. • We show empirically that FURL significantly improves the performance of models in the FL setting. The improvements are 8% and 51% on two real-world datasets. We also show that performance in the FL setting closely matches the centralized training with small reductions of only 0% and 4% on the datasets. • Finally, we analyze user embeddings learned in FL and compare with the user representations learned in centralized training, showing that both user representations have similar structures. Most existing work on collaborative personalization in the FL setting has focused on FL-specific implementations of personalization. Multi-task formulations of Federated Learning (MTL-FL) present a general way to leverage the relationship among users to learn personalized weights in FL. However, this approach is not scalable since the number of parameters increases quadratically with the number of users. We leverage existing, successful techniques for on-server personalization of neural networks that are more scalable but less general, i.e., they satisfy the split-personalization constraint. Transfer learning has also been proposed for personalization in FL , but it requires alternative freezing of local and global models, thus complicating the FL training process. Moreover, some versions need access to global proxy data. uses a twolevel meta-training procedure with a separate query set to personalize models in FL. FURL is a scalable approach to collaborative personalization that does not require complex multiphase training, works empirically on non-convex objectives, and leverages existing techniques used to personalize neural networks in the centralized setting. We show empirically that user representations learned by FURL are similar to the centralized setting. Collaborative filtering can be seen as a specific instance of the generalized approach in FURL. Finally, while fine-tuning individual user models after FL training can be effective, we focuses on more powerful collaborative personalization that leverages common behavior among users. The main constraint in preserving privacy while learning user embeddings is that embeddings should not be transferred back to the server nor distributed to other users. While typical model parameters are trained on data from all users, user embeddings are very privacy-sensitive because a user's embedding is trained only on that user's data. FURL proposes splitting model parameters into federated and private parts. In this section, we show that this parameter-splitting has no effect on the FL training algorithm, as long as the FL training algorithm satisfies the split-personalization constraint. Models using common personalization techniques like collaborative filtering, personalization via embeddings or user-specific weights satisfy the split-personalization constraint when trained using Federated Averaging. FL algorithms typically have two steps: 1. Local Training: Each user initializes their local model parameters to be the same as the latest global parameters stored on the server. Local model parameters are then updated by individual users by training on their own data. This produces different models for each user. 1 2. Global Aggregation: Locally-trained models are "aggregated" together to produce an improved global model on the server. Many different aggregation schemes have been proposed, from a simple weighted average of parameters , to a quadratic optimization . To protect user privacy and reduce network communication, user embeddings are treated as private parameters and not sent to the server in the aggregation step. Formal conditions under which this splitting does not affect the model quality are described as follows. Suppose we train a model on data from n users, and the k-th training example for user i has features x i k, and label y i k. The predicted label isŷ, where the model has federated parameters w f ∈ R f and private parameters w i p ∈ R p ∀i ∈ {1, . . ., n}. In order to guarantee no model quality loss from splitting of parameters, FURL requires the splitpersonalization constraint to be satisfied, i.e., any iteration of training produces the same irrespective of whether private parameters are kept locally, or shared with the server in the aggregation step. The two following constraints are sufficient (but not necessary) to satisfy the splitpersonalization constraint: local training must satisfy the independent-local-training constraint, and global aggregation must satisfy the independent-aggregation constraint. The independent-local-training constraint requires that the loss function used in local training on user i is independent of private parameters for other users w j p, ∀j = i. A corollary of this constraint is that for training example k on user i, the gradient of the local loss function with respect to other users' private parameters is zero: Equation 1 is satisfied by most implementations of personalization techniques like collaborative filtering, personalization via user embeddings or user-specific model weights, and MTL-FL . Note that is not satisfied if the loss function includes a norm of the global user representation matrix for regularization. In the FL setting, global regularization of the user representation matrix is impractical from a bandwidth and privacy perspective. Even in centralized training, regularization of the global representation matrix slows down training a lot, and hence is rarely used in practice . Dropout regularization does not violate. Neither does regularization of the norm of each user representation separately. The independent-aggregation constraint requires, informally, that the global update step for federated parameters is independent of private parameters. In particular, the global update for federated parameters depends only on locally trained values of federated parameters, and optionally, on some summary statistics of training data. Furthermore, the global update step for private parameters for user i is required to be independent of private parameters of other users, and independent of the federated parameters. The global update for private parameters for user i depends only on locally trained values of private parameters for user i, and optionally, on some summary statistics. The independent-aggregation constraint implies that the aggregation step has no interaction terms between private parameters of different users. Since interaction terms increase quadratically in the number of users, scalable FL approaches, like Federated Averaging and its derivatives satisfy the independent-aggregation assumption. However, MTL-FL formulations do not. More formally, at the beginning of training iteration t, let w t f ∈ R f denote federated parameters and w i,t p ∈ R p denote private parameters for user i. These are produced by the global aggregation step at the end of the training iteration t − 1. Local Training At the start of local training iteration t, model of user i initializes its local federated parameters as u Global Aggregation At the end of local training, these locally updated parameters u are sent to the server for global aggregation. Equation 2 for federated parameters and Equation 3 for private parameters must hold to satisfy the independent-aggregation constraint. In particular, the global update rule for federated parameters w f ∈ R f must be of the form: where u t f,i is the local update of w f from user i in iteration t, s i ∈ R s is summary information about training data of user i (e.g., number of training examples), and a f is a function from R f +nf +ns → R f. Also, the global update rule for private parameters of user i, w i p ∈ R p, must be of the form: where u i,t p,i is the local update of w i p from user i in iteration t, s i ∈ R s is summary information about training data of user i, and a p is a function from R 2p+ns → R p. In our empirical evaluation of FURL, we use Federated Averaging as the function a f, while the function a p is the identity function w i,t+1 p:= u i,t p,i (more details in Section 3.4). However, FURL's approach of splitting parameters is valid for any FL algorithm that satisfies and. FURL works for all FL algorithms that satisfy the split-personalization constraint. Our empirical evaluation of FURL uses Federated Averaging , the most popular FL algorithm. The global update rule of vanilla Federated Averaging satisfies the independent-aggregation constraint since the global update of parameter w after iteration t is: where c i is the number of training examples for user i, and u t i is the value of parameter w after local training on user i in iteration t. Recall that u t i is initialized to w t at the beginning of local training. Our implementation uses a small tweak to the global update rule for private parameters to simplify implementation, as described below. In practical implementations of Federated Averaging , instead of sending trained model parameters to the server, user devices send model deltas, i.e., the difference between the original model downloaded from the server and the locally-trained model: d Since most personalization techniques follow Equation 1, the private parameters of user i, w The second term in the equation above is multiplied by a noisy scaling factor z i, an artifact of peruser example weighting in Federated Averaging. While it is not an essential part of FURL, our implementation ignores this scaling factor z i for private parameters. Sparse-gradient approaches for learning representations in centralized training also ignore a similar scaling factor for efficiency reasons. Thus, for the private parameters of user i, we simply retain the value after local training on user i (i.e., z i = 1) since it simplifies implementation and does not affect the model performance: where u i,t p is the local update of w i p from user i in iteration t. In other words, the global update rule for private parameters of user i is to simply keep the locally trained value from user i. While this paper focuses on learning user embeddings, our approach is applicable to any personalization technique that satisfies the split-personalization constraint. The training process is as follows: 1. Local Training: Initially, each user downloads the latest federated parameters from the server. Private parameters of user i, w i p are initialized to the output of local training from the last time user i participated in training, or to a default value if this was the first time user i was trained. Federated and private parameters are then jointly trained on the task in question. 2. Global Aggregation: Federated parameters trained in the step above are transferred back to, and get averaged on the central server as in vanilla Federated Averaging. Private parameters (e.g., user embeddings) trained above are stored locally on the user device without being transferred back to the server. These will be used for the next round of training. They may also be used for local inference. We evaluate the performance of FURL on two document classification tasks that reflect real-world data distribution across users. Datasets We use two datasets, called Sticker and Subreddit. Their characteristics are as follows. 1. In-house production dataset (Sticker): This proprietary dataset from a popular messaging app has randomly selected, anonymized messages for which the app suggested a Sticker as a reply. The features are messages; the task is to predict user action (click or not click) on the Sticker suggestion, i.e., binary classification. The messages were automatically collected, de-identified, and annotated; they were not read or labeled by human annotators. 2. Reddit comment dataset (Subreddit): These are user comments on the top 256 subreddits on reddit.com. , we filter out users who have fewer than 150 or more than 500 comments, so that each user has sufficient data. The features are comments; the task is to predict the subreddit where the comment was posted, i.e., multiclass classification. The authors are not affiliated with this publicly available dataset . Sticker dataset has 940K samples and 3.4K users (274 messages/user on average) while Subreddit has 942K samples and 3.8K users (248 comments/user on average). Each user's data is split (0.8/0.1/0.1) to form train/eval/test sets. Table 1 presents the summary statistics of the datasets. We formulate the problems as document classification tasks and use the a LSTM-based neural network architecture in Figure 1. The text is encoded into an input representation vector by using character-level embeddings and a Bidirectional LSTM (BLSTM) layer. A trainable embedding layer translates each user ID into a user embedding vector. Finally, an Multi-Layer Perceptron (MLP) produces the prediction from the concatenation of the input representation and the user embedding. All the parameters in the character embedding, BLSTM and MLP layers are federated parameters that are shared across all users. These parameters are locally trained and sent back to the server and averaged as in standard Federated Averaging. User embedding is considered a private parameter. It is jointly trained with federated parameters, but kept privately on the device. Even though user embeddings are trained independently on each device, they evolve collaboratively through the globally shared model, i.e., embeddings are multiplied by the same shared model weights. Configurations We ran 4 configurations to evaluate the performance of the models with/without FL and personalization: Global Server, Personalized Server, Global FL, and Personalized FL. Global is a synonym for non-personalized, Server is a synonym for centralized training. The experiment combinations are shown in Table 2. Model Training Server-training uses SGD, while FL training uses Federated Averaging to combine SGD-trained client models . Personalization in FL uses FURL training as described in section 3.5 The models were trained for 30 and 40 epochs for the Sticker and Subreddit datasets, respectively. One epoch in FL means the all samples in the training set were used once. We ran hyperparameter sweeps to find the best model architectures (such as user embedding dimension, BLSTM and MLP dimensions) and learning rates. The FL configurations randomly select 10 users/round and run 1 epoch locally for each user in each round. Separate hyperparameter sweeps for FL and centralized training ed in the same optimal embedding dimension for both configurations. The optimal dimension was 4 for the Sticker task and 32 for the Subreddit task. Metrics We report accuracy for experiments on the Subreddit dataset. However, we report AUC instead of accuracy for the Sticker dataset since classes are highly unbalanced. Personalization improves the performance significantly. User embeddings increase the AUC on the Sticker dataset by 7.85% and 8.39% in the Server and FL configurations, respectively. The improvement is even larger in the Subreddit dataset with 37.2% and 50.51% increase in the accuracy for the Server and FL settings, respectively. As shown in Figure 2, these demonstrate that the user representations effectively learn the features of the users from their data. Personalization in FL provides similar performance as server training. There is no AUC reduction on the Sticker dataset while the accuracy drops only 3.72% on the Subreddit dataset (as shown in Figure 2). Furthermore, the small decrease of FL compared to centralized training is expected and consistent with other . The learning curves on the evaluation set on Figure 3 show the performance of FL models asymptotically approaches the server counterpart. Therefore, FL provide similar performance with the centralized setting while protecting the user privacy. User embeddings learned in FL have a similar structure to those learned in server training. Recall that for both datasets, the optimal embedding dimension was the same for both centralized and FL training. We visualize the user representations learned in both the centralized and FL settings using t-SNE (van der Maaten & Hinton, Nov 2008). The demonstrate that similar users are clustered together in both settings. Visualization of user embeddings learned in the Sticker dataset in Figure 4 shows that users having similar (e.g., low or high) click-through rate (CTR) on the suggested stickers are clustered together. For the Subreddit dataset, we highlight users who comment a lot on a particular subreddit, for the top 5 subreddits (AskReddit, CFB, The Donald, nba, and politics). Figure 5 indicates that users who submit their comments to the same subreddits are clustered together, in both settings. Hence, learned user embeddings reflect users' subreddit commenting behavior, in both FL and Server training. This paper proposes FURL, a simple, scalable, bandwidth-efficient technique for model personalization in FL. FURL improves performance over non-personalized models and achieves similar performance to centralized personalized model while preserving user privacy. Moreover, representations learned in both server training and FL show similar structures. In future, we would like to evaluate FURL on other datasets and models, learn user embeddings jointly across multiple tasks, address the cold start problem and personalize for users not participating in global FL aggregation.

Represent the short summary for finding the corresponding scientific article (long)

We propose Federated User Representation Learning (FURL), a simple, scalable, privacy-preserving and bandwidth-efficient way to utilize existing neural personalization techniques in the Federated Learning (FL) setting.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Recurrent models for sequences have been recently successful at many tasks, especially for language modeling and machine translation. Nevertheless, it remains challenging to extract good representations from these models. For instance, even though language has a clear hierarchical structure going from characters through words to sentences, it is not apparent in current language models. We propose to improve the representation in sequence models by augmenting current approaches with an autoencoder that is forced to compress the sequence through an intermediate discrete latent space. In order to propagate gradients though this discrete representation we introduce an improved semantic hashing technique. We show that this technique performs well on a newly proposed quantitative efficiency measure. We also analyze latent codes produced by the model showing how they correspond to words and phrases. Finally, we present an application of the autoencoder-augmented model to generating diverse translations. Autoencoders have a long history in deep learning BID3 BID10 BID16 BID7. In most cases, autoencoders operate on continuous representations, either by simply making a bottleneck BID3, denoising BID16, or adding a variational component BID7. In many cases though, a discrete latent representation is potentially a better fit. Language is inherently discrete, and autoregressive models based on sequences of discrete symbols yield impressive . A discrete representation can be fed into a reasoning or planning system or act as a bridge towards any other part of a larger system. Even in reinforcement learning where action spaces are naturally continuous, show that discretizing them and using autoregressive models can yield improvements. Unluckily, using discrete latent variables is challenging in deep learning. And even with continuous autoencoders, the interactions with an autoregressive component cause difficulties. Despite some success BID1 BID17, the task of meaningfully autoencoding text in the presence of an autoregressive decoder has remained a challenge. In this work we present an architecture that autoencodes a sequence s of N discrete symbols from any vocabulary (e.g., a tokenized sentence), into a K-fold (we test K = 8 and K = 32) compressed sequence c(s) of Since gradient signals can vanish when propagating over discrete variables, the compression function c(s) can be hard to train. To solve this problem, we draw from the old technique of semantic hashing BID11. There, to discretize a dense vector v one computes σ(v + n) where σ is the sigmoid function and n represents annealed Gaussian noise that pushes the network to not use middle values in v. We enhance this method by using a saturating sigmoid and a straight-through pass with only bits passed forward. These techniques, described in detail below, allow to forgo the annealing of the noise and provide a stable discretization mechanism that requires neither annealing nor additional loss factors. We test our discretization technique by amending language models over s with the autoencoded sequence c(s). We compare the perplexity achieved on s with and without the c(s) component, and contrast this value with the number of bits used in c(s). We argue that this number is a proper measure for the performance of a discrete autoencoder. It is easy to compute and captures the performance of the autoencoding part of the model. This quantitative measure allows us to compare the technique we introduce with other methods, and we show that it performs better than a GumbelSoftmax BID4 BID8 in this context. Finally, we discuss the use of adding the autoencoded part c(s) to a sequence model. We present samples from a character-level language model and show that the latent symbols correspond to words and phrases when the architecture of c(s) is local. ehen, we introduce a decoding method in which c(s) is sampled and then s is decoded using beam search. This method alleviates a number of problems observed with beam search or pure sampling. We show how our decoding method can be used to obtain diverse translations of a sentence from a neural machine translation model. To summarize, the main contributions of this paper are: a discretization technique that works well without any extra losses or parameters to tune, a way to measure performance of autoencoders for sequence models with baselines, an improved way to sample from sequence models trained with an autoencoder part. Below, we introduce our discretization method, the autoencoding function c(s) and finally the complete model that we use for our experiments. All code and hyperparameter settings needed to replicate our experiments will be available as open-source 1. As already mentioned above, our discretization method stems from semantic hashing BID11. To discretize a b-dimensional vector v, we first add noise, so v n = v + n. The noise n is drawn from a b-dimensional Gaussian distribution with mean 0 and standard deviation 1 (deviations between 0 and 1.5 all work fine, see ablations below). The sum is component-wise, as are all operations below. Note that noise is used only for training, during evaluation and inference n = 0. From v n we compute two vectors: v 1 = σ (v n) and v 2 = (v n < 0), where σ is the saturating sigmoid function from BID6 BID5: DISPLAYFORM0 The vector v 2 represents the discretized value of v and is used for evaluation and inference. During training, in the forward pass we use v 1 half of the time and v 2 the other half. In the backward pass, we let gradients always flow to v 1, even if we used v 2 in the forward computation 2.We will denote the vector v discretized in the above way by DISPLAYFORM1 Since in other parts of the system we will predict v d with a softmax, we want the number of bits to not be too large. In our experiments we stick with b = 16, so v d is a vector of 16 bits, and so can be interpreted as an integer between 0 and 2 16 − 1 = 65535.The dense vectors representing activations in our sequence models have much larger dimensionality than 16 (often 512, see the details in the experimental section below). To discretize such a high-dimensional vector w we first have a simple fully-connected layer converting it into v = dense(w, 16). In our notation, dense(x, n) denotes a fully-connected layer applied to x and mapping it into n dimensions, i.e., dense(x, n) = xW + B where W is a learned matrix of shape d × n, where d is the dimensionality of x, and B is a learned bias vector of size n. The discretized vector v d is converted back into a high-dimensional vector using a 3-layer feed-forward network: h1a = dense(vd, filter_size) h1b = dense(1.0 -vd, filter_size) h2 = dense(relu(h1a + h1b), filter_size) = dense(relu(h2), hidden_size)Above, every time we apply dense we create a new weight matrix an bias to be learned. The relu function is defined in the standard way: relu(x) = max(x, 0). In the network above, we usually use a large filter size; in our experiments we set it to 4096 while hidden size was usually 512. We suspect that this allows the above network to recover from the discretization bottleneck by simulating the distribution of w encountered during training. Given a dense, high-dimensional vector w we will denote the corresponding returned from the network above by bottleneck(w) and the corresponding discrete vector v 2 by discrete(w). As an alternative discretization method, we consider the recently studied Gumbel-Softmax BID4 BID8. In that case, given a vector w we compute discrete g (w) by applying a linear layer mapping into 2 16 elements, ing in the logits l. During evaluation and inference we simply pick the index of l with maximum value for discrete g (w) and the vector bottleneck g (w) is computed by an embedding. During training we first draw samples g from the Gumbel distribution: g ∼ − log(− log(u)), where u ∼ U are uniform samples. Then, as in BID4, we compute x, the log-softmax of l, and set: DISPLAYFORM0 With low temperature τ this vector is close to the 1-hot vector representing the maximum index of l. But with higher temperature, it is an approximation (see FIG0 in). We multiply this vector y by the embedding matrix to compute bottleneck g (w) during training. Having the functions bottleneck(w) and discrete(w) (respectively their Gumbel-Softmax versions), we can now describe the architecture of the autoencoding function c(s). We assume that s is already a sequence of dense vectors, e.g., coming from embedding vectors from a tokenized sentence. To halve the size of s, we first apply to it 3 layers of 1-dimensional convolutions with kernel size 3 and padding with 0s on both sides (SAME-padding). We use ReLU non-linearities between the layers and layer-normalization BID0. Then, we add the input to the , forming a residual block. Finally, we process the with a convolution with kernel size 2 and stride 2, effectively halving the size of s. In the local version of this function we only do the final strided convolution, without the residual block. To autoencode a sequence s and shorten it K-fold, with K = 2 k, we first apply the above step k times obtaining a sequence s that is K times shorter. Then we put it through the discretization bottleneck described above. The final compression function is given by c(s) = bottleneck(s) and the architecture described above is depicted in FIG0.Note that, since we perform 3 convolutions with kernel 3 in each step, the network has access to a large context: 3 · 2 k−1 just from the receptive fields of convolutions in the last step. That's why we also consider the local version. With only strided convolutions, the i-th symbol in the local c(s) has only access to a fixed 2 k symbols from the sequence s and can only compress them. Training with c(s) defined above from scratch is hard, since at the beginning of training s is generated by many layers of untrained convolutions that are only getting gradients through the discretization bottleneck. To help training, we add a side-path for c(s) without discretization: we just use c(s) = s for the first 10000 training steps. In this pretraining stage the network reaches loss of almost 0 as everything needed to reconstruct s is encoded in s. After switching to c(s) = bottleneck(s) the loss is high again and improves during further training. To test the autoencoding function c(s) we will use it to prefix the sequence s in a sequence model. Normally, a sequence model would generate the i-th element of s conditioning on all elements of s before that, s <i, and possibly on some other inputs. For example, a language model would just condition on s <i while a neural machine translation model would condition on the input sentence (in the other language) and s <i. We do not change the sequence models in any way other than adding the sequence c(s) as the prefix of s. Actually, for reasons analogous to those in BID13, we first reverse the sequence c(s), then add a separator symbol (#), and only then concatenate it with s, as depicted in FIG1. We also use a separate set of parameters for the model predicting c(s) so as to make sure that the models predicting s with and without c(s) have the same capacity. As the architecture for the sequence model we use the Transformer . Transformer is based on multiple attention layers and was originally introduced in the context of neural machine translation. We focused on the autoencoding function c(s) and did not tune the sequence model in this work: we used all the defaults from the baseline provided by the Transformer authors (6 layers, hidden size of 512 and filter size of 4096) and only varied parameters relevant to c(s). We experimented with autoencoding on 3 different sequence tasks: on a character-level language model, on a word-level language model, and on a word-level translation model. The goal for was to check if our technique works at all, since character sequences are naturally amenable to compression into shorter sequences of objects from a larger vocabulary. For, we wanted to check if the good obtained in will still hold if the input is from a larger vocabulary and inherently more compressed space. Finally, in we want to check if this method is applicable to conditional models and how it can be used to improve decoding. We use the LM1B corpus BID2 for language modelling and we tokenize it using a subword (wordpiece) tokenizer BID12 into a vocabulary of 32000 words and word-pieces. For translation, we use the WMT English-German corpus, similarly tokenized into a vocabulary of 32000 words and word-pieces 3.Below we report both qualitative and quantitative . First, we focus on measuring the performance of our autoencoder quantitatively. To do that, we introduce a measure of discrete autoen- coder performance on sequence tasks and compare our semantic hashing based method to GumbelSoftmax on this scale. Sequence models trained for next-symbol prediction are usually trained (and often also evaluated) based on the perplexity per token that they reach. Perplexity is defined as 2 H, where H is the entropy (in bits) of a distribution. Therefore, a language model that reaches a per-word perplexity of p, say p = 32, on a sentence s can be said to compress each word from s into log(p) = 5 bits of information. Let us now assume that this model is allowed to access some additional bits of information about s before decoding. In our autoencoding case, we let it peek at c(s) before decoding s, and c(s) has K = 8 times less symbols and b = 16 bits in each symbol. So c(s) has the information capacity of 2 bits per word. If our autoencoder was perfectly aligned with the needs of the language model, then allowing it to peek into c(s) would lower its information needs by these 2 bits per word. The perplexity p of the model with access to c(s) would thus satisfy log 2 (p) = 5 − 2 = 3, so its perplexity would be p = 8.Getting the autoencoder c(s) perfectly aligned with the language model is hard, so in practice the perplexity p is always higher. But since we measure it (and optimize for it during training), we can calculate how many bits has the c(s) part actually contributed to lowering the perplexity. We calculate log 2 (p) − log 2 (p) and then, if c(s) is K-times shorter than s and uses b bits, we define the discrete sequence autoencoding efficiency as: DISPLAYFORM0 The second formulation is useful when the raw numbers are given as natural logarithms, as is often the case during neural networks training. Defined in this way, DSAE measures how many of the available bits in c(s) are actually used well by the model that peeks into the autoencoded part. Note that some models may have autoencoding capacity higher than the number of bits per word that log(p) indicates. In that case achieving DSAE=1 is impossible even if log(p) = 0 and the autoencoding is perfect. One should be careful when reporting DSAE for such over-capacitated models. So how does our method perform on DSAE and how does it compare with Gumbel-Softmax? In Table 1 we list log-perplexties of baseline and autoencoder models. We report numbers for the global version of c(s) on our 3 problems and compare it to Gumbel-Softmax on word-level problems. We did not manage to run the Gumbel-Softmax on character-level data in our baseline configuration because it requires too much memory (as it needs to learn the embeddings for each latent discrete symbol). Also, we found that the for Gumbel-Softmax heavily depend on how the temperature parameter τ is annealed during training. We tuned this on 5 runs of a smaller model and chose the best configuration. This was still not enough, as in many runs the Gumbel-Softmax would only utilize a small portion of the discrete symbols. We added an extra loss term to increase the variance of the Gumbel-Softmax and ran another 5 tuning runs to optimize this loss term. We used the best configuration for the experiments above. Still, we did not manage to get any information autoencoded in the translation model, and got only 12% efficiency in the language model (see Table 1). Table 2: Autoencoder-augmented language models with different noise deviations. All values from no noise (0.0) upto a deviation of 1.5 yield DSAE between 40% and 50%.Our method, on the other hand, was most efficient on character-level language modeling, where we reach almost 60% efficiency, and it retained high 55% efficiency on the word-level language modeling task. On the translation task, our efficiency goes down to 19%, possibly because the c(s) function does not take inputs into account, and so may not be able to compress the right parts to align with the conditional model that outputs s depending on the inputs. But even with 19% efficiency it is still useful for sampling from the model, as shown below. To make sure that our autoencoding method is stable, we experiment with different standard deviations for the noise n in the semantic hashing part. We perform these experiments on word-level language modelling with a smaller model configuration (3 layers, hidden size of 384 and filter size of 2048). The , presented in Table 2, show that our method is robust to the amount of noise. Interestingly, we see that our method works even without any noise (standard deviation 0.0). We suspect that this is due to the fact that half of the time in the forward computation we use the discrete values anyway and pass gradients through to the dense part. Also, note that a standard deviation of 1.5 still works, despite the fact that our saturating sigmoid is saturated for values above 2.4 as 1.2 · σ(2.4) − 0.1 = 1.0002. Finally, with deviation 1.0 the small model achieves DSAE of 48.5%, not much worse than the 55% achieved by the large baseline model and better than the larger baseline model with Gumbel-Softmax. Having trained the models, we try to find out whether the discrete latent symbols have any interpretable meaning. We start by asking a simpler question: do the latent symbols correspond to some fixed phrases or topics?We first investigate this in a 32-fold compressed character-level language model. We set c(s) to 4 random latent symbols [l 1, l 2, l 3, l 4] and decode s with beam search, obtaining:All goods are subject to the Member States' environmental and security aspects of the common agricultural policy. Now, to find out whether the second symbol in c(s) stands for anything fixed, we replace the third symbol by the second one, hoping for some phrase to be repeated. Indeed, decoding s from the new c(s) = [l 1, l 2, l 2, l 4] with beam search we obtain:All goods are charged EUR 50.00 per night and EUR 50.00 per night stay per night. Note that the beginning of the sentence remained the same, as we did not change the first symbol, and we see a repetition of EUR 50.00 per night. Could it be that this is what that second latent symbol stands for? But there were no EUR in the first sentence. Let us try again, now changing the first symbol to a different one. With c(s) = [l 5, l 2, l 2, l 4] the decoded s is:All bedrooms suited to the large suite of the large living room suites are available. We see a repetition again, but of a different phrase. So we are forced to conclude that the latent code is structured, the meaning of the latent symbols can depend on other symbols before them. Failing to decipher the code from this model, we try again with an 8-fold compressed character-level language model that uses the local version of the function c(s). Recall (see Section 2.3) that a local function c(s) with 8-fold compression generates every latent symbol from the exact 8 symbols that correspond to it in s, without any context. With this simpler c(s) the model has lower DSAE, 35%, but we expect the latent symbols to be more context-independent. And indeed: if we pick the first 2 latent symbols at random but fix the third, fourth and fifth to be the same, we obtain the following:It's studio, rather after a gallery gallery... When prices or health after a gallery gallery... I still offer hotels at least gallery gallery... So the fixed latent symbol corresponds to the word gallery in various contexts. Let us now ignore context-dependence, fix the first three symbols, and randomly choose another one that we repeat after them. Here are a few sample decodes:Come to earth and culturalized climate climate... Come together that contribution itself, itself,... Come to learn that countless threat this gas threat... In the first two samples we see that the latent symbol corresponds to climate or itself, respectively. Note that all these words or phrases are 7-characters long (and one character for space), most probably due to the architecture of c(s). But in the last sample we see a different phenomenon: the latent symbol seems to correspond to X threat, where X depends on the context, showing that this latent code also has an interesting structure. From the above we know that our discretization method works quantitatively and we see interesting patterns in the latent code. But how can we use the autoencoder models in practice? One well-known problem with autoregressive sequence models is decoding. In settings where the possible outputs are fairly restricted, such as translation, one can obtain good with beam search. But obtained by beam search lack diversity BID15. Sampling can improve diversity, but it can introduce artifacts or even change semantics in translation. We present an example of this problem in FIG2. We pick an English sentence from the validation set of our English-German dataset and translate it using beam search and sampling (left and middle columns).In the left column, we show top 3 from beam search using our baseline model (without autoencoder). It is not necessary to speak German to see that they are all very similar; the only difference between the first and the last one are the spaces before "%". Further beams are also like this, providing no real diversity. In the middle column we show 3 sampled from the baseline model. There is more diversity in them, but they still share most of the first half and unluckily all of them actually changed the semantics of the sentence in the second half. The part African-Americans, who accounted however for only 13% of voters in the State becomes The american voters were only 13% of voters in the state in the first case, African-Americans, who accounted however for only 13% of all people in the State in the second one, and African-Americans, who elected only 13% of people in the State in the third case. This illustrates the dangers of just sampling different words during decoding. Using a model with access to the autoencoded part c(s) presents us with another option: sample c(s) and then run beam search for the sequence s appropriate for that c(s). In this way we do not introduce low-level artifacts from sampling, but still preserve high-level diversity. To sample c(s) we train a language model on c(s) with the same architecture as the model for s (and also conditioned on the input), but with a different set of weights. We then use the standard multinomial sampling from this model to obtain c(s) and run a beam search on the model for s with the sampled c(s).In the right column in FIG2 we show 3 samples obtained in this way. As you can see, these samples are much more diverse and they still preserve the semantics of the original sentence, even if with sometimes strange syntax. One would back-translate the first example as: In turned out, for example, in the course of the parliamentary elections in Florida, that 33% of the early voters are African-Americans, which were, however, only 13% of the voters of the state. Note the addition of It turned out and restructuring of the sentence. In the third sample the whole order is reversed, as it starts with 33% of the voters... instead of the election phrase. Obtaining such samples that differ in phrase order and other aspects but preserve semantics has been a challenge in neural translation. In this work, the study of text autoencoders BID1 BID17 is combined with the research on discrete autoencoders BID4 BID8. It turns out that the semantic hashing technique BID11 can be improved and then yields good in this context. We introduce a measure of efficiency of discrete autoencoders in sequence models and show that improved semantic hashing has over 50% efficiency. In some cases, we can decipher the latent code, showing that latent symbols correspond to words and phrases. On the practical side, sampling from the latent code and then running beam search allows to get valid but highly diverse samples, an important problem with beam search BID15.We leave a number of questions open for future work. How does the architecture of the function c(s) affect the latent code? How can we further improve discrete sequence autoencoding efficiency? Despite remaining questions, we can already see potential applications of discrete sequence autoencoders. One is the training of multi-scale generative models end-to-end, opening a way to generating truly realistic images, audio and video. Another application is in reinforcement learning. Using latent code may allow the agents to plan in larger time scales and explore more efficiently by sampling from high-level latent actions instead of just atomic moves.

Autoencoders for text with a new method for using discrete latent space.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent

Question: Auto-encoding and generative models have made tremendous successes in image and signal representation learning and generation. These models, however, generally employ the full Euclidean space or a bounded subset (such as $^l$) as the latent space, whose trivial geometry is often too simplistic to meaningfully reflect the structure of the data. This paper aims at exploring a nontrivial geometric structure of the latent space for better data representation. Inspired by differential geometry, we propose \textbf{Chart Auto-Encoder (CAE)}, which captures the manifold structure of the data with multiple charts and transition functions among them. CAE translates the mathematical definition of manifold through parameterizing the entire data set as a collection of overlapping charts, creating local latent representations. These representations are an enhancement of the single-charted latent space commonly employed in auto-encoding models, as they reflect the intrinsic structure of the manifold. Therefore, CAE achieves a more accurate approximation of data and generates realistic new ones. We conduct experiments with synthetic and real-life data to demonstrate the effectiveness of the proposed CAE. Autoencoding (; ;) is a central tool in unsupervised representation learning. The latent space therein captures the essential information of a given data set, serving the purposes of dimension reduction, denoising, and generative modeling. Even for models such as generative adversarial networks that do not employ an encoder, the generative component starts with a latent space. A common practice is to model the latent space as a low-dimensional Euclidean space R l or a bounded subset of it (e.g., l ), sometimes equipped with a prior probability distribution. Such spaces carry far simple geometry and may not be adequate for representing complexly structured data. In this work, we are concerned with a widely studied structure: manifold. A commonly held belief, known as the Manifold Hypothesis , states that real-world data often lies on, or at least near, some low-dimensional manifold embedded in the high-dimensional ambient space. Hence, a natural approach to representation learning is to introduce a low-dimensional latent space to which the data is mapped. It is desirable that such a mapping possesses basic properties such as invertibility and continuity. In differential geometry, such a notion is coined homeomorphism. Challengingly, it is known that even for some simple manifolds, there does not always exist a homeomorphic mapping to the Euclidean space of the intrinsic dimension of the data. We elaborate two examples such examples next. Consider a data set X lying on the 2-dimensional sphere S 2 embedded in the ambient space R n where n ≥ 3. It is well known that there exist no homeomorphic maps between S 2 and an open domain on R 2 . Therefore, it is impossible for a traditional autoencoder with a 2-dimensional latent space to faithfully capture the structure of the data. Consequently, the dimension of the latent space needs be increased beyond the intrinsic dimension (two in this case). For another example, we show in Figure 1 a double torus. When one uses an autoencoder to map uniform data points on this manifold to R 2, the distribution of the points is distorted and the shape destroyed, whereas if one maps to R 3, some of the points depart from the mass and become outliers. Fur- thermore, in the appendix (see Figure 11) we demonstrate that increasing the number of parameters of the autoencoder does not help overcome the coverage issue when the latent space is a single 2-dimensional space. To better reflect structures, in this work, we follow the definition of manifolds in differential geometry and propose Chart Auto-Encoder (CAE) for learning a low-dimensional representation of data lying on a manifold. Rather than using a single function mapping, the manifold is parameterized by a collection of overlapping chart functions, each of which describes a local neighborhood and they collectively cover the entire manifold. To the right of Figure 1, we show the same double torus aforementioned, now encoded by using four color-coded charts. One sees that the encoding faithfully preserves the shape of the data set, as well as the topology (two holes). To realize the parameterization, we develop a neural network architecture and propose a training regime to implement it. We conduct a comprehensive set of experiments on both synethic data and real-world data to demonstrate that CAE captures much better the structure of the data and enriches the understanding of them. One common approach to enhancing the capability of autoencoders is to impose a prior distribution on the flat Euclidean latent space, as in variational autoencoders (VAE) . The distributional assumption (e.g., Gaussian) introduces low-density regions that sometimes depart from the manifold. Then, paths in these regions will either trace off the manifold or become invariant. introduce a non-Euclidean latent space to guarantee the existence of a homeomorphic representation by using a so-called homeomorphic variational autoencoder (HVAC). There are two disadvantages of this approach. First, it requires the user to know the topological class of the data set, which can be extremely difficult for real-world data. Secondly, it requires the computation (or estimation) of the Lie group action on the latent space. If the topology of the data is relatively simple (e.g., a sphere or torus), the computation is possible, but for more complex objects the estimation is much difficult. Similarly, several recent works; have studied autoencoders with (hyper)-spherical latent spaces. These methods allow for the detection of cyclical features, but offer little insight into the homology of the manifold, since it will always be represented by a compact genus-zero surface. Exploring the low-dimensional structure of manifolds has led to many dimension reduction techniques in the past two decades (; ; ; ; ; ; ; van der). Isomap divides a data set into local neighborhoods, which are embedded into a low-dimensional space that preserves local properties. Similarly, Laplacian Eigenmaps use embeddings induced by the LaplaceBeltrami eigen-functions to represent the data. These methods employ a flat Euclidean space for embedding and may lose information as aforementioned. considers a manifold point of view to explain Wasserstein generative adversarial network (WGAN) using optimal transport by minimizing a distance between a manifold parameterize by a neural network and one estimated from some training data. Under the manifold hypothesis extend the work of and show the theoretical existence of neural networks that approximate functions supported on lowdimensional manifolds, with a number of parameters only weakly dependent on the embedding dimension. A key feature in their proposal is a chart determination sub-network that divides the manifold into charts and a pairing sub-network that re-combines them. All existing methods only have theoretical consideration of chart structure for data manifolds and assume that the manifold in question is known. However, multi-chart latent space representations have not been implemented and conducted computationally. Moreover, part of the constructions relies on information of the underlying manifold, which is unavailable in most real-world applications. Additionally, important questions regarding the loss function and the training method remain open. Our work introduces an implementable neural network architecture which is able to address these challenges by directly approximating the data manifold. A manifold is a topological space locally homeomorphic to a Euclidean domain. More formally, a d-dimensional manifold is defined as a collection of pairs {(M α, φ α)} α, referred to as charts, Figure 2 left). Smoothness of the transition functions controls the smoothness of the manifold. A well-known from differential geometry states that any compact manifold can be covered by a finite number of charts which obey these transition conditions. The intrinsic dimension of the manifold is the dimension of U α. for a thorough review. In practice, the coherent structure of data motivates us to model a given data as samples from an unknown ground manifold. One crucial task in machine learning is to explore the topological and geometric structure of the manifold and perform tasks such as classification and data generation. Mathematically, we explain the encoding and decoding process for a manifold as follows. Given some manifold M, usually embedded in a high dimension ambient space R n, the encoding network constructs a local parameterization φ α from the data manifold to the latent space U α; and the decoding network maps U α back to the data manifold M through φ −1 α. In standard autoencoders, only one single chart U α is used as the latent space. In our work, multiple charts are used. Different from classical dimension reduction methods where distance preservation is preferred, we do not require the local parameterization φ α to preserve metric but only bound its Lipschitz constant to control the regularity of the parameterization (see Section 4 for more details). To illustrate the utility of such a multi-charted parameterization, we consider a simple example: find a latent representation of data sampled from a 1-dimensional circle S 1 embedded in R 2. See Figure 2 right. A simple (non-chart) parameterization is (cos(z), sin(z)) with z ∈ (−∞, ∞). However, approximating this parameterization with a finite neural network is impossible, since z is unbounded and hence any multi-layer perceptron will have an infinite Vapnik-Chervonenkis dimension . One obvious alternative is to limit z ∈ [0, 2π), but this parameterization introduces a discontinuity and breaks the topology (it is theoretically known that the closed circle is not homeo-morhpic to [0, 2π) ). Following the definition of manifold, we instead parameterize the circle as: Although this function is more cumbersome to write, it is more suitable for representation learning, since each encoding function can be represented with finite neural networks. Moreover, the topological and geometric information of the data is maintained. Thus, instead of using only one chart as in standard autoencoders (; ;), we propose to model the latent space with multiple charts glued by their transition functions, akin to the concept of manifold. This geometric construction reflects the intrinsic structure of the manifold. Therefore, it is able to achieve more accurate approximation of the data and generate realistic new ones. Moreover, once the charts and the associated transition functions are learned, the geometric information of the data manifold, including metric, geodesics, and curvatures, can be approximated according to their definitions in differential geometry. Thus, this multi-chart latent construction leads to a better geometric understanding of the manifold. To integrate manifold structure in the latent space, we investigate a new representation of latent space based on a multi-chart structure. We implement a novel network architecture to learn the multi-chart latent space and its transition functions. The proposed network architecture, illustrated in Figure 3, can be summarized as follows: An input data x is passed into an encoding module, which creates an initial latent representation. Next, a collection of chart paramterizations-encoders E i as analogy of φ α -map the initial latent representation to N different chart spaces U α, which provides the new multi-chart latent space. Each chart representation is then passed into a decoding function, a decoder D i as analogy of φ −1 α, which produces approximation to the input data. Finally, the chart prediction module decides which chart and the associated decoder best represent the data. As detailed in A.3, this architecture, along with the proposed loss function, naturally enforces the chart transitions without explicit computation. Initial Encoder The initial encoder serves as a dimension reduction step to find a low-dimensional isometric embedding of the data from R n to R l. For example, given an R 3 torus embedded in R 1000, the initial encoder maps from R 1000 to a lower dimensional space, ideally R 3. (Note that however three is not the intrinsic dimension of the torus; rather, two is. Thus, subsequently we introduce chart encoders to map from the 3-dimensional space to R 2 .) We approximate this mapping using a neural net, denoted as E, with a combination of fully connected and convolution layers (see section A.5 for details). We choose l n; this encoding can be viewed as a dimension reduction step, which prepares the data to be split into each of the multi-chart latent spaces. Ideally, this step preserves the original topology and geometry information of the data set while also reducing its dimension to the minimal isometric embedding dimension of the manifold. It aims at improving computational efficiency for decoders to multi-chart latent space. This step can be replaced with a Homeomoric Variation Auto-Encoder in the cases where the topology of the data set is known, or with an appropriately chosen random matrix projection . Chart Encoder This step provides parameterization of the data manifold locally to a chart space, whose dimension is approximately close to the intrinsic dimension of the data manifold. This splitting is done with a small collection of networks {E α} which takes z ∈ R l as input and outputs N local coordinates z α ∈ U α. We denote the direct sum of these space as U = N α=1 U α, which is the proposed multi-chart latent space for our model. In practice, we choose U α = l and regularize the Lipschitz constant of the corresponding encoding map, to control size and regularity of the region M α ⊂ M paramterized by U α (more details in section 4). We remark that the proposed multi-chart architecture aims at constructing the correct latent space structure and understanding the geometric structure of the data manifold. The decoupled nature of the encoding operations mean that the model tends to be larger in terms of the number of parameters. However, the improvement shown in the experiments is not caused by the use of more parameters; rather, a correct latent space structure. Further experiments in Appendix A.10 show that increasing the number of parameters in a VAE alone (without increasing the latent dimension) does not allow one to simultaneously produce good reconstruction and generation. A latent space of too small dimension will not be able to cover a manifold, and one of a too large dimension will generate points far from the manifold. Thus, the structure of the latent space is more important than the number of parameters. Decoders Each of the latent chart U α is equipped with an associated decoder function D α, which maps from the chart latent space U α back to the ambient space. We represent each of these maps with a deep network, which are trained to reconstruct the input data. Chart Selection There are several options to the chart selection module P for an input x sampled in the data manifold. In general, this module must produce prediction or confidence measure {p α (x)} α = P(x) regarding which chart should be used for a given input. After training, this module can also be used to reduce the computational cost of evaluating the model, by ensuring that only a small set of decoder functions needs to be evaluated for any input signal. Output The output of the network will depend on the application and the training specifics (discussed further in section 4.1). In general, the network produces an internal latent representation z α (x) for an input x, a reconstruction signal y ≈ x to check the fidelity of the system, as well as some confidence {p α (x)} in this prediction. Each of y α = D α • E α • E(x) may be used as a proxy for y (and then each p α can be interpreted as this probability) or some combination of the y α s may be used (in which case the p α are interpreted as the partition of unity weights). In this section, we explain several task specific modeling options, loss functions, regularization, and pre-training schemes that promote training efficiency of the proposed model. The chart prediction module assigns any input to one or more charts. If the data manifold has relatively simple geometry, such a a circle, we may use the normalized distance from the data point to the center of the patch for prediction. This procedure is extremely efficient, but is not sufficiently powerful in cases where the geometry is more complex. For example, for the same surface area, a high curvature may require many small charts to cover, whereas a flat region may need only one chart. In this case we can compute p α with a deep network, denoted as the chart predictor in Figure 3, using some combination of x, z and/or z α as inputs. Using x as an input in a network which is independent of the rest of the network (and can potentially be trained separately), but the size and complexity of this network will depend on the ambient dimension of x. Using the internal representation z or z α as an input to this network allows the chart selection module to benefit from the dimension reduction of x preformed by E. We propose two loss functions which lead to two slightly different interpretations of the model, based on how to handle regions in which the charts overlap. In the first regime, we define a decoderwise loss for y α = D α • E α • E(x) as e α = x − y α 2 and an internal label α = softmax(e α). Then the Chart-Prediction Loss is given by: where Θ are network parameters. The first term models the reconstruction error of the predicted chart and the second term is the log-likelihood of the prediction, weighted by decoder-wise error. The second regime is based on the idea of partition of unity idea (see for details). Here, we view p α: M → R as a function with compact support in M, i.e. {x ∈ M | p α (x) = 0} ⊂ M, p α ≥ 0 for all α, and α p α = 1. They serve as the partition of unity (See Figure 8 for a example). Thus, we represent any point on the manifold as a combination of the charts and use the confidence weights predicted by the chart predictor as the coefficients. The loss is then given by the following Partition of Unity Loss: Since it is impossible to know a priori the number of charts necessary to cover a given data set, we instead overparameterize the model by using many charts and enforce a strong regularization on the decoder functions to eliminate unnecessary charts. Note that during training, a chart function (say E j) which is not utilized in the reconstruction of a point (i.e. p j ≈ 0) does not get any update from the loss function. Then, adding any convex penalty centered at 0 to the weights of E j will in weight decay and, if a decoder is never utilized during training, its weights will go to zero. In practice, we can automatically remove these charts by eliminating them from the network when the norm of the decoder weights falls bellow some tolerance. This mechanism provides a way of choosing the number of charts used in the network. Namely, we over estimate the number of charts and let the network automatically eliminate unnecessary ones, ing in an appropriate number. We also introduce an additional regularization to stabilize the training of our network by balancing the size of M α parameterized by E α and stopping a small number of charts from dominating the data manifold. For example, if we would like to use our network to model a (finitely sampled) sphere S 2, then we need at least two 2-dimensional charts. However, if we regularize a network with only l 2 weight decay, it may be able to reconstruct the training data well with only one chart without capturing the true structure of the data (see right panel of Figure 4 for such an example). To prevent this type of overreach, we add a Lipschitz regularization to the decoders to penalize how far away nearby inputs can be mapped. Formally, the Lipschitz constant of a function is defined as:. Since the chart spaces are fixed as l, controlling the Lipschitz constant of the encoder will control the max (euclidean) volume a chart can map onto. To do this, we note that the Lipschitz constant of a composition of functions can be upper bounded by the multiplication of the Lipschitz constants of its constituent functions. The Lipschitz constant of a matrix is its spectral norm and the Lipschitz constant of the ReLU is 1. Then, we can control the upper bound of the Lipschitz constant of an encoder function by regularizing multiplication of the spectral norm of its weights at each layer. Combining these ideas, and denoting the weights of the k th layer of E α as W k α, we propose the following regularization on the decoder functions for a K-layer network: Here the first term aims at stopping a single chart from dominating, and the second term works as a weight decay which also promotes the smoothness of each chart. Since our model jointly predicts the chart outputs and chart probabilities, it is important that our model is properly initialized, so that the range of each decoder lies somewhere on the manifold and the probability that a randomly sampled point lies in each chart is roughly equal. To do this, we begin by using the furthest point sampling (FPS) scheme to select N data points, {x α}, from the training set which are'far away' from each other. Then we assign each of these data points to a decoder and train each one to reconstruct. Additionally, we train the encoder such that x α is at the center of the chart space U α. We further define the chart prediction probability as the categorical distribution and use it to pre-train the chart predictor. Then the loss for the β th initialization example is: We can extend this idea of pre-training to also ensure that the charts are oriented consistently. Details are presented in A.4. We remark that the pre-training network does not aim at separating the data manifold as different clusters. The pre-training works to ensure that each of the decoders is on the manifold, so that when training begins no decoders stay inactive. Since the chart selection module is learned in conjunction with the rest of the model, we do not assume any prior segmentation of the data. During training the charts will move, change sizes, overlap, or disappear. We present numerical of several experiments to demonstrate the effectiveness of the proposed CAE. First, we study three illustrative geometric examples-sphere, double torus, and a genus-3 surface-to understand the behavior of the network. Afterward, we use the MINIST data set to further demonstrate the properties of CAE. In the end, we evaluate CAE and compare with other models on several metrics with the use of synethetic data sets as well as MNIST and SVHN. In our first experiment illustrated in Figure 4, we visualize the process of applying a four-chart CAE to a data set sampled from the unit sphere (see Appendix A.2 for the network architecture). We apply the proposed loss function with and without the Lipschitz regularization discussed in section 4.2. We use four copies of 2 as the chart latent space in this experiment. We color code 2 using the distance of each point to the origin. After training, we uniformly sample points on the latent space and use the learned decoders to generate points back to the unit sphere. As we can see from the middle panel of Figure 4, the four charts, when glued together, successfully cover the unit sphere. Moreover, all charts occupy the data manifold in a balanced and regularized way; that is, even thought they are not uniform, no single chart dominates the rest. From the right panel of Figure 4, we can see that, when no regularization is employed, the charts are less localized. This behavior shows the necessity of using Lipschitz regularization to control the regularity of the decoder. Our second experiment is conducted on a more complex object-the double torus-shown in Figure 1. The experiment illustrates some of the difficulties in using traditional auto-encoders to capture topologically complex data. Here, the data manifold has local dimension of 2, but it is not homeomorphic to a plane. We uniformly sample the latent space of each model and apply the ing decoder to generate points back to the ambient space. As we can see from the second left plot in Figure 1, a traditional model with a 2-dimensional single-chart latent space cannot capture the overall manifold. Since this object can be embedded in R 3, a model with a 3-dimensional latent space can capture the entire manifold. However, this type of model also likely generates points off the manifold, as we can see from the second right image in Figure 1. Finally, we see that our CAE with four 2-dimensional charts can produce points successfully covering the objects without introducing unfaithful points. Next, we test our CAE on a genus-3 surface with ten 2-dimensional charts (detailed as CAE 2 in A.2). The left of Figure 5 shows the of randomly sampling z α in the chart latent space U, and decoding the latent representations. The right of this figure shows which chart is active in each region of the manifold. Since this model uses a network to predict the chart segmentation, the ing parameterization has charts of varying sizes. This allows the network to place more charts in areas of high curvature, and allow charts to grow over more linear regions. Nevertheless, this example demonstrates the effectiveness of our method to handle objects with complex topology. We apply the 10-chart model on the MNIST data set (now using CAE 3 as detailed in A.2). The left panel of Figure 6 reports the reconstruction in the training data, for a given image showed in the second last row. Each of the first ten rows in the corresponding column shows the decoding from the i-th chart. Note that while each decoder may produce vastly different outputs, the chart selection module chooses which is most likely to be correct. As we can observe from the image, the chart selection model successfully picks up the most faithful decoding , as we circle and repeat in the last row of the image. This picture shows that the proposed multi-chart auto-encoder does provide faithful reconstruction for the training data. The middle panel of Figure 6 shows decoding by sampling the charts, where each row shows images generated from the corresponding decoder. Note that each chart produces only a few digits, even though every digit is covered by some chart. Additionally, on any chart the digits which that chart produces are "close" to each other (for example the 3s and 8s in chart 8 and the 5s and 6s in chart 1). This means the multi-chart latent space can cover the MINST data manifold in a balanced and regular way, similar to what we observe from the experiments conducted for geometric objects. The right panel of this figure shows the morphing of a'2' to a'3' by interpolating a linear path through the latent space. Since each of the latent representations decoded along this path produces output similar to examples found in the training set, we can conclude that approximation of the manifold given by our chart parameterization is close to the underlying distribution the training data is sampled from. In traditional approaches this is possible because the latent space (modeled as a normal distribution) is simply connected. Our model is able to do so without using a distributional assumptions, owing to the transition conditions and Lipshitz regularization. In this experiment, we apply four traditional models (2 auto-encoders and 2 variational autoencoders) as well as three CAEs on several data sets. Details of the exact architecture of these networks can be found in A.2. For each model and data set, we are primarily interested in three measures of success, including reconstruction error, unfaithfulness, and coverage (See A.10 for detailed definitions). The reconstruction error measures the fidelity of the model. The unfaithfulness measures how far synthesized data decoded from samples drawn on the latent space are to samples from the original training data. Coverage indicates how much of the training data is covered by the encoder. Models which produce unrealistic data when sampling from the latent space will have high unfaithfulness sores and models which experience mode collapse will have low coverage scores. We test these measurements on four data sets, in the order from the simplest to the most complex. 1 Sphere: The data consists of 2000 equally distributed points sampled uniformly form a sphere embedded in R 3. 2 Genus 3: The genus-3 object used in Figure 5 non-trivally embedded in The MNIST hand-written digits database containing 60k training and 10k testing images . 4 SVHN: A real-world image dataset from house numbers in Google Street View images. We focus on the individual digits problem and preprocess the images in gray scale . Results of the the evaluation metrics are summarized in Figure 7 and reported fully in Table A.10 in A.10. From these , clearly the CAE models consistently preform better than other models with simple latent spaces. More specifically, when the dimension of the latent space is fixed, the CAE model preforms better than the associated VAE and AE in every test. Moreover, because of the Lipschitz regularization in our model, we are able to achieve much better coverage than with the previous methods. We have proposed and investigated the use of chart-based paramterization to model manifold structured data, through introducing multiple-chart latent spaces, along with transition functions, to autoencoders. The parameterization allows us to significantly reduce the dimension of latent encoding for efficiently representing data with complex structures. Numerically, we design geometric examples to analyze the behavior of the proposed CAE and illustrate its advantage over single-chart Figure 7: Summary of benchmark test on Sphere, Genus-3, MNIST and SVHN data sets autoencoders. We also apply our method to real-life data sets, including MNIST and SVHN, to demonstrate the effectiveness of the proposed model. We believe that the proposed chart-based parameterization of manifold-structured data provides many opportunities for further analysis and applications. In future work, we will extend this architecture to other generative models (e.g, GAN) and apply the machinery to investigate the topology and geometry of real-world data. In this section we detail the architecture of the networks used in the numerical experiments. We denote fully connected layers as F C y where y is the number of units in the layers, Conv i,j,k.l as convolution layers with filters of size(i, j) input dimension k and output dimension l, and dim(U) to be the dimension of the latent space. Each model was trained using the chart prediction loss function as we have found it to be more stable during training. Variational Auto-Encoders: CAE 1 (4 2-dim charts, distance confidence, chart predictor): CAE 2 (10 2-dim charts, learned confidence chart predictor) CAE 3 (10 25-dim charts, convolution layers, learned chart predictor) A key feature of the chart-based parameterization of manifolds in differential geometry is the construction of chart transition functions. As show in figure 2, some points on the manifold may be parameterized by multiple charts. Let φ α and φ β be two charts with overlapping domains (M α ∩ M β = ∅), then the chart transition function φ αβ can be computed as: φ −1 α φ β. In our model the φ −1 s are represented as E i neural networks, but directly computing φ itself from φ −1 is not simple since ReLU-nets are non-invertable. It would be possible to add additional modules and to train a model to predict the transition function, but this adds up to N 2 new networks to train, many of which may be unnecessary (since we only need chart transition function for overlapping charts). However, we can exploit the structure the encoder module and re-encode the signal generated by the first decoder, using the second encoder define a chart transition. Then Each chart transition function can be modeled by the composition: Note that if x ∈ M α ∪ M β, then to estimate the chart transition between U α and U β we need: Each of these conditions are naturally enforced by both of loss functions equation 2, equation 3 discussed in the precious section. Therefore the chart transition function of this network can be computed without explicitly parameterizing them or adding new terms to the loss function. One could explicitly characterize the the chart transition by re-encoding the decoded signals in a second pass thought the network and computing an regularizer: which measures the exactly error in the transition and reconstruction. However this type of cyclic condition is computationally expensive to implement, and we have found it unnecessary in our empiric studies. We can further extend the idea of pre-training to also orient the rest of the chart around the center c α. To do so, we take a small sample of points N (c α) around the center and use principle component analysis (PCA) to define a d-dimensional embedding of the local neighborhood. Let the coordinates of this neighborhood embedding be: and C i is chosen as a local scaling constant. Then we can use this coordinate system to initialize the orientation of the local charts by adding an additional regularization the term to the equation 5: A.5 CONVOLUTION It has been widely argued that invariant and equivalent properties of convolutions layers promote manifold structured representations. For example, conjectures:'that in a representation obtained as an output of convolutional and pooling layers, the data concentrates near a collection of low-dimensional manifolds embedded in a high-dimensional space.' In other words, applying dimension reduction operations which has localized invariance (such as convolution and pooling) maps data to relatively simple manifolds by contracting the representation space. This suggests that adding convolution and pooling layers to the beginning of the encoder networks will in representations which are easier for our model to estimate since the underlying geometry will be simpler. We write D as a general notation for the decoder in the model. Measures fidelity of the output y: faithfulness Error Measures how close data decoded from samples drawn on the latent space are to samples from the original training data. We uniformly selecting M points in the latent space and define 2. Often, people hope that through sampling the latent space and decoding, one can get new data in the original space that novel or original. However, if the training set is sufficiently dense on the data manifold, newly generated data is far from anything observed during training will not be realistic. Since we are concerned with fitting a model to the data, a properly trained model should always stay close the the underlying manifold. To select the latent variables used in this experiment, we uniformly draw M = 100 samples from the latent space used in the single chart auto-encoder, CAE and VAE. Coverage Measures how well the latent space and decoders capture the entire data set. We uniformly draw M samples {z i} M i=1 from the latent space. Let M * be the number of the set {x M . This measurement provides quantitative way to describe the well known "mode collapse" problem of GAN wherein some the model captures part of the data well, but ignores large sections (or entire classes) of the data. A coverage score close to one indicates that the samples are well distributed on the manifold, while scores close to zero indicate that the model may be experiencing mode collapse. In this experiment, we train a model with four 1-dimensional charts to fit a circle in order to visualize the transition between charts. In figure 8, the first row shows the output of each chart using the latent variable z i sampled on. The top right shows the chart which has the largest p value. In the second row, we visualize the transition zones. Here, the solid colored lines are decoded samples taken from the U i space. The'+'s represent training data, and their color indicates which chart had maximum p value. The ground truth manifold is represented by the dashed lined. The last row show the partition of unity function, unwrapped from the circle to a line. From this experiment we can see that charts have very close values in these transition zones and the partition of unity functions are compactly supported. In this experiment, we train a model with four 2-dimensional charts for data sampled on a sphere in order to visualize the effect of the regularization scheme when using a neural network as the chart prediction module. Figure 9 shows the patch prediction function on the training data at the end of the pre-training and at the completion of training, respectively. From this figure we see that even though all charts cover the sphere at the beginning, the regularization is able to automatically remove some charts after training. In this experiment, we demonstrate a simple example of recovering geometric information from a trained model by computing the length of some geodesic curves on a sphere. Let p and q be points on the manifold with latent representations z p and z q. Then the line γ z (t) = αz p + (1 − α)z q in the latent space will correspond to a path on γ ⊂ M. To measure the length of this path, we can sample points along γ z (t), decode them and then measure the euclidean distance between the decoded points. Figure 10 shows an example of such a test using different numbers of sampling points for five difference curves on the sphere. From this experiment we observe convergence of these measurements as more points are sampled along the geodesic path, validating our geometric intuition. We remark that this is a very preliminary to show a potential of understanding geometric structure of data manifold using multi-chart latent space. We will explore in the direction in our future work. A.10 DETAILED BENCHMARK COMPARISONS Figure 11 shows an experiment for data sampled on a double torus using VAE with different choice of parameters. The latent space dimension is chosed as 2 which is compatible with the intrinsic diemisno of the object. This experiment shows that increasing the number of parameters in a VAE alone (without increasing the latent dimension) does not allow one to simultaneously produce good reconstruction and generation. A latent space which has too small of a dimension will not be able to cover a manifold, and one which is too large will generate points far from the data manifold. Thus the structure of latent space is more important than the number of parameters. This is one of the main objectives of this paper.

Manifold-structured latent space for generative models

Represent the short summary for finding the corresponding scientific article (long)

We train in random subspaces of parameter space to measure how many dimensions are really needed to find a solution.