Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Predicting the future in real-world settings, particularly from raw sensory observations such as images, is exceptionally challenging. Real-world events can be stochastic and unpredictable, and the high dimensionality and complexity of natural images requires the predictive model to build an intricate understanding of the natural world. Many existing methods tackle this problem by making simplifying assumptions about the environment. One common assumption is that the outcome is deterministic and there is only one plausible future. This can lead to low-quality predictions in real-world settings with stochastic dynamics. In this paper, we develop a stochastic variational video prediction (SV2P) method that predicts a different possible future for each sample of its latent variables. To the best of our knowledge, our model is the first to provide effective stochastic multi-frame prediction for real-world video. We demonstrate the capability of the proposed method in predicting detailed future frames of videos on multiple real-world datasets, both action-free and action-conditioned. We find that our proposed method produces substantially improved video predictions when compared to the same model without stochasticity, and to other stochastic video prediction methods. Our SV2P implementation will be open sourced upon publication. Understanding the interaction dynamics of objects and predicting what happens next is one of the key capabilities of humans which we heavily rely on to make decisions in everyday life BID3. A model that can accurately predict future observations of complex sensory modalities such as vision must internally represent the complex dynamics of real-world objects and people, and therefore is more likely to acquire a representation that can be used for a variety of visual perception tasks, such as object tracking and action recognition BID31 BID25 BID7. Furthermore, such models can be inherently useful themselves, for example, to allow an autonomous agent or robot to decide how to interact with the world to bring about a desired outcome BID27.However, modeling future distributions over images is a challenging task, given the high dimensionality of the data and the complex dynamics of the environment. Hence, it is common to make various simplifying assumptions. One particularly common assumption is that the environment is deterministic and that there is only one possible future BID5 BID31 BID1 BID25. Models conditioned on the actions of an agent frequently make this assumption, since the world is more deterministic in these settings BID27 BID10. However, most real-world prediction tasks, including the action-conditioned settings, are in fact not deterministic, and a deterministic model can lose many of the nuances that are present in real physical interactions. Given the stochastic nature of video prediction, any deterministic model is obliged to predict a statistic of all the possible outcomes. For example, deterministic models trained with a mean squared error loss function generate the expected value of all the possibilities for each pixel independently, which is inherently blurry BID26. Figure 1: Importance of stochasticity in video prediction. In each video, a random shape follows a random direction (first row). Given only the first frame, the deterministic model from BID10 predicts the average of all the possibilities. The third row is the output of SV2P with latent sampled from approximated posterior which predicts the correct motion. Last two rows are stochastic outcomes using random latent values sampled from assumed prior. As observed, these outcomes are random but within the range of possible futures. Second sample of Figure 1c shows a case where the model predicts the average of more than one outcome. Our main contribution in this paper is a stochastic variational method for video prediction, named SV2P, that predicts a different plausible future for each sample of its latent random variables. We also provide a stable training procedure for training a neural network based implementation of this method. To the extent of our knowledge, SV2P is the first latent variable model to successfully predict multiple frames in real-world settings. Our model also supports action-conditioned predictions, while still being able to predict stochastic outcomes of ambiguous actions, as exemplified in our experiments. We evaluate SV2P on multiple real-world video datasets, as well as a carefully designed toy dataset that highlights the importance of stochasticity in video prediction (see Figure 1). In both our qualitative and quantitative comparisons, SV2P produces substantially improved video predictions when compared to the same model without stochasticity, with respect to standard metrics such as PSNR and SSIM. The stochastic nature of SV2P is most apparent when viewing the predicted videos. Therefore, we highly encourage the reader to check the project website https://goo.gl/iywUHc to view the actual videos of the experiments. The TensorFlow BID0 implementation of this project will be open sourced upon publication. A number of prior works have addressed video frame prediction while assuming deterministic environments BID28 BID31 BID34 BID38 BID1 BID25. In this work, we build on the deterministic video prediction model proposed by BID10, which generates the future frames by predicting the motion flow of dynamically masked out objects extracted from the previous frames. Similar transformationbased models were also proposed by BID6. Prior work has also considered alternative objectives for deterministic video prediction models to mitigate the blurriness of the predicted frames and produce sharper predictions BID26 BID33. Despite the adversarial objective, BID26 found that injecting noise did not lead to stochastic predictions, even for predicting a single frame. BID27; BID5 make sharp video predictions by assuming deterministic outcomes in video games given the actions of the agents. However, this assumption does not hold in real-world settings, which almost always have stochastic dynamics. Auto-regressive models have been proposed for modeling the joint distribution of the raw pixels. Although these models predict sharp images of the future, their training and inference time is extremely high, making them difficult to use in practice. BID29 proposed a parallelized multi-scale algorithm that significantly improves the training and prediction time but still requires more than a minute to generate one second of 64×64 video on a GPU. Our comparisons suggest that the predictions from these models are sharp, but noisy, and that our method produces substantially better predictions, especially for longer horizons. Another approach for stochastic prediction uses generative adversarial networks (GANs) BID14, which have been used for video generation and prediction BID32. BID35; applied adversarial training to predict video from a single image. Although GANs generate sharp images, they tend to suffer from modecollapse BID13, particularly in conditional generation settings BID40.Variational auto-encoders (VAEs) BID21 ) also have been explored for stochastic prediction tasks. BID36 uses conditional VAEs to predict dense trajectories from pixels. BID39 predicts a single stochastic frame using cross convolutional networks in a VAElike architecture. BID30 uses conditional VAEs and Gaussian mixture priors for stochastic prediction. Both of these works have been evaluated solely on synthetic datasets with simple moving sprites and no object interaction. Real images significantly complicate video prediction due to the diversity and variety of stochastic events that can occur. BID11 compared various architectures for multimodal motion forecasting and one-frame video prediction, including variational inference and straightforward sampling from the prior. Unlike these prior models, our focus is on designing a multi-frame video prediction model to produce stochastic predictions of the future. Multi-frame prediction is dramatically harder than single-frame prediction, since complex events such as collisions require multiple frames to fully resolve, and single-frame predictions can simply ignore this complexity. We believe, our approach is the first latent variable model to successfully demonstrate stochastic multi-frame video prediction on real world datasets. In order to construct our stochastic variational video prediction model, we first formulate a probabilistic graphical model that explains the stochasticity in the video. Since our goal is to perform conditional video prediction, the predictions are conditioned on a set of c context frames x 0,..., x c−1 (e.g., if we are conditioning on one frame, c = 1), and our goal is to sample from p(x c:T |x 0:c−1), where x i denotes the i th frame of the video FIG1 ). Video prediction is stochastic as a consequence of the latent events that are not observable from the context frames alone. For example, when a robot's arm pushes a toy on a table, the unknown weight of that toy affects how it moves. We therefore introduce a vector of latent variables z into our model, distributed according to a prior z ∼ p(z), and build a model p(x c:T |x 0:c−1, z). This model is still stochastic but uses a more general representation, such as a conditional Gaussian, to explain just the noise in the image, while z accounts for the more complex stochastic phenomena. We can then factorize this model to T t=c p θ (x t |x 0:t−1, z). Learning then involves training the parameters of these factors θ, which we assume to be shared between all the time steps. At inference time we need to estimate values for the true posterior p(z|x 0:T), which is intractable due its dependency on p(x 0:T). We overcome this problem by approximating the posterior with an inference network q φ (z|x 0:T) that outputs the parameters of a conditionally Gaussian distribution N (µ φ (x 0:T), σ φ (x 0:T)). This net- work is trained using the reparameterization trick BID21, according to: DISPLAYFORM0 Here, θ and φ are the parameters of the generative model and inference network, respectively. To learn these parameters, we can optimize the variational lower bound, as in the variational autoencoder (VAE) BID21: DISPLAYFORM1 where D KL is the Kullback-Leibler divergence between the approximated posterior and assumed prior p(z) which in our case is the standard Gaussian N (0, I).In Equation 2, the first term on the RHS represents the reconstruction loss while the second term represents the divergence of the variational posterior from the prior on the latent variable. It is important to emphasize that the approximated posterior is conditioned on all of the frames, including the future frames x t:T. This is feasible during training, since x t:T is available at the training time, while at test time we can sample the latents from the assumed prior. Since the aim in our method is to recover latent variables that correspond to events which might explain the variability in the videos, we found that it is in fact crucial to condition the inference network on future frames. At test time, the latent variables are simply sampled from the prior which corresponds to a smoothing-like inference process. In principle, we could also perform a filtering-like inference procedure of the form q φ (z|x 0:t−1) for time step t to infer the most likely latent variables based only on the conditioning frames, instead of sampling from the prior, which could produce more accurate predictions at test time. However, it would be undesirable to use a filtering process at training time: in order to incentivize the forward prediction network to make use of the latent variables, they must contain some information that is useful for predicting future frames that is not already present in the context frames. If they are predicted entirely from the context frames, no such information is present, and indeed we found that a purely filtering-based model simply ignores the latent variables. So far, we've assumed that the latent events are constant over the entire video. We can relax this assumption by conditioning prediction on a time-variant latent variable z t that is sampled at every time step from p(z). The generative model then becomes p(z t)T t=c p θ (x t |x 0:t−1, z t) and, assuming a fixed posterior, the inference model will be approximated by q φ (z t |x 0:T), where the model parameters φ are shared across time. In practice, the only difference between these two formulations is the frequency of sampling z from p(z) and q φ (z|x 0:T). In the time-invariant version, we sample z once per video, whereas with the time-variant latent, sampling happens every frame. The main benefit of time-variant latent variable is better generalization beyond T, since the model does not have to encode all the events of the video in one vector z. We provide an empirical comparison of these formulations in Section 5.2. With latent loss Finn el al. Naive Training (Only Step 3)Step 1 Then 3Step FORMULA1 In the first phase, the inference network is turned off and only the generative network is being trained, ing in deterministic predictions. The inference network is used in the second phase without a KL-loss. The last phase includes D KL q φ (z|x 0:T)||p(z) to enable accurate sampling latent from p(z). (a) the KL-loss (b) the reconstruction loss (c) Training stability. This graph compares reconstruction loss at the end of five training sessions on the BAIR robot pushing dataset, with and without following all the steps of the training procedure. The proposed training is quite stable and in lower error compared to naïve training. In action-conditioned settings, we modify the generative model to be conditioned on action vector a t. This in p(z t)T t=c p θ (x t |x 0:t−1, z t, a t) as generative model while keeping the posterior approximation intact. Conditioning the outcome on actions can decrease future variability; however it will not eliminate it if the environment is inherently stochastic or the actions are ambiguous. In this case, the model is still capable of predicting stochastic outcomes in a narrower range of possibilities. To model the approximated posterior q φ (z|x 0:T) we used a deep convolutional neural network as shown in the top row of FIG2. Since we assumed a diagonal Gaussian distribution for q φ (z|x 0:T), this network outputs the mean µ φ (x 0:T) and standard deviation log σ φ (x 0:T) of the approximated posterior. Hence the entire inference network is convolutional, the predicted parameters are 8×8 single channel response maps. We assume each entry in this response maps is pairwise independent, forming the latent vector z. The latent value is then sampled using Equation 1. As discussed before, this sampling happens every frame for time-varying latent, and once per video in time-invariant case. For p(x t |x 0:t−1, z), we used the CDNA architecture proposed by BID10, which is a deterministic convolutional recurrent network that predicts the next frame x t given the previous frame x t−1 and an optional action a t. This model constructs the next frames by predicting the motions of segments of the image (i.e., objects) and then merging these predictions via masking. Although this model directly outputs pixels, it is partially-appearance invariant and can generalize to unseen objects BID10. To condition on the latent value, we modify the CDNA architecture by stacking z t as an additional channel on tiled action a t. Our model can be trained end-to-end. However, our experiments show that naïve training usually in the model ignoring the latent variables and converging to a suboptimal deterministic solution FIG4. Therefore, we train the model end-to-end in three phases, as follows:1. Training the generative network: In this phase, the inference network has been disabled and the latent value z will be randomly sampled from N (0, I). The intuition behind this phase is to train the generative model to predict the future frames deterministically (i.e. modeling p θ (x t |x 0:t−1)). In the second phase, the inference network is trained to estimate the approximate posterior q φ (z|x 0:T); however, the KL-loss is set to 0. This means that the model can use the latent value without being penalized for diverging from p(z). As seen in FIG4, this phase in very low reconstruction error, however it is not usable at the test time since D KL q φ (z|x 0:T)||p(z) 0 and sampling z from the assumed prior will be inaccurate. In the last phase, the KL-loss is added, ing in a sudden drop of KLdivergence and an increase of reconstruction error. The reconstruction loss converging to a value lower than the first phase and KL-loss converging to zero are indicators of successful training. This means that z can be sampled from p(z) at test time for effective stochastic prediction. To gradually transition from the second phase to the third, we add a multiplier to KL-loss that is set to zero during the first two phases and then increased slowly in the last phase. This is similar to the β hyper-parameter in BID15 and BID2 that is used to balance latent channel capacity and independence constraints with reconstruction accuracy. We found that this training procedure is quite stable and the model almost always converges to the desired parameters. To demonstrate this stability, we trained the model with and without the proposed training procedure, five times each. FIG4 shows the average and standard deviation of reconstruction loss at the end of these training sessions. Naïve training in a slightly better error compared to BID10, but with high variance. When following the proposed training algorithm, the model consistently converges to a much lower reconstruction error. To highlight the importance of stochasticity in video prediction, we created a toy video dataset with intentionally stochastic motion. Each video in this dataset is four frames long. The first frame contains a random shape (triangle, rectangle or circle) with random size and color, centered in the frame, which then randomly moves to one of the eight directions (up, down, left, right, up-left, upright, down-left, down-right). Each frame is 64×64×3 and the is static gray. The main intuition behind this design is that, given only the first frame, a model can figure out the shape, color, and size of the moving object, but not its movement direction. We train BID10 and SV2P to predict the future frames, given only the first frame. Figure 1 shows the video predictions from these two models. Since BID10 is a deterministic model with mean squared error as loss, it predicts the average of all possible outcomes, as expected. In contrast, SV2P predicts different possible futures for each sample of the latent variable z ∼ N (0, I). In our experiments, all the videos predicted by SV2P are within the range of plausible futures (e.g. we never saw the shape moves in any direction other than the original eight). However, in some cases, SV2P still predicts the average of more than one future, as it can be seen in the first random sample of Figure 1c. The main reason for this problem seems to be overlapping posterior distributions in latent space which can cause some latent values (sampled from p(z)) to be ambiguous. To demonstrate that the inference network is working properly and that the latent variable does indeed learn to store the information necessary for stochastic prediction (i.e., the direction of movement), we include predicted futures when z ∼ q φ (x 0:T). By estimating the correct parameters of the latent distribution, using the inference network, the model always generates the right outcome. However, this cannot be used in practice, since the inference network requires access to all the frames, including the ones in the future. Instead, z will be sampled from assumed prior p(z). To evaluate SV2P, we test it on three real-world video datasets by comparing it to the CDNA model BID10, as a deterministic baseline, as well as a baseline that outputs the last seen frame as the prediction. We compare SV2P with an auto-regressive stochastic model, video pixel networks (VPN). We use the parallel multi-resolution implementation of VPN from BID29, which is an order of magnitude faster than the original VPN, but still requires more than a minute to generate one second of 64×64 video. In all of these experiments, we plot the of sampling the latent once per video (SV2P time-invariant latent) and once per frame (SV2P time-variant latent). We strongly encourage readers to view https://goo.gl/iywUHc for videos of the which are more illustrative than printed frames. We quantitatively and qualitatively evaluate SV2P on following real-world datasets:• BAIR robot pushing dataset BID8: This dataset contains action-conditioned videos collected by a Sawyer robotic arm pushing a variety of objects. All of the videos in this datasets have similar table top settings with static . Each video also has recorded actions taken by the robotic arm which correspond to the commanded gripper pose. An interesting property of this dataset is the fact that the arm movements are quite unpredictable in the absence of actions (compared to the robot pushing dataset BID10 which the arm moves to the center of the bin). For this dataset, we train the models to predict the next ten frames given the first two, both in action-conditioned and action-free settings.• Human3.6M BID18: Humans and animals are one of the most interesting sources of stochasticity in natural videos, which behave in complex ways as a consequence of unpredictable intentions. To study human motion prediction, we use the Human3.6M dataset which consists of actors performing various actions in a room. We used the pre-processing and testing format of BID10: a 10 Hz frame rate and 10-frame prediction given the previous ten. The videos from this datasets contains various actions performed by humans (walking, talking on the phone, . . .). Similar to BID10, we included videos from all the performed actions in training dataset while keeping all the videos from an specific actor out for testing.• Robotic pushing prediction BID10: We use the robot pushing prediction dataset to compare SV2P with another stochastic prediction method, video pixel networks (VPNs). VPNs demonstrated excellent on this dataset in prior work, and therefore robot pushing dataset provides a strong point of comparison. However, in contrast to our method, VPNs do not include latent stochastic variables that represent random events, and rely on an expensive auto-regressive architecture. In this experiment, the models have been trained to predict the next ten frames, given the first two. Similar to BAIR robot pushing dataset, this dataset also contains actions taken by the robotic arm which are the pose of the commanded gripper. Figure 5: Stochasticity of SV2P predictions on the action-free BAIR dataset. Each line presents the sample with highest PSNR compared to ground truth, after multiple sampling. The number on the right indicates the number of random samples. As can be seen, SV2P predicts highly stochastic videos and, on average, only three samples is enough to predict outcomes with higher quality compared to BID10. In our quantitative evaluation, we aim to understand whether the range of possible futures captured by our stochastic model includes the true future. Models that are more stochastic do not necessarily score better on average standard metrics such as PSNR BID17 and SSIM BID37. However, if we are interested primarily in understanding whether the true outcome is within the set of predictions, we can instead evaluate the score of the best sample from multiple random priors. We argue that this is a better metric for stochastic models, since it allows us to understand if uncertain futures contain the true outcome. Figure 5 illustrates how this metric changes with different numbers of samples. By predicting more possible futures, the probability of predicting the true outcome increases, and therefore it is more likely to get a sample with higher PSNR compared to the ground truth. Of course, as with all video prediction metrics, it is imperfect, and is only suitable for understanding the performance of the model when combined with a visual examination of the qualitative in Section 5.3.To use this metric, we sample 100 latent values from prior z ∼ N (0, I) and use them to predict 100 videos and show the of the sample with highest PSNR. For a fair comparison to VPN, we use the same best out of 100 samples for our stochastic baseline. Since even the fast implementation of VPN is quite slow, we limit the comparison with VPN to only last dataset with 256 test samples. Overall, SV2P with both time-variant and time-invariant latent sampling outperform all of the other baselines, by predicting higher quality videos with higher PSNR and SSIM. Time-varying latent sampling is more stable beyond the time horizon used during training FIG6 ). One possible explanation for this behaviour is that the time-invariant latent has to include the information required for predicting all the frames and therefore, beyond training time, it collapses. This issue is mitigated by a time-variant latent variable which takes a different value at each time step. However, this stability is not always the case as it is more evident in late frames of FIG6.One other interesting observation is that the time-invariant model outperforms the time-variant model in the Human3.6M dataset. In this dataset, the most important latent event -the action performed by the actor -is consistent across the whole video which is easier to capture using timeinvariant latent. We can better understand the performance of the proposed model by visual examination of the qualitative . We highlight some of the most important and observable differences in predictions by different models in FIG9 1. In all of these figures, the x-axis is time (i.e., each row is one video). The first row is the ground truth video, and the second row is the of BID10. The of sampling the latent from approximated posterior is provided in the third row. For stochastic methods, we show the best (highest PSNR) and worst (lowest PSNR) predictions out of 100 samples (as discussed in Section 5.2), as well as two random predicted videos from our model. FIG9 illustrates two examples from the BAIR robot pushing dataset in the action-free setting. As a consequence of the high stochasticity in the movement of the arm in absence of actions, BID10 only blurs the arm out, while SV2P predicts varied but coherent movements of the arm. Note that, although each predicted movements of the arm is random, it is still in the valid range of possible outcomes (i.e., there is no sudden jump of the arm nor random movement of the objects). The proposed model also learned how to move objects in cases where they have been pushed by the predicted movements of the arm, as can be seen in the zoomed images of both samples. Quantitative comparison of the predicted frames on Human3.6M dataset using confidence of object detection as quality metric. The y-axis demonstrates the average confidence of BID16 in detecting humans in predicted frames. Based on this metric, SV2P predicts images with more meaningful semantics compared to to BID10.In the action-conditioned setting FIG10 ), the differences are more subtle: the range of possible outcomes is narrower, but we can still observe stochasticity in the behavior of the pushed objects. Interactions between the arm and objects are uncertain due to ambiguity in depth, friction, and mass, and SV2P is able to capture some of this variation. Since these variations are subtle and occupy a smaller part of the images, we illustrate this with zoomed insets in FIG10. Some examples of varied object movements can be found in last three rows of right example of FIG10. SV2P also generates sharper outputs, compared to BID10 as is evident in the left example of FIG10.Please note that the approximate posterior q φ (z|x 0:T) is still trained with the evidence lower bound (ELBO), which means that the posterior must compress the information of the future events. Perfect reconstruction of high-quality images from posterior distributions over latent states is an open problem, and the in our experiments compare favorably to those typically observed even in single-image VAEs (e.g. see BID39). This is why the model cannot reconstruct all the future frames perfectly, even though when latent values are sampled from q φ (z|x 0:T).Figure 10 displays two examples from the Human3.6M dataset. In absence of actions, BID10 manages to separate the foreground from , but cannot predict what happens next accurately. This in distorted or blurred foregrounds. On the other hand, SV2P predicts a variety of different outcomes, and moves the actor accordingly. Note that PSNR and SSIM are measuring reconstruction loss with respect to the ground truth and they may not generally present a better prediction. For some applications, a prediction with lower PSNR/SSIM might have higher quality and be more interesting. A good example is the prediction with the worst PSNR in Figure 10 right, where the model predicts that the actor is spinning in his chair with relatively high quality. However, this output has the lowest PSNR compared to the ground truth. However, pixel-wise metrics such as PSNR and SSIM may not be the best measures for semantic evaluation of predicted frames. Therefore, we use the confidence of an object detector to show the predicted frames contain useful semantic information. For this purpose, we use the open-sourced implementation of BID16 to compare the quality of predicted frames in Human3.6M dataset. As it can be seen in FIG8, SV2P predicted frames which the human inside can be detected with higher confidence, compared to BID10.Finally, FIG11 demonstrates on the Google robot pushing dataset. The qualitative and quantitative in FIG11 and 6 both indicate that SV2P produces substantially better predictions than VPNs. The quantitative suggest that our best-of-100 metric is a reasonable measure of performance: the VPN predictions are more noisy, but simply increasing noise is not sufficient to increase the quality of the best sample. The stochasticity in our predictions is more coherent, corresponding to differences in object or arm motion, while much of the stochasticity in the VPN predictions resembles noise in the image, as well as visible artifacts when predicting for substantially longer time horizons. We proposed stochastic variational video prediction (SV2P), an approach for multi-step video prediction based on variational inference. Our primary contributions include an effective stochastic prediction method with latent variables, a network architecture that succeeds on natural videos, and a training procedure that provides for stable optimization. The source code for our method will be released upon acceptance. We evaluated our proposed method on three real-world datasets in actionconditioned and action-free settings, as well as one toy dataset which has been carefully designed to highlight the importance of the stochasticity in video prediction. Both qualitative and quantitative indicate higher quality predictions compared to other deterministic and stochastic baselines. SV2P can be expanded in numerous ways. First, the current inference network design is fully convolutional, which exposes multiple limitations, such as unmodeled spatial correlations between the latent variables. The model could be improved by incorporating the spatial correlation induced by the convolutions into the prior, using a learned structured prior in place of the standard spherical Gaussian. Time-variant posterior approximation to reflect the new information that is revealed as the video progresses, is another possible SV2P improvement. However, as discussed in Section 3, this requires incentivizing the inference network to incorporate the latent information at training time. This would allow time-variant latent distributions which is more aligned with generative neural models for time-series BID19 BID12 BID22.Another exciting direction for future research would be to study how stochastic predictions can be used to act in the real world, producing model-based reinforcement learning methods that can execute risk-sensitive behaviors from raw image observations. Accounting for risk in this way could be especially important in safety-critical settings, such as robotics. In lack of actions and therefore high stochasticity, BID10 only blurs the robotic arm out while the proposed method predicts sharper frames on each sampling. SV2P also predicts the interaction dynamics between random movements of the arm and the objects. BID10. This is mostly evident in zoomed in objects which have been pushed by the arm. Figure 10: Prediction on the action-free Human3.6M dataset. SV2P predicts a different outcome on each sampling given the latent. In the left example, the model predicts walking as well as stopping which in different outputs in predicted future frames. Similarly, the right example demonstrates various outcomes including spinning. BID29 with SV2P on the robotic pushing dataset. We use the same best PSNR out of 100 random samples for both methods. Besides stochastic movements of the pushed objects, another source of stochasticity is the starting lag in movements of the robotic arm. SV2P generates sharper images compared to BID10 (notice the pushed objects in zoomed images) with less noise compared to BID29 (look at the accumulated noise in later frames).A TRAINING DETAILS FIG2 contains details of the network architectures used as generative and inference models. In all of the experiments we used the same set of hyper-parameters which can be found in TAB1. In the first step of training, we disable the inference network and instead sample latent values from N (0, I). In step 2, the latent values will be sampled from the approximated posterior q φ (z|x 0:T) = N µ(x 0:T), σ(x 0:T). Please note that the inference network approximates log(σ) instead of σ for numerical stability. To gradually switch from Step 2 of training procedure to Step 3, we increase β linearly from its starting value to its end value over the length of training.

Represent the TLDR for finding the corresponding scientific article

Stochastic variational video prediction in real-world settings.

Represent the short summary for finding the corresponding scientific article (long)

We propose a novel way to incorporate conditional image information into the discriminator of GANs using feature fusion that can be used for structured prediction tasks.


Question: Represent the article for finding a single-sentence TLDR

Question: In multi-agent systems, complex interacting behaviors arise due to the high correlations among agents. However, previous work on modeling multi-agent interactions from demonstrations is primarily constrained by assuming the independence among policies and their reward structures. In this paper, we cast the multi-agent interactions modeling problem into a multi-agent imitation learning framework with explicit modeling of correlated policies by approximating opponents’ policies, which can recover agents' policies that can regenerate similar interactions. Consequently, we develop a Decentralized Adversarial Imitation Learning algorithm with Correlated policies (CoDAIL), which allows for decentralized training and execution. Various experiments demonstrate that CoDAIL can better regenerate complex interactions close to the demonstrators and outperforms state-of-the-art multi-agent imitation learning methods. Our code is available at \url{https://github.com/apexrl/CoDAIL}. Modeling complex interactions among intelligent agents from the real world is essential for understanding and creating intelligent multi-agent behaviors, which is typically formulated as a multiagent learning (MAL) problem in multi-agent systems. When the system dynamics are agnostic and non-stationary due to the adaptive agents with implicit goals, multi-agent reinforcement learning (MARL) is the most commonly used technique for MAL. MARL has recently drawn much attention and achieved impressive progress on various non-trivial tasks, such as multi-player strategy games , traffic light control , taxi-order dispatching etc. A central challenge in MARL is to specify a good learning goal, as the agents' rewards are correlated and thus cannot be maximized independently . Without explicit access to the reward signals, imitation learning could be the most intuitive solution for learning good policies directly from demonstrations. Conventional solutions such as behavior cloning (BC) learn the policy in a supervised manner by requiring numerous data while suffering from compounding error . Inverse reinforcement learning (IRL) alleviates these shortcomings by recovering a reward function but is always expensive to obtain the optimal policy due to the forward reinforcement learning procedure in an inner loop. Generative adversarial imitation learning (GAIL) leaves a better candidate for its model-free structure without compounding error, which is highly effective and scalable. However, real-world multi-agent interactions could be much challenging to imitate because of the strong correlations among adaptive agents' policies and rewards. Consider if a football coach wants to win the league, he must make targeted tactics against various opponents, in addition to the situation of his team. Moreover, the multi-agent environment tends to give rise to more severe compounding errors with more expensive running costs. Motivated by these challenges, we investigate the problem of modeling complicated multi-agent interactions from a pile of off-line demonstrations and recover their on-line policies, which can regenerate analogous multi-agent behaviors. Prior studies for multi-agent imitation learning typically limit the complexity in demonstrated interactions by assuming isolated reward structures (; ; ;) and independence in per-agent policies that overlook the high correlations among agents . In this paper, we cast the multi-agent interactions modeling problem into a multi-agent imitation learning framework with correlated policies by approximating opponents' policies, in order to reach inaccessible opponents' actions due to concurrently execution of actions among agents when making decisions. Consequently, with approximated opponents model, we develop a Decentralized Adversarial Imitation Learning algorithm with Correlated policies (CoDAIL) suitable for learning correlated policies under our proposed framework, which allows for decentralized training and execution. We prove that our framework treats the demonstrator interactions as one of -Nash Equilibrium (-NE) solutions under the recovered reward. In experiments, we conduct multi-dimensional comparisons for both the reward gap between learned agents and demonstrators, along with the distribution divergence between demonstrations and regenerated interacted trajectories from learned policies. Furthermore, the reveal that CoDAIL can better recover correlated multi-agent policy interactions than other state-of-the-art multi-agent imitation learning methods in several multi-agent scenarios. We further illustrate the distributions of regenerated interactions, which indicates that CoDAIL yields the closest interaction behaviors to the demonstrators. 2.1 MARKOV GAME AND -NASH EQUILIBRIUM Markov game (MG), or stochastic game , can be regarded as an extension of Markov Decision Process (MDP). Formally, we define an MG with N agents as a tuple N, S, A,..., A (N), P, r,..., r (N), ρ 0, γ, where S is the set of states, A (i) represents the action space of agent i, where i ∈ {1, 2, . . ., N}, P: S × A × A × · · · × A (N) × S → R is the state transition probability distribution, ρ 0: S → R is the distribution of the initial state s 0, and γ ∈ is the discounted factor. Each agent i holds its policy ] to make decisions and receive rewards defined as r We use −i to represent the set of agents except i, and variables without superscript i to denote the concatenation of all variables for all agents (e.g., π represents the joint policy and a denotes actions of all agents). For an arbitrary function f: s, a → R, there is a fact that The objective of agent i is to maximize its own total expected return R In Markov games, however, the reward function for each agent depends on the joint agent actions. Such a fact implies that one's optimal policy must also depend on others' policies. For the solution to the Markov games, -Nash equilibrium (-NE) is a commonly used concept that extends Nash equilibrium (NE) . where ) is the value function of agent i under state s, and Π (i) is the set of policies available to agent i. -NE is weaker than NE, which can be seen as sub-optimal NE. Every NE is equivalent to an -NE where = 0. Imitation learning aims to learn the policy directly from expert demonstrations without any access to the reward signals. In single-agent settings, such demonstrations come from behavior trajectories sampled with the expert policy, denoted as τ E = {(s t, a . However, in multi-agent settings, demonstrations are often interrelated trajectories, that is, which are sampled from the interactions of policies among all agents, denoted as Ω E = {(s t, a . For simplicity, we will use the term interactions directly as the concept of interrelated trajectories, and we refer to trajectories for a single agent. Typically, behavior cloning (BC) and inverse reinforcement learning (IRL) are two main approaches for imitation learning. Although IRL theoretically alleviates compounding error and outperforms to BC, it is less efficient since it requires resolving an RL problem inside the learning loop. Recent proposed work aims to learn the policy without estimating the reward function directly, notably, GAIL , which takes advantage of Generative Adversarial Networks (GAN ), showing that IRL is the dual problem of occupancy measure matching. GAIL regards the environment as a black-box, which is non-differentiable but can be leveraged through Monte-Carlo estimation of policy gradients. Formally, its objective can be expressed as min where D is a discriminator that identifies the expert trajectories with agents' sampled from policy π, which tries to maximize its evaluation from D; H is the causal entropy for the policy and λ is the hyperparameter. In multi-agent learning tasks, each agent i makes decisions independently while the ing reward ) depends on others' actions, which makes its cumulative return subjected to the joint policy π. One common joint policy modeling method is to decouple the π with assuming conditional independence of actions from different agents : However, such a non-correlated factorization on the joint policy is a vulnerable simplification which ignores the influence of opponents. And the learning process of agent i lacks stability since the environment dynamics depends on not only the current state but also the joint actions of all agents . To solve this, recent work has taken opponents into consideration by decoupling the joint policy as a correlated policy conditioned on state s and a where ) is the conditional policy, with which agent i regards all potential actions from its opponent policies π (−i) (a (−i) |s), and makes decisions through the marginal policy In multi-agent settings, for agent i with policy π (i), it seeks to maximize its cumulative reward against demonstrator opponents who equip with demonstrated policies π (−i) E via reinforcement learning: where H(π (i) ) is the γ-discounted entropy of policy π (i) and λ is the hyperparameter. By coupling with Eq., we define an IRL procedure to find a reward function r (i) such that the demonstrated joint policy outperforms all other policies, with the It is worth noting that we cannot obtain the demonstrated policies from the demonstrations directly. To solve this problem, we first introduce the occupancy measure, namely, the unnormalized distribution of s, a pairs correspond to the agent interactions navigated by joint policy π: With the definition in Eq., we can further formulate ρ π from agent i's perspective as where. Furthermore, with the support of Eq., we have In analogy to the definition of occupancy measure of that in a single-agent environment, we follow the derivation from and state the directly 1. Proposition 1. The IRL regarding demonstrator opponents is a dual form of following occupancy measure matching problem with regularizer ψ, and the induced optimal policy is the primal optimum: With setting the regularizer ψ = ψ GA similar to , we can obtain a GAIL-like imitation algorithm to learn π by introducing the adversarial training procedures of GANs which lead to a saddle point (where D (i) denotes the discriminator for agent i, which plays a role of surrogate cost function and guides the policy learning. However, such an algorithm is not practical, since we are unable to access the policies of demonstrator opponents π because the demonstrated policies are always given through sets of interactions data. To alleviate this deficiency, it is necessary to deal with accessible counterparts. Thereby we propose Proposition 2. Proposition 2. Let µ be an arbitrary function such that µ holds a similar form as π (−i), then Proof. Substituting π (−i) with µ in Eq. by importance sampling. Proposition 2 raises an important point that a term of importance weight can quantify the demonstrator opponents. By replacing π where is the importance sampling weight. In practice, it is challenging to estimate the densities and the learning methods might suffer from large variance. Thus, we fix α = 1 in our implementation, and as the experimental have shown, it has no significant influences on performance. Besides, a similar approach can be found in. So far, we've built a multi-agent imitation learning framework, which can be easily generalized to correlated or non-correlated policy settings. No prior has to be considered in advance since the discriminator is able to learn the implicit goal for each agent. With the objective shown in Eq., demonstrated interactions can be imitated by updating discriminators to offer surrogate rewards and learning their policies alternately. Formally, the update of discriminator for each agent i can be expressed as: and the update of policy is: where discriminator D (i) is parametrized by ω, and the policy π (i) is parametrized by θ. It is worth noting that the agent i considers opponents' action a (−i) while updating its policy and discriminator, with integrating all its possible decisions to find the optimal response. However, it is unrealistic to have the access to opponent joint policy π(a (−i) |s) for agent i. Thus, it is possible to estimate opponents' actions via approximating π (−i) (a (−i) |s) using opponent modeling. To that end, we construct a function, as the approximation of opponents for each agent i. Then we rewrite Eq. and Eq. as: and respectively. Therefore, each agent i must infer the opponents model σ (i) to approximate the unobservable policies π (−i), which can be achieved via supervised learning. Specifically, we learn in discrete action space by minimizing a cross-entropy (CE) loss, and a mean-square-error (MSE) loss in continuous action space: With opponents modeling, agents are able to be trained in a fully decentralized manner. We name our algorithm as Decentralized Adversarial Imitation Learning with Correlated policies (Correlated DAIL, a.k.a. CoDAIL) and present the training procedure in Appendix Algo. 1, which can be easily scaled to a distributed algorithm. As a comparison, we also present a non-correlated DAIL algorithm with non-correlated policy assumption in Appendix Algo. 2. In this section, we prove that the reinforcement learning objective against demonstrator counterparts shown in the last section is essentially equivalent to reaching an -NE. Since we fix the policies of agents −i as π E, the RL procedure mentioned in Eq. can be regarded as a single-agent RL problem. Similarly, with a fixed π (−i) E, the IRL process of Eq. is cast to a single-agent IRL problem, which recovers an optimal reward function r (i) * which achieves the best performance following the joint action π E. Thus we have We can also rewrite Eq. as Given the value function defined in Eq. for each agent i, for H(, then we finally obtain which is exactly the -NE defined in Definition 1. We can always prove that is bounded in small values such that the -NE solution concept is meaningful. Generally, random policies that keep vast entropy are not always considered as sub-optimal solutions or demonstrated policies π E in most reinforcement learning environments. As we do not require those random policies, we can remove them from the candidate policy set Π (i), which indicates that H(π (i) ) is bounded in small values, so as. Empirically, we adopt a small λ, and attain the demonstrator policy π E with an efficient learning algorithm to become a close-to-optimal solution. Thus, we conclude that the objective of our CoDAIL assumes that demonstrated policies institute an -NE solution concept (but not necessarily unique) that can be controlled the hyperparameter λ under some specific reward function, from which the agent learns a policy. It is worth noting that claimed that NE is incompatible with maximum entropy inverse reinforcement learning (MaxEnt IRL) because NE assumes that the agent never takes sub-optimal actions. Nevertheless, we prove that given demonstrator opponents, the multi-agent MaxEnt IRL defined in Eq. is equivalent to finding an -NE. Albeit non-correlated policy learning guided by a centralized critic has shown excellent properties in couple of methods, including MADDPG , COMA , MA Soft-Q , it lacks in modeling complex interactions because its decisions making relies on the independent policy assumption which only considers private observations while ignores the impact of opponent behaviors. To behave more rational, agents must take other agents into consideration, which leads to the studies of opponent modeling where an agent models how its opponents behave based on the interaction history when making decisions (; ;). For multi-agent imitation learning, however, prior works fail to learn from complicated demonstrations, and many of them are bounded with particular reward assumptions. For instance, proposed Parameter Sharing Generative Adversarial Imitation Learning (PS-GAIL) that adopts parameter sharing trick to extend GAIL to handle multi-agent problems directly, but it does not utilize the properties of Markov games with strong constraints on the action space and the reward function. Besides, there are many works built-in Markov games that are restricted under tabular representation and known dynamics but with specific prior of reward structures, as fully cooperative games (; ; Šošic et al., 2016;), two-player zero-sum games , two-player general-sum games , and linear combinations of specific features . Recently, some researchers take advantage of GAIL to solve Markov games. Inspired by a specific choice of Lagrange multipliers for a constraint optimization problem , derived a performance gap for multi-agent from NE. It proposed multi-agent GAIL (MA-GAIL), where they formulated the reward function for each agent using private actions and observations. As an improvement, presented a multi-agent adversarial inverse reinforcement learning (MA-AIRL) based on logistic stochastic best response equilibrium and MaxEnt IRL. However, both of them are inadequate to model agent interactions with correlated policies with independent discriminators. By contrast, our approach can generalize correlated policies to model the interactions from demonstrations and employ a fully decentralized training procedure without to get access to know the specific opponent policies. Except for the way of modeling multi-agent interactions as recovering agents' policies from demonstrations, which can regenerate similar interacted data, some other works consider different effects of interactions. proposed to learn a policy representation function of the agents based on their interactions and sets of generalization tasks using the learned policy embeddings. They regarded interactions as the episodes that contain only k (in the paper they used 2 agents), which constructs an agent-interaction graph. Different from us, they focused on the potential relationships among agents to help characterize agent behaviors. and proposed to use the Dynamic Bayesian Model that describes physical relationships among vehicles and driving behaviors to model interaction-dependent behaviors in autonomous driving scenario. Correlated policy structures that can help agents consider the influence of other agents usually need opponents modeling to infer others' actions. Opponent modeling has a rich history in MAL , and lots of researches have recently worked out various useful approaches for different settings in deep MARL, e.g., DRON and ROMMEO . In this paper, we focus on imitation learning with correlated policies, and we choose a natural and straightforward idea of opponent modeling that learning opponents' policies in the way of supervised learning with historical trajectories. Opponent models are used both in the training and the execution stages. Environment Description We test our method on the Particle World Environments , which is a popular benchmark for evaluating multi-agent algorithms, including several cooperative and competitive tasks. Specifically, we consider two cooperative scenarios and two com- petitive ones as follows: 1) Cooperative-communication, with 2 agents and 3 landmarks, where an unmovable speaker knowing the goal, cooperates with a listener to reach a particular landmarks who achieves the goal only through the message from the speaker; 2) Cooperative-navigation, with 3 agents and 3 landmarks, where agents must cooperate via physical actions and it requires each agent to reach one landmark while avoiding collisions; 3) Keep-away, with 1 agent, 1 adversary and 1 landmark, where the agent has to get close to the landmark, while the adversary is rewarded by pushing away the agent from the landmark without knowing the target; 4) Predator-prey, with 1 prey agent with 3 adversary predators, where the slower predactor agents must cooperate to chase the prey agent that moves faster and try to run away from the adversaries. Experimental Details We aim to compare the quality of interactions modeling in different aspects. To obtain the interacted demonstrations sampled from correlated policies, we train the demonstrator agent via a MARL learning algorithm with opponents modeling to regard others' policies into one's decision making, since the ground-truth reward in those simulated environments is accessible. Specifically, we modify the multi-agent version ), an efficient model-free policy gradient algorithm, by keeping an auxiliary opponents model and a conditioned policy for each agent, which can transform the original centralized on-policy learning algorithm to be decentralized. Note that we do not necessarily need experts that can do well in our designated environments. Instead, any demonstrator will be treated as it is from an -NE strategy concept under some unknown reward functions, which will be recovered by the discriminator. In our training procedure, we first obtain demonstrator policies induced by the ground-truth rewards and then generate demonstrations, i.e., the interactions data for imitation training. Then we train the agents through the surrogate rewards from discriminators. We compare CoDAIL with MA-AIRL, MA-GAIL, non-correlated DAIL (NC-DAIL) (the only difference between MA-GAIL and NC-DAIL is whether the reward function depends on joint actions or individual action) and a random agent. We do not apply any prior to the reward structure for all tasks to let the discriminator learn the implicit goals. All training procedures are pre-trained via behavior cloning to reduce the sample complexity, and we use 200 episodes of demonstrations, each with a maximum of 50 timesteps. Tab. 1 and Tab. 2 show the averaged absolute differences of reward for learned agents compared to the demonstrators in cooperative and competitive tasks, respectively. The learned interactions are considered superior if there are smaller reward gaps. Since cooperative tasks are reward-sharing, we show only a group reward for each task in Tab. 1. Compared to the baselines, CoDAIL achieves smaller gaps in both cooperative and competitive tasks, which suggests that our algorithm has a robust imitation learning capability of modeling the demonstrated interactions. It is also worth noting that CoDAIL achieves higher performance gaps in competitive tasks than cooperative ones, for which we think that conflict goals motivate more complicated interactions than a shared goal. Besides, MA-GAIL and NC-DAIL are about the same, indicating that less important is the surrogate reward structure on these multi-agent scenarios. To our surprise, MA-AIRL does not perform well in some environments, and even fails in Predator-prey. We list the raw obtained rewards in Appendix C, and we provide more hyperparameter sensitivity in Appendix D. Since we aim to recover the interactions of agents generated by the learned policies, it is proper to evaluate the relevance between distributions of regenerated interactions and demonstration data. Specifically, we collect positions of agents over hundreds of state-action tuples, which can be regarded as the low-dimension projection of the state-action interactions. We start each episode from a different initial state but the same for each algorithm in one episode. We run all the experiments under the same random seed, and collect positions of each agent in the total 100 episodes, each with a maximum of 50 timesteps. We first estimate the distribution of position (x, y) via Kernel Density Estimation (KDE) with Gaussian kernel to compute the Kullback-Leibler (KL) divergence between the generated interactions with the demonstrated ones, shown in Tab. 3. It is evident that in terms of the KL divergence between regenerated interactions with demonstrator interactions, CoDAIL generates the interaction data that obtains the minimum gap with the demonstration interaction, and highly outperforms other baseline methods. Besides, MA-GAIL and NC-DAIL reflect about-thesame performance to model complex interactions, while MA-AIRL behaves the worst, even worse than random agents on Predator-prey. To further understand the interactions generated by learned policies compared with the demonstrators, we visualize the interactions for demonstrator policies and all learned ones. We plot the density distribution of positions, (x, y) and marginal distributions of x-position and y-position. We illustrate the conducted on Keep-away in Fig. 1, other scenarios can be found in the Appendix E. Higher frequency positions in collected data are colored darker in the plane, and higher the value with respect to its marginal distributions. As shown in Fig. 1, the interaction densities of demonstrators and CoDAIL agents are highly similar (and with the smallest KL divergence), which tend to walk in the right-down side. In contrast, other learned agents fail to recover the demonstrator interactions. It is worth noting that even different policies can interact to earn similar rewards, but still keep vast differences among their generated interactions. Furthermore, such a reminds us that the real reward is not the best metric to evaluate the quality of modeling the demonstrated interactions or imitation learning . In this paper, we focus on modeling complex multi-agent interactions via imitation learning on demonstration data. We develop a decentralized adversarial imitation learning algorithm with correlated policies (CoDAIL) with approximated opponents modeling. CoDAIL allows for decentralized training and execution and is more capable of modeling correlated interactions from demonstrations shown by multi-dimensional comparisons against other state-of-the-art multi-agent imitation learning methods on several experiment scenarios. In the future, we will consider covering more imitation learning tasks and modeling the latent variables of policies for diverse multi-agent imitation learning. We list the raw obtained rewards of all algorithms in each scenarios. We evaluate how the stability of our algorithm when the hyperparameters change during our experiments on Communication-navigation. Tab. 6 shows the total reward difference between learned agents and demonstrators when we modify the training frequency of D and G (i.e., the policy), which indicates that the frequencies of D and G are more stable when D is trained slower than G, and the reaches a relative better performance when the frequency is 1:2 or 1:1. Fig. 2 illustrates that the choice of λ has little effect on the total performance. The reason may be derived from the discrete action space in this environment, where the policy entropy changes gently. The density and marginal distribution of agents' positions, (x, y), in 100 repeated episodes with different initialized states, generated from different learned policies upon Cooperative-navigation. Experiments are done under the same random seed. The top of each sub-figure is drawn from state-action pairs of all agents while the below explain for each one. KL is the KL divergence between generated interactions (top figure) with the demonstrators. The density and marginal distributions of agents' positions, (x, y), in 100 repeated episodes with different initialized states, generated from different learned policies upon Predator-prey. Experiments are conducted under the same random seed. The top of each sub-figure is drawn from state-action pairs of all agents while the below explains for each one. The KL term means the KL divergence between generated interactions (top figure) with the demonstrators.

Modeling complex multi-agent interactions under multi-agent imitation learning framework with explicit modeling of correlated policies by approximating opponents’ policies.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.


Question: Represent the article for finding a single-sentence TLDR

Question: Plagiarism and text reuse become more available with the Internet development. Therefore it is important to check scientific papers for the fact of cheating, especially in Academia. Existing systems of plagiarism detection show the good performance and have a huge source databases. Thus now it is not enough just to copy the text as is from the source document to get the original work. Therefore, another type of plagiarism become popular -- cross-lingual plagiarism. We present a CrossLang system for such kind of plagiarism detection for English-Russian language pair. The key idea for CrossLang 1 system is that we use the monolingual approach. We have suspicious Russian document and English reference collection. We reduce the task to the one language -we translate the suspicious document into English, because the reference collection is in English. After this step we perform the subsequent document analysis. Due to this fact the main challenge with the CrossLang design is that the algorithms should be stable to the translation ambiguity. The main stages of CrossLang service is depicted in Figure 1. CrossLang receives the suspicious document from Antiplagiat system, when user send it for originality checking. Then it goes to Entry pointmain service, that routes the data between following stages: 1. Machine Translation system -microservice, that translates suspicious document into English. For these purposes we use Transformer Vaswani et al., open-source neural machine translation framework. 2. Source retrieval -this stage unites two microservices: Shingle index and Document storage. Entry point receives the translated suspicious document's shingles (n -grams) and Shingle index returns to it the documents ids from the reference English collection. To deal with the translation ambiguity we use modified shingle-based approach. Document storage returns the Source texts from the collection by these ids. 3. Document comparison -this microservice performs the comparison between translated suspicious document and source documents. We compare not the texts themselves, but the vectors corresponding to the phrases of these texts. Thus we deal with the translation ambiguity problem. We create machine translation system using state-of-the-art The CrossLang BLEU score lower than Google's BLEU score -this was to be expected. But it is very important to notice that we are not interested in ideal translation. Our main goal is to translate with sufficient quality for the next stages: Source retrieval and Document comparison. The method of source retrieval in the case of verbatim plagiarism is inverted index construction,where a document from the reference collection is represented as a set of its shingles, i.e. overlapping word n -grams, and a suspicious document's shingles are checked for matches with the indexed documents. There is one major problem with using the standard shingles -in our case the machine translation stage generates texts that differ too much from the sources of plagiarism. We argue that the source retrieval task can be solved with the help of a similar method that performs better than the method mentioned above; this improvement is achieved by moving from word shingles to word-class shingles, where each word is substituted by the label of the class it belongs to: {word 1, . . ., word n} → {class(word 1),..., class(word n)}. Clustering the word vectors is a convenient and relatively fast way of obtaining semantic word classes. For the word embedding model we used fastText Bojanowski et al. trained on English Wikipedia. The dimension for word embedding model was set to 100. For the semantic word classes construction we applied agglomerative clustering on word embeddings with the cosine similarity measure to group words into word classes. We got 777K words clustered into 30K classes. For the comparison between retrieved documents and translated suspicious documents we introduce the phrase embedding model. We split documents (retrieved and suspicious) into phrases s and compare its vectors. For mapping the word sequence into low dimensional space we use the encoderdecoder scheme with L-2 reconstruction error minimization E rec = s −ŝ 2. Encoder-decoder model is completely unsupervised and does not use any information whether the phrase pair is paraphrased or not. We train Seq2Seq model with attention Bahdanau et al. on 10M sentences from Wikipedia. In order to use information about phrase similarity we extend the objective function. We employ the margin-base loss from Wieting et al. with the limited number of similar phrase pairs S = {(s i, s j)}: where The sampling of so named "false neighbour" s i during training helps to improve the final quality without strict limitations on what phrases we should use at dissimilar. This part of objective requires a dataset of similar sentences S = {(s i, s j)}. We used double translation method as a method of similar sentences generation comparable to paraphrase. The final objective function is: where α is a tunable hyperparameter that weights both of errors. 6. For each phrase embedding from the suspicious document find nearest vectors by cosine similarity from source documents using Annoy 7 library. 1. The best of our knowledge it is the first system for cross-lingual plagiarism detection for English-Russian language pair. It is deployed on production and we could analyze the . We could not find another examples of such system (even for other language pairs). 2. The Source retrieval 1.2 stage is often employed using rather simple heuristical algorithms such as shingle-based search or keyword extraction. However, these methods can significantly suffer from word replacements and usually detect only near-duplicate paraphrase. We present modified method, see 1.2. 3. Many articles on the cross-lingual plagiarism detection topic investigate the solutions based on bilingual or monolingual word embeddings Ferrero et al. for documents comparison, but almost none of them uses the phrase embeddings for this problem solution. We present phrase embeddings comparison in 1.3. There are no and datasets for cross-lingual plagiarism detection task for language pair EnglishRussian. We create dataset for the problem and make it available. Visit 8 for dataset download and details about generation. For the whole framework we got Precision = 0.83, Recall = 0.79 and F 1 = 0.80. Since our system translates the suspicious document into the language of the collection it's natural to analyze the performance of our system for monolingual problem. For such experiment we do not use the machine translation service. In order to check performance of monolingual paraphrased plagiarism detection we exploit PAN'11 contest dataset and quality metrics Potthast et al.. Results of CrossLang and top-3 known previous methods are in Table 2. Our service is deployable on an 8-GPU cluster with Tesla-K100 GPUs, 128GB RAM and 64 CPU Cores. Depending on the requirements, the service is able to scale horizontally. For the fast rescaling we use Docker containerization and Consul and Consul-template for the service discovery and automatic load balancing. The stress testing of our system showed that the system is able to check up to 100 documents in a minute. Despite the fact the average loading on our service is much lower, this characteristic of our service is important for withstanding peak loads. We introduced CrossLang -a framework for cross-lingual plagiarism detection for English Russian language pair. We decomposed the problem of cross-lingual plagiarism detection into several stages and provide a service, consists of a set of microservices. The CrossLang use a monolingual approachreducing the problem to the one language. For this purpose we trained the neural machine translation system. Another two main algoithmic components are Source Retrieval and Document Comparison stages. For the Source Retrieval problem we used a modification of shingling method that allow us to deal with ambiguity after translation. For the Document Comparison stage we used phrase embeddings that were trained with slight supervision. We evaluated the effectiveness of main stages.

Represent the short summary for finding the corresponding scientific article (long)

A system for cross-lingual (English-Russian) plagiarism detection

represent this text to find a much much longer article about it with say hundreds-thousands of words

For example, A novel cluster-based algorithm of curriculum learning is proposed to solve the robust training of generative models. should be similar to Training generative models like Generative Adversarial Network (GAN) is challenging for noisy data. A novel curriculum learning algorithm pertaining to clustering is proposed to address this issue in this paper. The curriculum construction is based on the centrality of underlying clusters in data points. The data points of high centrality takes priority of being fed into generative models during training. To make our algorithm scalable to large-scale data, the active set is devised, in the sense that every round of training proceeds only on an active subset containing a small fraction of already trained data and the incremental data of lower centrality. Moreover, the geometric analysis is presented to interpret the necessity of cluster curriculum for generative models. The experiments on cat and human-face data validate that our algorithm is able to learn the optimal generative models (e.g. ProGAN) with respect to specified quality metrics for noisy data. An interesting finding is that the optimal cluster curriculum is closely related to the critical point of the geometric percolation process formulated in the paper. Deep generative models have piqued researchers' interest in the past decade. The fruitful progress has been achieved on this topic, such as auto-encoder and variational auto-encoder (VAE) , generative adversarial network (GAN) (;, normalizing flow (; ;), and autoregressive models (van den b; a; . However, it is non-trivial to train a deep generative model that can converge to a proper minimum of the associated optimization. For example, GAN suffers non-stability, mode collapse, and generative distortion during training. Many insightful algorithms have been proposed to circumvent those issues, including feature engineering , various discrimination metrics , distinctive gradient penalties , spectral normalization to discriminator , and orthogonal regularization to generator . What is particularly of interest is that the breakthrough for GANs has been made with a simple technique of progressively growing neural networks of generators and discriminators from low-resolution images to high-resolution counterparts (a). This kind of progressive growing also helps push the state of the arts to a new level by enabling StyleGAN to produce photo-realistic and detail-sharp (b), shedding new light on wide applications of GANs in solving real problems. This idea of progressive learning is actually a general manner of cognition process , which has been formally named curriculum learning in machine learning . The central topic of this paper is to explore a new curriculum for training deep generative models. To facilitate robust training of deep generative models with noisy data, we propose curriculum learning with clustering. The key contributions are listed as follows: • We first summarize four representative curricula for generative models, i.e. architecture (generation capacity), semantics (data content), dimension (data space), and cluster (data structure). Among these curricula, cluster curriculum is newly proposed in this paper. • Cluster curriculum is to treat data according to the centrality of each data point, which is pictorially illustrated and explained in detail. To foster large-scale learning, we devise the active set algorithm that only needs an active data subset of small fixed size for training. • The geometric principle is formulated to analyze hardness of noisy data and advantage of cluster curriculum. The geometry pertains to counting a small sphere packed in an ellipsoid, on which is based the percolation theory we use. The research on curriculum learning is diverse. Our work focuses on curricula that are closely related to data attributes, beyond which is not the scope we concern in this paper. Curriculum learning has been a basic learning approach to promoting performance of algorithms in machine learning. We quote the original words from the seminal paper as its definition: Curriculum learning. " The basic idea is to start small, learn easier aspects of the task or easier sub-tasks, and then gradually increase the difficulty level" according to pre-defined or self-learned curricula. From cognitive perspective, curriculum learning is common for human and animal learning when they interact with environments , which is the reason why it is natural as a learning rule for machine intelligence. The learning process of cognitive development is gradual and progressive . In practice, the design of curricula is task-dependent and data-dependent. Here we summarize the representative curricula that are developed for generative models. Architecture curriculum. The deep neural architecture itself can be viewed as a curriculum from the viewpoint of learning concepts or disentangling representations . For example, the different layers decompose distinctive features of objects for recognition (; ;) and generation . Besides, Progressive growing of neural architectures is successfully exploited in GANs (a; ; ; b). Semantics curriculum. The most intuitive content for each datum is the semantic information that the datum conveys. The hardness of semantics determines the difficulty of learning knowledge from data. Therefore, the semantics can be a common curriculum. For instance, the environment for a game in deep reinforcement learning and the number sense of learning cognitive concepts with neural networks can be such curricula. Dimension curriculum. The high dimension usually poses the difficulty of machine learning due to the curse of dimensionality , in the sense that the amount of data points for learning grows exponentially with dimension of variables . Therefore, the algorithms are expected to be beneficial from growing dimensions. The effectiveness of dimension curriculum is evident from recent progress on deep generative models, such as ProGANs (a; b) by gradually enlarging image resolution and language generation from short sequences to long sequences of more complexity . For fitting distributions, dense data points are generally easier to handle than sparse data or outliers. To train generative models robustly, therefore, it is plausible to raise cluster curriculum, meaning that generative algorithms first learn from data points close to cluster centers and then with more data progressively approaching cluster boundaries. Thus the stream of feeding data points to models for curriculum learning is the process of clustering data points according to cluster centrality that will be explained in section 3.2. The toy example in Figure 1 illustrates how to form cluster curriculum. The importance of clusters for data points is actually obvious from geometric point of view. The data sparsity in high-dimensional spaces causes the difficulty of fitting the underlying distribution of n = 100 n = 200 n = 300 n = 400 Figure 1: Cluster Curriculum. From magenta color to black color, the centrality of data points reduces. The value n is the number of data points taken with centrality order. data points . So generative algorithms may be beneficial when proceeding from the local spaces where data points are relatively dense. Such data points form clusters that are generally informative subsets with respect to the entire dataset. In addition, clusters contain common regular patterns of data points, where generative models are easier to converge. What is most important is that noisy data points deteriorate performance of algorithms. For classification, the effectiveness of curriculum learning is theoretically proven to circumvent the negative influence of noisy data . We will analyze this aspect for generative models with geometric facts. With cluster curriculum, we are allowed to gradually learn generative models from dense clusters to cluster boundaries and finally to all data points. In this way, generative algorithms are capable of avoiding the direct harm of noise or outliers. To this end, we first need a measure called centrality that is the terminology in graph-based clustering. It quantifies the compactness of a cluster in data points or a community in complex networks . A large centrality implies that the associated data point is close to one of cluster centers. For easy reference, we provide the algorithm of the centrality we use in Appendix. For experiments in this paper, all the cluster curricula are constructed by the centrality of stationary probability distribution, i.e. the eigenvector corresponding to the largest eigenvalue of the transition probability matrix drawn from the data. To be specific, let c ∈ R m denote the centrality vector of m data points. Namely, the i-th entry c i of c is the centrality of data point x i. Sorting c in descending order and adjusting the order of original data points accordingly give data points arranged by cluster centrality. Let.., − → X l } signify the set of centrality-sorted data points, where − → X 0 is the base set that contains sufficient data to attain convergent generative models, and the rest of − → X is evenly divided into l subsets according to centrality order. In general, the number of data points in − → X 0 is much less than m and determined according to X. Such division of − → X 0 serves to efficiency of training, because we do not need to train models from a very small dataset. The cluster curriculum learning is carried out by incrementally feeding subsets in − → X into generative algorithms. In other words, algorithms are successively trained on, meaning that the curriculum for each round of training is accumulated with − → X i. In order to determine the optimal curriculum we need the aid of quality metric of generative models, such as Fréchet inception distance (FID) or sliced Wasserstein distance (SWD) . For generative models trained with each curriculum, we calculate the associated score s i via the specified quality metric. The optimal curriculum for effective training can be identified by the minimal value for all s i, where i = 1,..., l + 1. The interesting phenomenon of this score curve will be illustrated in the experiment. The minimum of score s is apparently metric-dependent. One can refer to for the review of evaluation metrics. In practice, we can opt one of reliable metrics to use or multiple metrics for decision-making of the optimal model. There are two ways of using the incremental subset − → X i during training. One is that the parameters of models are re-randomized when the new data are used, the procedure of which is given in Algorithm 1 in Appendix. The other is that the parameters are fine-tuned based on pre-training of the previous model, which will be presented with a fast learning algorithm in the following section. To obtain the precise minimum of s, the cardinality of − → X i needs to be set much smaller than m, meaning that l will be large even for a dataset of moderate scale. The training of many loops will be time-consuming. Here we propose the active set to address the issue, in the sense that for each loop of cluster curriculum, the generative models are always trained with a subset of a small fixed size instead of − → X 0 ← − → X 0 ∪ − → X i whose size becomes incrementally large. (a) Figure 2: Schematic illustration of active set for cluster curriculum. The cardinality |A| of the active set A is 200. When − → X 2 is taken for training, we need to randomly sample another 100 (i.e. − → X 2|) data points from the history data Then the complete active set is composed by We can see that data points in become less dense after sampling. To form the active set A, the subset − → A 0 of data points are randomly sampled from − → X 0 to combine with − → X i for the next loop, where For easy understanding, we illustrate the active set with toy example in Figure 2. In this scenario, progressive pre-training must be applied, meaning that the update of model parameters for the current training is based on parameters of previous loop. The procedure of cluster curriculum with active set is detailed in Algorithm 2 in Appendix. The active set allows us to train generative models with a small dataset that is actively adapted, thereby significantly reducing the training time for large-scale data. Cluster curriculum bears the interesting relation to high-dimensional geometry, which can provide geometric understanding of our algorithm. Without loss of generality, we work on a cluster obeying the normal distribution. The characteristic of the cluster can be extended into other clusters of the same distribution. For easy analysis, let us begin with a toy example. As Figure 3(a) shows, the confidence ellipse E 2 fitted from the subset of centrality-ranked data points is nearly conformal to E 1 of all data points, which allows us to put the relation of these two ellipses by virtue of the confidence-level equation. Let N (0, Σ) signify the center and covariance matrix of the cluster C of interest, where C = {x i |x i ∈ R d, i = 1, . . ., n}. To make it formal, we can write the equation by The annulus formed by of removing the inner ellipse from the outer one. where χ To analyze the hardness of training generative models, a fundamental aspect is to examine the number n(E) of given data points falling in a geometric entity E 1 and the number N (E) of lattice points in it. The less n(E) is compared to N (E), the harder the problem will be. However, the enumeration of lattice points is computationally prohibitive for high dimensions. Inspired by the information theory of encoding data of normal distributions , we count the number of small spheres S ε of radius ε packed in the ellipsoid E instead. Thus we can use this number to replace the role of N (E) as long as the radius of the sphere S ε is set properly. With a little abuse of normal d=2 d=20 d=50 d=100 d=500 d=1000 lattice d=2 d=20 d=50 d=100 d=500 d=1000 Figure 4: Comparison between the number n(A) of data points sampled from the isotropic normal distributions and N (A) of spheres (lattice) packed in the annulus A with respect to the Chi quantile χ α2. d is the dimension of data points. For each dimension, we sample 70,000 data points from N (0, I). The scales of y-axis and x-axis are normalized by 10,000 and χ α1, respectively. notation, we still use N (E) to denote the packing number in the following context. Theorem 1 gives the exact form of N (E). Theorem 1. For a set C = {x i |x i ∈ R d} of n data points drawn from normal distribution N (0, Σ), the ellipsoid E α of confidence 1 − α is defined as x Σ −1 x ≤ χ 2 α, where Σ has no zero eigenvalues and α ∈. Let N (E α) be the number of spheres of radius ε packed in the ellipsoid E α. Then we can establish We can see that N (E α) admits a tidy form with Mahalanobis distance χ α, dimension d, and sphere radius ε as variables. The proof is provided in Appendix. The geometric region of interest for cluster curriculum is the annulus A formed by removing the ellipsoid 2 E α2 from the ellipsoid E α1, as Figure 3 (b) displays. We investigate the varying law between n(A) and N (A) in the annulus A when the inner ellipse E α2 grows with cluster curriculum. For this purpose, we need the following two corollaries that immediately follows from Theorem 1. Corollary 1. Let N (A) be the number of spheres of radius ε packed in the annulus A that is formed by removing the ellipsoid E α1 from the ellipsoid E α1, where α 1 ≤ α 2. Then we have It is obvious that N (A) goes infinite when d → ∞ under the conditions that χ α1 > χ α2 and ε is bounded. Besides, when E α2 (cluster) grows, N (A) reduces with exponent d if E α1 is fixed. In light of Corollary 1, we can now demonstrate the functional law between n(A) and N (A). First, we determine χ α1 as follows which means that E α1 is the ellipsoid of minimal Mahalanobis distance to the center that contains all the data points in the cluster. In addition, we need to estimate a suitable sphere radius ε, such that n(E α1) and N (E α1) have comparable scales in order to make n(A) and N (A) comparable in scale. To achieve this, we define an oracle ellipse E where n(E) = N (E). For simplicity, we let E α1 be the oracle ellipse. Thus we can determine ε with Corollary 3. Corollary 3. If we let E α1 be the oracle ellipse such that n(E α1) = N (E α1), then the free parameter ε can be computed with ε = χ α1 det(Σ)/n(E α1) To make the demonstration amenable to handle, data points we use for simulation are assumed to obey the isotropic normal distribution, meaning that data points are generated with nearly equal Figure 5: Examples of LSUN cat dataset and CelebA face dataset. The samples in the first row are of high centrality and the samples of low centrality in the second row are noisy data or outliers that we call in the context. variance along each dimension. Figure 4 shows that n(A) gradually exhibits the critical phenomena of percolation processes 3 when the dimension d goes large, implying that the data points in the annulus A are significantly reduced when E α2 grows a little bigger near the critical point. In contrast, the number N (A) of lattice points is still large and varies negligibly until E α2 approaches the boundary. This discrepancy indicates clearly that fitting data points in the annulus is pretty hard and guaranteeing the precision is nearly impossible when crossing the critical point of n(A) even for a moderate dimension (e.g. d = 500). Therefore, the plausibility of cluster curriculum can be drawn naturally from this geometric fact. The generative model that we use for experiments are Progressive growing of GAN (ProGAN) (a). This algorithm is chosen because ProGAN is the state-of-the-arts algorithm of GANs with official open sources available. According to convention, we opt the Fréchet inception distance (FID) for ProGAN as the quality metric. We randomly sample the 200,000 cat images from the LSUN dataset . These cat images are captured in the wild. So their styles vary significantly. Figure 5 shows the cat examples of high and low centralities. We can see that the noisy cat images differ much from the clean ones. There actually contain the images of very few informative cat features, which are the outliers we refer to. The curriculum parameters are set as | − → X 0 | = 20, 000 and | − → X i | = 10, 000, which means that the algorithms are trained with 20,000 images first and after the initial training, another 10,000 images according to centrality order are merged into the current training data for further re-training. For active set, its size is fixed to be 30, 000. The CelebA dataset is a large-scale face attribute dataset . We use the cropped and well-aligned faces with a bit of image s preserved for generation task. For clustercurriculum learning, we randomly sample 70,000 faces as the training set. The face examples of different centralities are shown in Figure 5. The curriculum parameters are set as | − → X 0 | = 10, 000 and | − → X i | = 5, 000. We bypass the experiment of the active set on faces because it is used for the large-scale data. Each image in two databases is resized to be 64 × 64. To form cluster curricula, we exploit ResNet34 pre-trained on ImageNet to extract 512-dimensional features for each face and cat images. The directed graphs are built with these feature vectors. We determine the parameter σ of edge weights by enforcing the geometric mean of weights to be 0.8. The robustness of varying the value was validated in for clustering. The number of nearest neighbors is set to be K = 4 log m. The centrality is the stationary probability distribution. All codes are written with TensorFlow.: FID curves of cluster-curriculum learning for ProGAN on the cat dataset and CelebA face dataset. The centrality and the FID share the x-axis due to that they have the same order of data points. The same colors of the y-axis labels and the curves denote the figurative correspondence. The network parameters for "normal training" are randomly re-initialized for each re-training. The active set is based on progressive pre-training of the fixed small dataset. The scale of the x-axis is normalized by 10,000. From Figure 6a, we can see that the FID curves are all nearly V-shaped, indicating that the global minima exist amid the training process. This is the clear evidence that the noisy data and outliers deteriorate the quality of generative models during training. From the optimal curricula found by two algorithms (i.e. curricula at 110,000 and 100,000), we can see that the curriculum of the active set differs from that of normal training only by one-step data increment, implying that the active set is reliable for fast cluster-curriculum learning. The performance of the active set measured by FID is much worse than that of normal training, especially when more noisy data are fed into generative models. However, this does not change the whole V-shape of the accuracy curve. Namely, it is applicable as long as the active set admits the metric minimum corresponding to the appropriate curriculum. The V-shape of the centrality-FID curve on the cat data is due to that the noisy data of low centrality contains little effective information to characterize the cats, as already displayed in Figure 5. However, it is different for the CelebA face dataset where the face images of low centrality also convey the part of face features. As evident by Figure 6b, ProGAN keeps being optimized by the majority of the data until the curriculum of size 55, 000. To highlight the meaning of this nearly negligible minimum, we also conduct the exactly same experiment on the FFHQ face dataset containing 70, 000 face images of high-quality (b). For FFHQ data, the noisy face data can be ignored. The gray curve of normal training in Figure 6b indicates that the FID of ProGAN is monotonically decreased for all curricula. This gentle difference of the FID curves at the ends between CelebA and FFHQ clearly demonstrates the difficulty of noisy data to generative algorithms. To understand cluster curriculum deeply, we employ the geometric method formulated in section 4 to analyze the cat and face data. The percolation processes are both conducted with 512-dimensional features from ResNet34. Figure 7 displays the curve of n(A) that is the variable of interest in this scenario. As expected, the critical point in the percolation process occurs for both cases, as shown by blue curves. An obvious fact is that the optimal curricula (red strips) both fall into the (feasible) domains of percolation processes after the critical points, as indicated by gray color. This is a desirable property because data become rather sparse in the annuli when crossing the critical points. Then noisy data play the non-negligible role on tuning the parameters of generative models. Therefore, a fast learning strategy can be derived from the percolation process. Training may begin from the curriculum specified by the critical point, thus significantly accelerating cluster-curriculum learning. The pink strips are intervals of optimal curricula derived by generative models. For example, the value 9 of the pink interval in (a) is obtained by 9 = 20 − 11, where 11 is one of the minima (i.e. 110,000) in Figure 6a. The others are derived in the same way. The subtraction transforms the data number in the cluster to be the one in the annulus. The critical points are determined by searching the maxima of the absolute discrete difference of the associated curves. The scales of y-axes are normalized by 10,000. Another intriguing phenomenon is that the more noisy the data, the closer the optimal interval (red strip) is to the critical point. We can see that the optimal interval of the cat data is much closer to the critical point than that of the face data. What surprises us here is that the optimal interval of cluster curricula associated with the cat data nearly coincides with the critical point of the percolation process in the annulus! This means that the optimal curriculum may be found at the intervals close to the critical point of of n(A) percolation for heavily noisy data, thus affording great convenience to learning an appropriate generative model for such datasets. Cluster curriculum is proposed for robust training of generative models. The active set of cluster curriculum is devised to facilitate scalable learning. The geometric principle behind cluster curriculum is analyzed in detail as well. The experimental on the LSUN cat dataset and CelebA face dataset demonstrate that the generative models trained with cluster curriculum is capable of learning the optimal parameters with respect to the specified quality metric such as Fréchet inception distance and sliced Wasserstein distance. Geometric analysis indicates that the optimal curricula obtained from generative models are closely related to the critical points of the associated percolation processes established in this paper. This intriguing geometric phenomenon is worth being explored deeply in terms of the theoretical connection between generative models and high-dimensional geometry. It is worth emphasizing that the meaning of model optimality refers to the global minimum of the centrality-FID curve. As we already noted, the optimality is metric-dependent. We are able to obtain the optimal model with cluster curriculum, which does not mean that the algorithm only serves to this purpose. We know that more informative data can help learn a more powerful model covering the large data diversity. Here a trade-off arises, i.e. the robustness against noise and the capacity of fitting more data. The centrality-FID curve provides a visual tool to monitor the state of model training, thus aiding us in understanding the learning process and selecting suitable models according to noisy degree of given data. For instance, we can pick the trained model close to the optimal curriculum for heavily noisy data or the one near the end of the centrality-FID curve for datasets of little noise. In fact, this may be the most common way of using cluster curriculum. In this paper, we do not investigate the cluster-curriculum learning for the multi-class case, e.g. the ImageNet dataset with BigGAN . The cluster-curriculum learning of multiple classes is more complex than that we have already analyzed on the face and cat data. We leave this study for future work. The centrality or clustering coefficient pertaining to a cluster in data points or a community in a complex network is a well-studied traditional topic in machine learning and complex systems. Here we introduce the graph-theoretic centrality for the utilization of cluster curriculum. Firstly, we construct a directed graph (digraph) with K nearest neighbors by the method in . The weighted adjacency matrix W of the digraph can be formed in this way: 2 ) if x j is one of the nearest neighbors of x i and 0 otherwise, where d ij is the distance between x i and x j and σ is a free parameter. The density of data points can be quantified with the stationary probability distribution of a Markov chain. For a digraph built from data, the transition probability matrix can be derived by row normalization, say, P ij = W ij / j W ij. Then the stationary probability u can be obtained by solving an eigenvalue problem P u = u, where denotes the matrix transpose. It is straightforward to know that u is the eigen-vector of P corresponding to the largest eigenvalue (i.e. 1). u is also defined as a kind of PageRank in many scenarios. For density-based cluster curriculum, the centrality c coincides with the stationary probability u. Figure 1 in the main context shows the plausibility of using the stationary probability distribution to quantify the data density. Then we derive the length of semi-axis with respect to χ α, i.e. For a d-dimensional ellipsoid E, the volume of E is where r i the leng of semi-axis of E and Γ(·) is the Gamma function. Substituting into the above equation, we obtain the final formula of volume Using the volume formula in, it is straightforward to get the volume of the packing sphere S ε By the definition of N (E α), we can write We conclude the proof of the theorem.

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.