Question: Represent the scientific article to find a short TLDR

Question: Long Short-Term Memory (LSTM) is one of the most powerful sequence models. Despite the strong performance, however, it lacks the nice interpretability as in state space models. In this paper, we present a way to combine the best of both worlds by introducing State Space LSTM (SSL), which generalizes the earlier work \cite{zaheer2017latent} of combining topic models with LSTM. However, unlike \cite{zaheer2017latent}, we do not make any factorization assumptions in our inference algorithm. We present an efficient sampler based on sequential Monte Carlo (SMC) method that draws from the joint posterior directly. Experimental confirms the superiority and stability of this SMC inference algorithm on a variety of domains. State space models (SSMs), such as hidden Markov models (HMM) and linear dynamical systems (LDS), have been the workhorse of sequence modeling in the past decades From a graphical model perspective, efficient message passing algorithms BID34 BID17 are available in compact closed form thanks to their simple linear Markov structure. However, simplicity comes at a cost: real world sequences can have long-range dependencies that cannot be captured by Markov models; and the linearity of transition and emission restricts the flexibility of the model for complex sequences. A popular alternative is the recurrent neural networks (RNN), for instance the Long Short-Term Memory (LSTM) BID14 ) which has become a standard for sequence modeling nowadays. Instead of associating the observations with stochastic latent variables, RNN directly defines the distribution of each observation conditioned on the past, parameterized by a neural network. The recurrent parameterization not only allows RNN to provide a rich function class, but also permits scalable stochastic optimization such as the backpropagation through time (BPTT) algorithm. However, flexibility does not come for free as well: due to the complex form of the transition function, the hidden states of RNN are often hard to interpret. Moreover, it can require large amount of parameters for seemingly simple sequence models BID35.In this paper, we propose a new class of models State Space LSTM (SSL) that combines the best of both worlds. We show that SSLs can handle nonlinear, non-Markovian dynamics like RNNs, while retaining the probabilistic interpretations of SSMs. The intuition, in short, is to separate the state space from the sample space. In particular, instead of directly estimating the dynamics from the observed sequence, we focus on modeling the sequence of latent states, which may represent the true underlying dynamics that generated the noisy observations. Unlike SSMs, where the same goal is pursued under linearity and Markov assumption, we alleviate the restriction by directly modeling the transition function between states parameterized by a neural network. On the other hand, we bridge the state space and the sample space using classical probabilistic relation, which not only brings additional interpretability, but also enables the LSTM to work with more structured representation rather than the noisy observations. Indeed, parameter estimation of such models can be nontrivial. Since the LSTM is defined over a sequence of latent variables rather than observations, it is not straightforward to apply the usual BPTT algorithm without making variational approximations. In BID35, which is an instance of SSL, an EM-type approach was employed: the algorithm alternates between imputing the latent states and optimizing the LSTM over the imputed sequences. However, as we show below, the inference implicitly assumes the posterior is factorizable through time. This is a restrictive assumption since the benefit of rich state transition brought by the LSTM may be neutralized by breaking down the posterior over time. We present a general parameter estimation scheme for the proposed class of models based on sequential Monte Carlo (SMC) BID8, in particular the Particle Gibbs BID1. Instead of sampling each time point individually, we directly sample from the joint posterior without making limiting factorization assumptions. Through extensive experiments we verify that sampling from the full posterior leads to significant improvement in the performance. Related works Enhancing state space models using neural networks is not a new idea. Traditional approaches can be traced back to nonlinear extensions of linear dynamical systems, such as extended or unscented Kalman filters , where both state transition and emission are generalized to nonlinear functions. The idea of parameterizing them with neural networks can be found in BID12, as well as many recent works BID22 BID2 BID15 BID23 BID18 thanks to the development of recognition networks BID20 BID32. Enriching the output distribution of RNN has also regain popularity recently. Unlike conventionally used multinomial output or mixture density networks BID4, recent approaches seek for more flexible family of distributions such as restricted Boltzmann machines (RBM) BID6 or variational auto-encoders (VAE) BID13 BID7.On the flip side, there have been studies in introducing stochasticity to recurrent neural networks. For instance, BID30 and BID3 incorporated independent latent variables at each time step; while in BID9 the RNN is attached to both latent states and observations. We note that in our approach the transition and emission are decoupled, not only for interpretability but also for efficient inference without variational assumptions. On a related note, sequential Monte Carlo methods have recently received attention in approximating the variational objective BID27 BID24 BID29. Despite the similarity, we emphasize that the context is different: we take a stochastic EM approach, where the full expectation in E-step is replaced by the samples from SMC. In contrast, SMC in above works is aimed at providing a tighter lower bound for the variational objective. In this section, we provide a brief review of some key ingredients of this paper. We first describe the SSMs and the RNNs for sequence modeling, and then outline the SMC methods for sampling from a series of distributions. Consider a sequence of observations x 1:T = (x 1, . . ., x T) and a corresponding sequence of latent states z 1:T = (z 1, . . ., z T). The SSMs are a class of graphical models that defines probabilistic dependencies between latent states and the observations. A classical example of SSM is the (Gaussian) LDS, where real-valued states evolve linearly over time under the first-order Markov assumption. Let x t ∈ R d and z t ∈ R k, the LDS can be expressed by two equations: DISPLAYFORM0 where A ∈ R k×k, C ∈ R d×k, and Q and R are covariance matrices of corresponding sizes. They are widely applied in modeling the dynamics of moving objects, with z t representing the true state of the system, such as location and velocity of the object, and x t being the noisy observation under zero-mean Gaussian noise. We mention two important inference tasks BID21 ) associated with SSMs. The first tasks is filtering: at any time t, compute p(z t |x 1:t), i.e. the most up-to-date belief of the state z t conditioned on all past and current observations x 1:t. The other task is smoothing, which computes p(z t |x 1:T), i.e. the update to the belief of a latent state by incorporating future observations. One of the beauties of SSMs is that these inference tasks are available in closed form, thanks for p = 1,..., P.to the simple Markovian dynamics of the latent states. For instance, the forward-backward algorithm BID34, the Kalman filter BID17, and RTS smoother BID31 are widely appreciated in the literature of HMM and LDS.Having obtained the closed form filtering and smoothing equations, one can make use of the EM algorithm to find the maximum likelihood estimate (MLE) of the parameters given observations. In the case of LDS, the E-step can be computed by RTS smoother and the M-step is simple subproblems such as least-squares regression. We refer to BID11 for a full exposition on learning the parameters of LDS using EM iterations. RNNs have received remarkable attention in recent years due to their strong benchmark performance as well as successful applications in real-world problems. Unlike SSMs, RNNs aim to directly learn the complex generative distribution of p(x t |x 1:t−1) using a neural network, with the help of a deterministic internal state s t: DISPLAYFORM0 where RNN(·, ·) is the transition function defined by a neural network, and g(·) is an arbitrary differentiable function that maps the RNN state s t to the parameter of the distribution of x t. The flexibility of the transformation function allows the RNN to learn from complex nonlinear nonGaussian sequences. Moreover, since the state s t is a deterministic function of the past observations x 1:t−1, RNNs can capture long-range dependencies, for instance matching brackets in programming languages BID19.The BPTT algorithm can be used to find the MLE of the parameters of RNN(·, ·) and g(·). However, although RNNs can, in principle, model long-range dependencies, directly applying BPTT can be difficult in practice since the repeated application of a squashing nonlinear activation function, such as tanh or logistic sigmoid, in an exponential decay in the error signal through time. LSTMs BID14 are designed to cope with the such vanishing gradient problems, by introducing an extra memory cell that is constructed as a linear combination of the previous state and signal from the input. In this work, we also use LSTMs as building blocks, as in BID35. Sequential Monte Carlo (SMC) BID8 ) is an algorithm that samples from a series of potentially unnormalized densities π 1 (z 1),..., π T (z 1:T). At each step t, SMC approximates the target density π t with P weighted particles using importance distribution f (z t |z 1:t−1): DISPLAYFORM0 where α p t is the importance weight of the p-th particle and δ x is the Dirac point mass at x. Repeating this approximation for every t leads to the SMC method, outlined in Algorithm 1.The key to this method lies in the resampling, which is implemented by repeatedly drawing the ancestors of particles at each step. Intuitively, it encourages the particles with a higher likelihood to survive longer, since the weight reflects the likelihood of the particle path. The final Monte Carlo estimate consists of only survived particle paths, and sampling from this point masses is equivalent to choosing a particle path according to the last weights α T. We refer to BID8; BID1 for detailed proof of the method. In this section, we present the class of State Space LSTM (SSL) models that combines interpretability of SSMs and flexibility of LSTMs. The key intuition, motivated by SSMs, is to learn dynamics in the state space, rather than in the sample space. However, we do not assume transition in the state space is linear, Gaussian, or Markovian. Existing approaches such as the extended Kalman filter (EKF) attempted to work with a general nonlinear transition function. Unfortunately, additional flexibility also introduced extra difficulty in the parameter estimation: EKF relies heavily on linearizing the nonlinear functions. We propose to use LSTM to model the dynamics in the latent state space, as they can learn from complex sequences without making limiting assumptions. The BPTT algorithm is also well established so that no additional approximation is needed in training the latent dynamics. Generative process Let h(·) be the emission function that maps a latent state to a parameter of the sample distribution. As illustrated in FIG2 (a), the generative process of SSL for a single sequence is: DISPLAYFORM0 The generative process specifies the following joint likelihood, with a similar factorization as SSMs except for the Markov transition: DISPLAYFORM1 where p ω (z t |z 1:t−1) = p(z t ; g(s t)), ω is the set of parameters of LSTM(·, ·) and g(·), and φ is the parameters of h(·). The structure of the likelihood function is better illustrated in FIG2, where each latent state z t is dependent to all previous states z 1:t−1 after substituting s t recursively. This allows the SSL to have non-Markovian state transition, with parsimonious parameterization thanks to the recurrent structure of LSTMs. Parameter estimation We continue with a single sequence for the ease of notation. A variational lower bound to the marginal data likelihood is given by DISPLAYFORM2 where q(z 1:T) is the variational distribution. Following the (stochastic) EM approach, iteratively maximizing the lower bound w.r.t. q and the model parameters (ω, φ) leads to the following updates:• E-step: The optimal variational distribution is given by the posterior: DISPLAYFORM3 In the case of LDS or HMM, efficient smoothing algorithms such as the RTS smoother or the forward-backward algorithm are available for computing the posterior expectations of sufficient statistics. However, without Markovian state transition, although the forward messages can still be computed, the backward recursion can no longer evaluated or efficiently approximated.• S-step: Due to the difficulties in taking expectations, we take an alternative approach to collect posterior samples instead: DISPLAYFORM4 given only the filtering equations. We discuss the posterior sampling algorithm in detail in the next section.• M-step: Given the posterior samples z 1:T, which can be seen as Monte Carlo estimate of the expectations, the subproblem for ω and φ are DISPLAYFORM5 which is exactly the MLE of an LSTM, with z 1:T serving as the input sequence, and the MLE of the given emission model. Having seen the generative model and the estimation algorithm, we can now discuss some instances of the proposed class of models. In particular, we provide two examples of SSL, for continuous and discrete latent states respectively. Example 1 (Gaussian SSL) Suppose z t and x t are real-valued vectors. A typical choice of the transition and emission is the Gaussian distribution: DISPLAYFORM6 where g µ (·) and g σ (·) map to the mean and the covariance of the Gaussian respectively, and similarly h µ (·) and h σ (·). For closed form estimates for the emission parameters, one can further assume DISPLAYFORM7 where C is a matrix that maps from state space to sample space, and R is the covariance matrix with appropriate size. The MLE of φ = (C, b, R) is then given by the least squares fit. Example 2 (Topical SSL, BID35) Consider x 1:T as the sequence of websites a user has visited. One might be tempted to model the user behavior using an LSTM, however due to the enormous size of the Internet, it is almost impossible to even compute a softmax output to get a discrete distribution over the websites. There are approximation methods for large vocabulary problems in RNN, such as the hierarchical softmax BID28. However, another interesting approach is to operate on a sequence with a "compressed" vocabulary, while learning how to perform such compression at the same time. Let z t be the indicator of a "topic", which is a distribution over the vocabulary as in BID5. Accordingly, define DISPLAYFORM8 where W is a matrix that maps LSTM states to latent states, b is a bias term, and φ zt is a point in the probability simplex. If z t lies in a lower dimension than x t, the LSTM is effectively trained over a sequence z 1:T with a reduced vocabulary. On the other hand, the probabilistic mapping between z t and x t is interpretable, as it learns to group similar x t's together. The estimation of φ is typically performed under a Dirichlet prior, which then corresponds to the MAP estimate of the Dirichlet distribution BID35. In this section, we discuss how to draw samples from the posterior, corresponding to the S-step of the stochastic EM algorithm: DISPLAYFORM0 Assuming the integration and normalization can be performed efficiently, the following quantities can be evaluated in the forward pass without Markov state transition: DISPLAYFORM1 The task is to draw from the joint posterior of z 1:T only given access to these forward messages. One way to circumvent the tight dependencies in z 1:T is to make a factorization assumption, as in BID35. More concretely, the joint distribution is decomposed as DISPLAYFORM2 where z prev 1:t−1 is the assignments from the previous inference step. However, as we confirm in the experiments, this assumption can be restrictive since the flexibility of LSTM state transitions is offset by considering each time step independently. In this work, we propose to use a method based on SMC, which is a principled way of sampling from a sequence of distributions. More importantly, it does not require the model to be Markovian BID10 BID26. As described earlier, the idea is to approximate the posterior with point masses, i.e., weighted particles. Let f (z t |z 1:t−1, x t) be the importance density, and P be the number of particles. We then can run Algorithm 1 with π t (z 1:t) = p(x 1:t, z 1:t) being the unnormalized target distribution at time t, where the weight becomes DISPLAYFORM3 As for the choice of the proposal distribution f (·), one could use the transition density p ω (z t |z 1:t−1), in which case the algorithm is also referred to as the bootstrap particle filter. An alternative is the predictive distribution, a locally optimal proposal in terms of variance BID1: DISPLAYFORM4 which is precisely one of the available forward messages: DISPLAYFORM5 Notice the similarity between terms in FORMULA8 and FORMULA0. Indeed, with the choice of predictive distribution as the proposal density, the importance weight simplifies to DISPLAYFORM6 which is not a coincidence that the name collides with the message α t. Interestingly, this quantity no longer depends on the current particle z p t. Instead, it marginalizes over all possible particle assignments of the current time step. This is beneficial computationally since the intermediate terms from can be reused in. Also note that the optimal proposal relies on the fact that the normalization in can be performed efficiently, otherwise the bootstrap proposal should be used. After a full pass over the sequence, the algorithm produces Monte Carlo approximation of the posterior and the marginal likelihood: The inference is completed by a final draw from the approximate posterior, DISPLAYFORM7 DISPLAYFORM8 which is essentially sampling a particle path indexed by the last particle. Specifically, the last particle z p T is chosen according to the final weights α T, and then earlier particles can be obtained by tracing backwards to the beginning of the sequence according to the ancestry indicators a p t at each position. Since SMC produces a Monte Carlo estimate, as the number of particles P → ∞ the approximate posterior is guaranteed to converge to the true posterior for a fixed sequence. However, as the length of the sequence T increases, the number of particles needed to provide a good approximation grows exponentially. This is the well-known depletion problem of SMC BID1.One elegant way to avoid simulating enormous number of particles is to marry the idea of MCMC with SMC BID1. The idea of such Particle MCMC (PMCMC) methods is to treat the particle estimatep(·) as a proposal, and design a Markov kernel that leaves the target distribution invariant. Since the invariance is ensured by the MCMC, it does not demand SMC to provide an accurate approximation to the true distribution, but only to give samples that are approximately distributed according to the target. As a , for any fixed P > 0 the PMCMC methods ensure the target distribution is invariant. We choose the Gibbs kernel that requires minimal modification from the basic SMC. The ing algorithm is Particle Gibbs (PG), which is a conditional SMC update in a sense that a reference path z ref 1:T with its ancestral lineage is fixed throughout the particle propagation of SMC. It can be shown that this simple modification to SMC produces a transition kernel that is not only invariant, but also ergodic under mild assumptions. In practice, we use the assignments from previous step as the reference path. The final algorithm is summarized in Algorithm 2. Combined with the stochastic EM outer iteration, the final algorithm is an instance of the particle SAEM BID25 BID33, under non-Markovian state transition. We conclude this section by deriving forward messages for the previous examples. Example 1 (Gaussian SSL, continued) The integration and normalization preserves normality thanks to the Gaussian identify. The messages are given by DISPLAYFORM9 where DISPLAYFORM10 Example 2 (Topical SSL, continued) Let θ t = softmax(W s t + b). Since the distributions are discrete, we have DISPLAYFORM11 where • denotes element-wise product. Note that the integration for α t corresponds to a summation in the state space. It is then normalized across P particles to form a weight distribution. For γ t the normalization is performed in the state space as well, hence the computation of the messages are manageable. We now present empirical studies for our proposed model and inference (denoted as SMC) in order to establish that SSL is flexible in capturing underlying nonlinear dynamics, our inference is accurate yet easily applicable to complicated models, and it opens new avenues for interpretable yet nonlinear and non-Markovian sequence models, previously unthinkable. To illustrate these claims, we evaluate on synthetic sequence tracking of varying difficulties, language modeling, and user modeling utilizing complicated models for capturing the intricate dynamics. For SMC inference, we gradually increase the number of particles P from 1 to K during training. Software & hardware All the algorithms are implemented on TensorFlow BID0. We run our experiments on a commodity machine with Intel R Xeon R CPU E5-2630 v4 CPU, 256GB RAM, and 4 NVidia R Titan X (Pascal) GPU. To test the flexibility of SSL, we begin with inference using synthetic data. We consider four different dynamics in 2D space: (i) a straight line, (ii) a sine wave, (iii) a circle, and (iv) a swiss role. Note that we do not add additional states such as velocity, keeping the dynamics nonlinear except for the first case. Data points are generated by adding zero mean Gaussian noise to the true underlying dynamics. The true dynamics and the noisy observations are plotted in the top row of FIG4. The first 60% of the sequence is used for training and the rest is left for testing. The middle and bottom row of FIG4 show the of SSL and vanilla LSTM trained for same number of iterations until both are sufficiently converged. The red points refer to the prediction of z t after observing x 1:t, and the green points are blind predictions without observing any data. We can observe that while both methods are capturing the dynamics well in general, the predictions of LSTM tend to be more sensitive to initial predictions. In contrast, even when the initial predictions are not incorrect, SSL can recover in the end by remaining on the latent dynamic. For Topical SSL, we compare our SMC inference method with the factored old algorithm BID35 documents and test on the rest, using the same settings in BID35. FIG5 shows the test perplexity (lower is better) and number of nonzeros in the learned word topic count matrix (lower is better). In all cases, the SMC inference method consistently outperforms the old factored method. For comparison, we also run LSTM with the same number of parameters, which gives the lowest test perplexity of 1942.26. However, we note that LSTM needs to perform expensive linear transformation for both embedding and softmax at every step, which depends linearly on the vocabulary size V. In contrast, SSL only depends linearly on number of topics K V. Ablation study We also want to explore the benefit of the newer inference as dataset size increases. We observe that in case of natural languages which are highly structured the gap between factored approximation and accurate SMC keeps reducing as dataset size increases. But as we will see in case of user modeling when the dataset is less structured, the factored assumption leads to poorer performance. Also when the data size is fixed and the number of topics are varying, the SMC algorithm gives better perplexity compared to the old algorithm. Therefore we the SMC inference is consistently better in various settings. Visualizing particle paths In FIG7, we show the particle paths on a snippet of an article about a music album 1. As we can see from the top row, which plots the particle paths at the initial iteration, the model proposed a number of candidate topic sequences since it is uncertain about the latent semantics yet. However, after 100 epochs, as we can see from the bottom row, the model is much more confident about the underlying topical transition. Moreover, by inspecting the learned parameters φ of the probabilistic emission, we can see that the topics are highly concentrated on topics related to music and time. This confirms our claim about flexible sequence modeling while retaining interpretability. We use an anonymized sample of user search click history to measure the accuracy of different models on predicting users future clicks. An accurate model would enable better user experience by presenting the user with relevant content. The dataset is anonymized by removing all items appearing less than a given threshold, this in a dataset with 100K vocabulary and we vary the number of users from 500K to 1M. We fix the number of topics at 500 for all user experiments. We used the same setup to the one used in the experiments over the Wikipedia dataset for parameters. The dataset is less structured than the language modeling task since users click patterns are less predictable than the sequence of words which follow definite syntactic rules. As shown in table 1, the benefit of new inference method is highlighted as it yields much lower perplexity than the factored model. In this paper we revisited the problem of posterior inference in Latent LSTM models as introduced in BID35. We generalized their model to accommodate a wide variety of state space models and most importantly we provided a more principled Sequential Monte-Carlo (SMC) algorithm for posterior inference. Although the newly proposed inference method can be slower, we showed over a variety of dataset that the new SMC based algorithm is far superior and more stable. While computation of the new SMC algorithm scales linearly with the number of particles, this can be naively parallelized. In the future we plan to extend our work to incorporate a wider class of dynamically changing structured objects such as time-evolving graphs.

We present State Space LSTM models, a combination of state space models and LSTMs, and propose an inference algorithm based on sequential Monte Carlo.

Represent the short summary for finding the corresponding scientific article (long)

We introduce two approaches for efficient and scalable inference in stochastic simulators for which the density cannot be evaluated directly due to, for example, rejection sampling loops.


Question: represent the article for finding a single-sentence tldr

Question: Given a large database of concepts but only one or a few examples of each, can we learn models for each concept that are not only generalisable, but interpretable? In this work, we aim to tackle this problem through hierarchical Bayesian program induction. We present a novel learning algorithm which can infer concepts as short, generative, stochastic programs, while learning a global prior over programs to improve generalisation and a recognition network for efficient inference. Our algorithm, Wake-Sleep-Remember (WSR), combines gradient learning for continuous parameters with neurally-guided search over programs. We show that WSR learns compelling latent programs in two tough symbolic domains: cellular automata and Gaussian process kernels. We also collect and evaluate on a new dataset, Text-Concepts, for discovering structured patterns in natural text data. A grand challenge for building more flexible AI is developing learning algorithms which quickly pick up a concept from just one or a few examples, yet still generalise well to new instances of that concept. In order to instill algorithms with the correct inductive biases, research in few-shot learning usually falls on a continuum between model-driven and data-driven approaches. Model-driven approaches place explicit domain-knowledge directly into the learner, often as a stochastic program describing how concepts and their instances are produced. For example, we can model handwritten characters with a motor program that composes distinct pen strokes BID13, or spoken words as sequences of phonemes which obey particular phonotactic constraints. Such representationally explicit models are highly interpretable and natural to compose together into larger systems, although it may be difficult to completely pre-specify the required inductive biases. By contrast, data-driven approaches start with only minimal assumptions about a domain, and instead acquire the inductive biases themselves from a large dataset. This is typified by recent work in deep meta-learning, such as the Neural Statistian BID5; see also BID9 ), MAML BID6; see also BID14 ) and Prototypical Networks BID15. Crucially, these models rely on stochastic gradient descent (SGD) for the meta-learning phase, as it is a highly scalable algorithm that applies easily to datasets with thousands of classes. Ideally these approaches would not be exclusive -for many domains of AI we have access to large volumes of data and also rich domain knowledge, so we would like to utilise both. In practice, however, different algorithms are suited to each end of the continuum: SGD requires objectives to be differentiable, but explicit domain knowledge often introduces discrete latent variables, or programs. Thus, meta-learning from large datasets is often challenging in more explicit models. In this work, we aim to bridge these two extremes: we learn concepts represented explicitly as stochastic programs, while meta-learning generative parameters and an inductive bias over programs from a large unlabelled dataset. We introduce a simple learning algorithm, Wake-Sleep-Remember (WSR), which combines SGD over continuous parameters with neurally-guided search over latent programs to maximize a variational objective, the evidence lower bound (ELBo).In evaluating our algorithm, we also release a new dataset for few-shot concept learning in a highlystructured natural domain of short text patterns (see TAB0). This dataset contains 1500 concepts such as phone numbers, dates, email addresses and serial numbers, crawled from public GitHub repositories. Such concepts are easy for humans to learn using only a few examples, and are well described as short programs which compose discrete, interpretable parts. Thus, we see this as an excellent challenge domain for structured meta-learning and explainable AI. 2 : HELMHOLTZ MACHINES AND VARIATIONAL BAYES Suppose we wish to learn generative models of spoken words unsupervised, using a large set of audio recordings. We may aim to include domain knowledge that words are built up from different short phonemes, without defining in advance exactly what the kinds of phoneme are, or exactly which phonemes occur in each recording. This means that, in order to learn a good model of words in general, we must also infer the particular latent phoneme sequence that generated each recording. This latent sequence must be re-estimated whenever the global model is updated, which itself can be a hard computational problem. To avoid a costly learning'inner-loop', a longstanding idea in machine learning is to train two distinct models simultaneously: a generative model which describes the joint distribution of latent phonemes and sounds, and a recognition model which allows phonemes to be inferred quickly from data. These two models together are often called a Helmholtz Machine BID2.Formally, algorithms for training a Helmholtz Machine are typically motivated by Variational Bayes. Suppose we wish to learn a generative model p(z, x), which is a joint distribution over latent variables z and observations x, alongside a recognition model q(z; x), which is a distribution over latent variables conditional on observations. It can be shown that the marginal likelihood of each observation is bounded below by DISPLAYFORM0 where D KL [q(z; x)||p(z|x)] is the KL divergence from the true posterior p(z|x) to the recognition model's approximate posterior q(z; x). Learning a Helmholtz machine is then framed as maximisation of this evidence lower bound (or ELBo), which provides the shared basis for two historically distinct approaches to learning. The first method, proposed by Hinton et al., is an alternating optimisation algorithm: alternate between updates to the generative model p and recognition model q. The update for p(x|z), called the'wake' phase, can be derived simply from Eq. 2 as:Wake phase: Maximise E[log p(x|z)] of observed data x using inferred latent variables z ∼ q(z; x)Unfortunately, the exact update for q(z; x), which is minimisation of D KL [q(z; x) More recently BID12 proposed the Variational Autoencoder (VAE). This offers an alternative solution to the problem of training q without relying on the above KL-divergence approximation. Instead, the authors note that it is possible to construct an unbiased approximation to the ELBo (Eq. 2) using only a single sample from q. Under specific assumptions about the form of q -specifically, that it is a continuous distribution which can be reparametrised by transforming a fixed auxiliary distribution -they use this to construct a low variance estimate for the gradient of the ELBo. As it is unbiased, this gradient estimate can used in SGD to train both q and p, typically neural networks, simultaneously towards the ELBo VAE Update: Sample z ∼ q(z; x) and take a gradient step on log DISPLAYFORM0 When z are discrete, VAEs cannot be trained through the use of reparameterisation but instead rely on the policy gradient (otherwise called) estimator from reinforcement learning. This estimator is notoriously high variance, in many cases rendering SGD ineffectual. This difficulty has motivated a wide literature on variance reduction techniques, BID8 BID11 BID16 BID7 ), yet training VAEs with discrete latent variables remains a challenging open research problem. The above description highlights a bias-variance tension between these two approaches (TAB2). The wake-sleep algorithm applies well to a wide variety of models, including structured models with discrete latent variables, but relies on an approximate update for q which may be heavily biased. By contrast, VAEs are proven to converge to a local optimum of the evidence lower bound (and so are often seen as more 'principled') but require much stronger assumptions on the form of the model in order for learning to be practical. Additionally, both VAEs and Wake-sleep rely on the ability of the recognition model, q(z; x), to learn to carry out posterior inference accurately; any departure from this changes the optimal p. This strong constraint is often unrealistic and unnecessary: on hard problems, a recognition model may still be useful if only one in a hundred samples are of high quality. Recent work aims to address this in both VAEs BID1 and Wake-sleep BID0 ) by using importance weighting over many samples from q. This solution is well suited when fully amortised inference is just out of reach of the recognition model, but is bottlenecked by how many samples it is practical to evaluate per gradient step. The next section describes our alternative approach, motivated by the idea that good explanations needn't be forgotten. Simply put, we mitigate the difficulties of discrete inference by introducing a separate'memory' into the Helmholtz Machine, explicitly keeping track of the best discovered latent explanations, or programs z i, for each observation x i. Comparison of VAE and Wake-Sleep algorithms for training Helmholtz machines. Wakesleep uses an approximation to the correct update for q, which may be heavily biased. VAE updates are unbiased, but for discrete variables they are often too high variance for learning to succeed. Figure 1: For VAEs and Wake-sleep, the recognition model q also serves as the variational distribution that trains p. WSR distinguishes these, learning a recognition model r and a categorical variational posterior q which is separate from r. This means that like VAEs, WSR jointly trains p and q using an unbiased estimate of the variational objective (blue). Like wake-sleep, the recognition model can train self-supervised (green), allowing WSR to handle discrete latent variables. To optimise the finite support of q, WSR incorporates a memory module M that remembers the best values of z i found by r(z i ; x i) across iterations. In this work we start from a different set of modelling assumptions to those typical of VAE-family models. Rather than describe each observation with a latent vector z which lacks explicit structure, we assume each observation is generated by an explicit latent program, and wish to learn:1. A posterior distribution over the latent program q i (z i) for each instance x i 2. A prior p(z) that captures a global inductive bias over programs. 4. An approximate recognition network r(z; x) which helps infer programs for novel data. Using programs as a latent representation makes this setup challenging for two reasons. First, as seen in TAB2, training discrete Helmholtz machines usually requires accepting either high bias or high variance in the learning objective. Second, by assumption, inferring programs from data is a hard problem, so performing highly accurate amortised inference may be overly ambitious. We therefore desire a learning algorithm for which weaker recognition models may reduce the speed of learning but will not change the set of stable solutions to the learning problem. To achieve this, we depart from the usual Helmholtz machines formulation by separating the recognition model from the variational distribution (Figure 1). As in Wake-sleep, we train the recognition model r(z; x) self supervised using samples from the prior -an effective strategy when z is discrete. Figure 2: What trains what? r is a recognition network trained self-supervised on samples from the p. The memory M i for each task i is a set of the'best' z values ever sampled by r, selected according their joint probability p(z, x i). p is then trained using samples from M.However, unlike Wake-sleep we do not use samples from r directly to train the generative model p. Instead, they are routed through a separate Memory module, M, which maintains a set of the best values of z found for each x, across training iterations, weighted by their joint probability p(z, x). Then, we simply resample from M in order to train p. By weighting each z proportional to its join probability, we guarantee that every update to M decreases the KL divergence between M i and the true posterior p(z i |x i). Thus, we may view each M i as a truncated variational posterior over z i, which is optimised towards the ELBo using samples from r as proposals. Our full training procedure is detailed in Algorithm 1.Algorithm 1: Basic WSR training procedure (batching omitted for notational simplicity). In practice, we avoid evaluation of p θ in the each wake phase by maintainging a cache of p θ (z M, x) in the sleep phase. We re-calculate each p θ (z, x) only as a final correctness check before modifying DISPLAYFORM0 1. Update memory with sample from recognition network DISPLAYFORM1 2. Train generative model with sample from memory DISPLAYFORM2 3. Train recognition network with sample from generative model DISPLAYFORM3 4. Train prior with sample from reference distribution g = ∇ θ s sleep + s wake (+s hyper) θ = θ + λg Gradient step (e.g. SGD, Adam) until convergence In the above problem setup, we assumed that the prior p(z) either was fixed, or was learned to maximise the ELBo training objective. However, for many modelling problems neither is adequate: we often have some idea about the global distribution over of latent programs, but deciding on the exact p(z) in advance would be too strong a commitment. In these cases we would rather provide a reference distribution p (z) as a first approximation to the global distribution, while but still allow the model to update its prior p(z) to move away from p as it learns from data. In this situation we may place a hyperprior over p(z), defined with respect to the reference distribution as: DISPLAYFORM0 where α is a concentration parameter controlling the level of confidence in the reference distribution p. This form of hyperprior can be integrated into the training objective simply by addition of an extra term: Ez∼p α log p(z), estimated by sampling from p. Algorithm 1 includes this as an optional variant, which corresponds to maximum a posteriori estimation over p. We first test our algorithm at learning the rules for noisy 1-dimensional cellular automata, from the images they generate. We create 64 × 64 binary images generated row by row, sampling each pixel using a rule that depends only on its'neighbours' in the row above. Specifically, given a'neighbourhood size' D and'corruption probability', we generate images by the following procedure:• Choose a binary vector z ∈ {0, 1} 2 D to represent the update rule for the cellular automaton.• Sample the first generation uniformly at random, as g 1 ∈ {0, 1} • For each subsequent row i = 2,..., 64 and each cell (i, j):1. read the neighbouring D cells from the previous row: DISPLAYFORM0 2. sample g ij according to: p(g ij = z sij) = 1 − Figure 3: One-shot generalisations produced by each algorithm on each the cellular automata datasets. For each input image we sample a program z from the variational distribution q, then synthesize a new image in the same style from p (z|x) using the learned.We create easy, medium and hard datasets corresponding to increasingly complex cellular automaton rules, with neighbourhood sizes of D = 3, 4, 5 respectively (easy corresponds to the full set of 256 elementary automata studied by BID18 . For medium and hard, we sample 10,000 of the 65,000 and 4 billion available rules). All datasets share a noise parameter = 0.01.Our goal is discover a latent rule, or program, z i corresponding to each image in the dataset, while also learning the global noise. Thus, we learn a p (z, x) with same structure as the true generative process, and use a CNN with independent Bernoulli outputs as the recognition network r(z; x). Fixing this architecture, we train WSR using k = 5 as the memory size, and compare performance of for the against three baseline algorithms:• VAE. We use policy-gradient (REINFORCE) for discete choices, and additionally reduce variance by subtracting a learned baseline for each task.• Wake-Sleep. We perform gradient descent on the recognition model q and generative model p together, using samples from the p to train q, and samples from q to train p.• No Recognition. We evaluate a lesioned Algorithm 1 in which no recognition model is learned. We instead propose updates to M i using samples from the prior z p ∼ p(z).Our highlight clear differences between these approaches. Despite our efforts at variance reduction, a VAE reliably struggles to get off the ground on any dataset, and instead learns quickly to model all instances as noise (Figure 3 and 4 bottom). Wake-sleep is able to learn accurate rules for images from the easiest datasets, but on the most challenging dataset but its performance appears to asymptote prematurely. By contrast, WSR reliably learns accurate programs that can be used to classify unseen images 100-way with > 99% accuracy, even on the hard dataset. Figure 4: Quantitative on all variants of the cellular automata dataset. In all cases WSR learns programs which generalise to unseen images of the same concepts, achieving > 99% accuracy on a 100-way classification task (second row). WSR also best recovers the true noise parameter = 0.01 (third row). Note: x-axis is wallclock time on a single Titan-X GPU to allow a fair comparison, as WSR requires several times more computation per iteration. Next, we evaluate our algorithm on the the task of finding explainable models for time-series data. We draw inspiration from BID4, who frame this problem as Gaussian process (GP) kernel learning. They describe a grammar for building kernels compositionally, and demonstrate that inference in this grammar can produce highly interpretable and generalisable descriptions of the structure in a time series. Inference is achieved on a single time-series through a custom greedy search algorithm, requiring a costly inner loop that approximately marginalises over kernel parameters. Here, we follow a similar approach embedded within a larger goal: we use a dataset of many different time-series, and learn a hierarchical model over time-series. That is, we learn a separate GP kernel for each series in the dataset while also learning an inductive bias over kernels themselves. We start with time series data provided by the UCR Time Series Classification Archive. This dataset contains 1-dimensional times series data from a variety of sources (such as household electricity usage, apparent brightness of stars, and seismometer readings). In this work, we use 1000 time series randomly drawn from this archive, and normalise each to zero mean and unit variance. For our model, we define the following simple grammar over kernels: DISPLAYFORM0 • WN is the White Noise kernel, DISPLAYFORM1 We wish to learn a prior distribution over both the symbolic structure of a kernel and its continuous variables (σ, l, etc.). Rather than describe a prior over kernel structures directly, we define the latent program to z to be a symbolic kernel'expression': a string over the characters {(,), +, *, WN, SE, Per, C} We define an LSTM prior p θ (z) over these kernel expressions, alongside parametric prior distributions over continuous latent variables (p θσ (σ), p θ l (l),...). As in previous work, exact evaluation of the marginal likelihood p(x|z) of a kernel expression z is intractable and so requires an approximation. For this we use a simple variational inference scheme which cycles through coordinate updates to each continuous latent variable (up to 100 steps), and estimates a lowerbound on p(x|z) using 10 samples from the variational distribution. Finally, following section 3.1, we place a hyperprior on the distribution over kernel expressions, using the grammar above (Eq. 3) as a reference distribution. Examples of latent programs discovered by our model are displayed in Figure 5. These programs describe meaningful compositional structure in the time series data, and can also be used to make highly plausible extrapolations. Figure 5: Kernels inferred by the WSR for various real time series in the UCR dataset. Blue (left) is a 256-timepoint observation, and orange (right) is a sampled extrapolation using the inferred kernel (top, simplified where possible). The explicit compositional structure of this latent representation allows each discovered concept to be easily translated into natural language. Finally, we test our model on the task of learning short text concepts, such as'phone number' or'email address', from a few examples. For this task we created a new dataset, Text-Concepts comprising 1500 of such concepts with 10 examples of each (Figure 1).To collect the data, we first crawled several thousand randomly chosen spreadsheets from online public repositories on GitHub. We then automatically selected a subset of 1500 the columns from this set, filtered to remove columns that contain only numbers, English words longer than 5 characters, or common first names and surnames. To promote diversity in the dataset, we also filtered so that no two columns originated from the same spreadsheet, and no more than 3 columns share the same column header. This us to capture a wide variety of concept types (e.g. 'date', 'time') while maintaining variation that exists within each type. Common to most patterns in the Text-Concepts dataset is that they can be well described by concatenative structure, as they usually involve the composition of discrete parts. With this in mind, we aim to model of this dataset using the language of Regular Expressions (regex).We first define a grammar over regular expressions as follows, borrowing the standard syntax that is common to many programming languages: DISPLAYFORM0 where Character can produce any printable ASCII character, and is the empty string. We assume that each class x i in the Text-Conceptsdataset can be described by a latent regular expression z i from this grammar. However, for our purposes, we endow each regex z with probabilistic, generative semantics. We define a likelihood (decoder) p θ (x|z) by placing probability distributions over every random choice involved in generating a string from regex, as given in TAB3.To evaluate the probability of a regex z generating a set of strings i, we use dynamic programming to efficiently calculate the exact probability of the most probable parse for each string in the set, and multiply these to serve as our likelihood p θ (x|z). As in the Gaussian Process example, our p θ (z) is parametrised as a simple LSTM, and we define a hyperprior over this by using the above grammar (Eq. 4) the reference grammar distribution. For the recognition model we require a network which is able to generate a sequence of tokens (the regex) taking a set of strings as an input. We achieve this using a variant of the RobustFill architecture, introduced in BID3. We pass each string in the set individually through an LSTM, and then attend to these while decoding a regular expression character by character. Given this problem setup, our goal is to learn a regex z corresponding to each set of strings, x in the dataset, while also learning a global distribution p(z) and a recognition model r(z; x) to guide inference on novel sets. For any regex expression e: e * evaluates to e+ with probability θ+, and with otherwise. e+ evaluates to ee *. e|e2 evaluates to e with probability θ |, and e2 otherwise. e? evaluates to e with probability θ?, and otherwise.. evaluates to any character, with probabilities θ. w evaluates to any alphanumeric character, with probabilities θw d evaluates to any digit, with probabilities θ d u evaluates to any uppercase character, with probabilities θu l evaluates to any lowercase character, with probabilities θ l s evaluates to any whitespace character, with probabilities θs where θ are parameters to be learned Quantitative from training the above model using the WSR algorithm (with k = 5), are shown in FIG0. From five examples of each concept, WSR learns a regular expression that generalises well to new examples, achieving over 75% accuracy in a challenging 100-way classification task. Comparing to Wake-Sleep and No-Recognition baselines, we find that WSR crucially utilises on both its recognition model and its memory in order to achieve this -neither are sufficient alone. The VAE algorithm was unable to learn effectively in our regex model, even when using control variates to reduce variance. For a more fair comparison, we also provide from training a VAE using a different model architecture to which is better suited: for VAE-LSTM we use a 32-dimensional vector for the latent representation, with a fixed Gaussian prior p(z), and LSTM networks for both p(x|z) and q(z|x). While this model is able to optimise its training objective effectively, it instead suffers from the lack of domain knowledge built into its structure. The latent representations it infers for concepts are not only less explicit but also generalise less effectively to new examples of a given concept. EA., SD., CSB.,... TAB5.3 ordered by WSR ELBo (descending).Finally, investigate whether WSR learns a realistic inductive bias over concepts, by sampling new concepts from the learned prior p θ (z) and then for each of these sampling a set of instances from p θ (x|z). In TAB5.3, we see that our model generalises meaningfully from the training data, learning higher level part structure that is common in the dataset (e.g. strings of uppercase characters) and then composing these parts in new ways. In this paper, we consider learning interpretable concepts from one or a few examples: a difficult task which gives rise to both inductive and computational challenges. Inductively, we aim to achieve strong generalisation by starting with rich domain knowledge and then'filling in the gaps', using a large amount of data. Computationally, we aim to tackle the challenge of finding high-probability programs by using a neural recognition model to guide search. Putting these pieces together we propose the Wake-Sleep-Remember algorithm, in which a Helmholtz machine is augmented with an persistent memory of discovered latent programs -optimised as a finite variational posterior. We demonstrate on several domains that our algorithm can learn generalisable concepts, and comparison with baseline models shows that WSR (a) utilises both its recognition model and its memory in order to search for programs effectively, and (b) utilises both domain knowledge and extensive data in order to make strong generalisations.

We extend the wake-sleep algorithm and use it to learn to learn structured models from few examples,

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: The knowledge regarding the function of proteins is necessary as it gives a clear picture of biological processes. Nevertheless, there are many protein sequences found and added to the databases but lacks functional annotation. The laboratory experiments take a considerable amount of time for annotation of the sequences. This arises the need to use computational techniques to classify proteins based on their functions. In our work, we have collected the data from Swiss-Prot containing 40433 proteins which is grouped into 30 families. We pass it to recurrent neural network(RNN), long short term memory(LSTM) and gated recurrent unit(GRU) model and compare it by applying trigram with deep neural network and shallow neural network on the same dataset. Through this approach, we could achieve maximum of around 78% accuracy for the classification of protein families. Proteins are considered to be essentials of life because it performs a variety of functions to sustain life. It performs DNA replication, transportation of molecules from one cell to another cell, accelerates metabolic reactions and several other important functions carried out within an organism. Proteins carry out these functions as specified by the informations encoded in the genes. Proteins are classified into three classes based on their tertiary structure as globular, membrane and fibrous proteins. Many of the globular proteins are soluble enzymes. Membrane proteins enables the transportation of electrically charged molecules past the cell membranes by providing channels. Fibrous proteins are always structural. Collagen which is a fibrous protein forms the major component of connective tissues. Escherichia coli cell is partially filled by proteins and 3% and 20% fraction of DNA and RNA respectively contains proteins. All of this contributes in making proteomics as a very important field in modern computational biology. It is therefore becoming important to predict protein family classification and study their functionalities to better understand the theory behind life cycle. Proteins are polymeric macromolecules consisting of amino acid residue chains joined by peptide bonds. And proteome of a particular cell type is a set of proteins that come under the same cell type. Proteins is framed using a primary structure represented as a sequence of 20-letter alphabets which is associated with a particular amino acid base subunit of proteins. Proteins differ from one another by the arrangement of amino acids intent on nucleotide sequence of their genes. This in the formation of specific 3D structures by protein folding which determines the unique functionality of the proteins. The primary structure of proteins is an abstracted version of the complex 3D structure but retains sufficient information for protein family classification and infer the functionality of the families. Protein family consists of a set of proteins that exhibits similar structure at sequence as well as molecular level involving same functions. The lack of knowledge of functional information about sequences in spite of the large number of sequences known, led to many works identifying family of proteins based on primary sequences BID0 BID1 BID2. Dayhoff identified the families of numerous proteins BID3. Members of the same protein family can be identified using sequence homology which is defined as the evolutionary relatedness. It also exhibits similar secondary structure through modular protein domains which further group proteins families into super families BID4. These classifications are listed in database like SCOP BID5. Protein family database (Pfam) BID6 is an extremely large source which classify proteins into family, domain, repeat or motif. Protein classification using 3D structure is burdensome and require complex techniques like X-ray crystallography and NMR spectroscopy. This led to the works BID7 BID8 BID9 which uses only primary structure for protein family classification. In this work we use data from Swiss-Prot for protein family classification and obtain a classification accuracy of about 96%.In our work we gathered family information of about 40433 protein sequences in Swiss-Prot from Protein family database(Pfam), which consists of 30 distinct families. The application of keras embedding and n-gram technique is used with deep learning architectures and traditional machine learning classifiers respectively for text classification problems in the cyber security BID33, BID34, BID35, BID36, BID37. By following, we apply keras word embedding and pass it to various deep neural network models like recurrent neural network(RNN), long short term memory(LSTM) and gated recurrent unit(GRU) and then compare it performance by applying trigram with deep and shallow neural networks for protein family classification. To verify the model used in our work, we test it over dataset consisting of about 12000 sequences from the same database. The rest of the part of this paper are organized as follows. Section 2 discusses the related work, Section 3 provides details of deep learning architecture, Section 4 discusses the proposed methodology, Section 5 provides and submissions and at last the and future work directions are placed in Section 6. There have been many works till date to identify protein functions based on the primary structures aka protein sequences. In this section we describe briefly about the works done in that area. Needleman BID10 along with Wunsch developed an algorithm using dynamic programming which uses global alignment to find similarity between protein and DNA sequences. This method is used when the sequences does not share similar patterns. Whereas in Smith work BID11 they used local alignment of protein and DNA sequences and does clustering of protein sequences based on the length of the different fragments in the sequence. In the current decade people mostly rely on computational techniques like machine learning, deep learning and pattern recognition for the classification of protein families instead of depending on the old techniques which make use of alignment of the sequences. Some of the works which uses machine learning techniques are explained briefly below. In the works BID12 BID13 BID14 primary structure aka protein sequences is used to classify protein families using classifiers like support vector machines(SVM). But apart from protein sequences these methods require additional information for feature extraction. Some of theese are polarity, hydrophobicity, surface tension, normalized Van der Waals volume, charge, polarizability, solvent accessibility and seconday structure which requires a lot of computational power to analyze. In the work BID12 protein classification for 54 families achieved 69.1-99.6% accuracy. In another study, Jeong et al. BID15 used position-specific scoring matrix(PSSM) for extracting feature from a protein sequence. They used classifiers such as Naive Bayesian(NB), Decision Tree(DT), Support Vector Machine(SVM) and Random Forest(RF) to verify their approach and achieved maximum accuracy of about 72.5%.Later on hashing was introduced for mapping high dimentional features to low dimentional features using has keys. Caragea et al BID16 used this technique to map high dimenstional features obtained through k-gram representation, by storing frequency count of each k-gram in the feature vectors obtained and hashing it together with the same hash key. This method gave accuracy of about 82.83%. Yu et al. BID17 proposed a method to represent protein sequences in the form of a k-string dictionary. For this singular value decomposition(SVD) was applied to factorize the probability matrix. Mikolov et.al. BID18 proposed a model architecture to represent words as continous vectors. This approach aka word2vec map words from the lexicon to vectors of real numbers in low dimentional space. When it is trained over a large set of data the linguistic context could be studied from the given data and it will be mapped close to each other in the euclidean space. In BID19 they have applied word2vec architecture to the biological sequences. And introduced a new representation called bio-vectors (BioVec) for the biological sequences with ProtVec for protein sequences and GeneVec for gene sequences. The k-mers derived from the data is then given as input to the embedding layer. They achieved family classification accuracy of about 93% by using ProtVec as a dense representation for biologial sequences. In our work, the proposed architecture is trained solely on primary sequence information, achieving a high accuracy when used for classification of protein families. Text representation aka text encoding can be done in several ways which are mainly of two types, sequential representation and non-sequential representation. Transforming the raw text data to these representations involves preprocessing and tokenizing the texts. During preprocessing, all uppercase characters are changed to lowercase and a dictionary is maintained which assigns a unique identification key to all the characters present in the text corpus. Later we use it in order to map input texts to vector sequence representation. After mapping character with a unique id, vocabulary is created using the training data. The preprocessing of the text data is completed by finally converting all varying length sequences to fixed length sequences. In this work we represent the text data as a sequence and therefore maintains the word order which incorporates more information to the representation. A network can be modeled by training the data using one's own representation or by using the existing ones. For our work, we have used keras embedding for text representation. This maps the discrete character ids to its vectors of continuous numbers. The character embedding captures the semantic meaning of the given protein sequence by mapping them into a high dimensional geometric space. This high dimensional geometric space is called as character embedding space. The newly formed continuous vectors are fed to other layers in the network. Features for text data can be obtained using several techniques, one of them being n-grams. N-grams can be used to represent text data which gives unique meaning when combined together, these combinations are obtained by taking continuous sequences of n characters from the given input sequence. The general equation for the N-gram approximation to the conditional probability of the next word in a sequence is, DISPLAYFORM0 In this work trigrams is used as feature for some of the models. Trigrams is a combination of three adjacent elements of a set of tokens. While computing trigram probability, we use two pseudo-words in the beginning of each sentence to create the first trigram (i.e., P(I | <s><s>).After the data is preprocessed and represented in the form of continuous vectors, it is fed into other layers like RNN LSTM GRU. RNN was developed to improve the performance of feed forward network(FFN) introduced in 1990 BID20. Both these networks differ by the way they pass the information to the nodes of the network where a series of mathematical operations are performed. FFN pass the information never touching a node twice whereas RNN pass it through a loop and ingesting their own outputs at a later moment as input, hence called recurrent. In a sequence there will be some information and RNN use it to perform the tasks that FNNs fail to do. RNN handles sequence data efficiently for natural language processing(NLP) tasks because it acts on arbitrary length sequence and the unfolded RNN model shares the weight across time steps. We can represent mathematically the process of carrying memory forward in a RNN as follows: DISPLAYFORM0 In the equation, x t is the input state at time step t and s t is the hidden state(or memory) at time step t. The function f is a nonlinearity activation function such as tanh or ReLU. During training of long sequences, vanishing and exploding gradient problem will arise due to this form of transition function BID21 BID22. To cope up with this issue long short-term memory(LSTM) was introduced BID23 using a special unit called memory block. Afterwards many variants to the LSTM architecture was introduced, prominent ones being inclusion of forget gate BID24 and peephole connections BID25.RNN shares same parameters(M, N, W in Fig. 1) at each layer unlike other traditional deep neural networks. This method reduces the total number of parameters to be learnt. To minimize the loss function an optimal weight parameter (M, N, W) is to be found, using stochastic gradient descent(SGD). Gradient estimation in RNN is done using backpropogation through time(BPTT) BID26.LSTM is an upgraded version of vanilla RNN BID27. Both LSTM and RNN uses backpropagation through time for training the network. While training a traditional RNN, there arises a case where the gradient becomes very small and further learning becomes extremely slow. This happens because the gradient vector can end up multiplied by the weight matrix a large number of times. If the values of the weight matrix is small, then it can lead to vanishing gradient. Similarly if the value of the weight matrix is high, then it can lead to exploding gradient. These problems makes the learning very difficult. The weight matrix plays a major role in training a RNN. These limitations of RNN are the key motivation of LSTM model. The LSTM model introduced the concept of memory cell. The memory cell consists of: an input gate, a neuron with a self recurrent connection, a forget gate and an output gate. In the self recurrence connection of the LSTM network, identity function is used as the activation function and has derivative 1.0. This ensures that the gradient neither explodes nor vanishes since the back-propagated gradient remains constant. Therefore the LSTM is able to learn long term dependencies BID28.Gated Recurrent Units (GRU) is a variant of LSTM recurrent neural networks BID29. Unlike other deep neural networks, GRU and LSTM have parameters specifically to control memory updation. GRU and LSTM are widely used in sequence modelling. They both can capture short term and long term dependencies in sequences. Even though both can be used in sequence modelling, the parameters in a GRU network is less compared to an LSTM network and hence the training in a GRU network is faster when compared to LSTM network. The mathematical expression for a GRU network is as follows: DISPLAYFORM1 In the above equations, x t, h t, f t represents the input, output, forget vector respectively. And W, U, b are parameter matrices and b bias respectively. Theoretically the reset and forget gate in a GRU network ensures that the memory doesn't get used up by tracing short-term dependencies. In a GRU network the memory is protected by learning how to use its gates, so as to make long term predictions. In this section, the performance of deep and shallow neural networks using bigram and RNN, LSTM and GRU with word embedding are evaluated on a data set of protein sequences. First the description of proteins and data set is discussed followed by the proposed architecture. Proteins is framed using a primary structure represented as a sequence of 20-letter alphabets which is associated with a particular amino acid base subunit of proteins. Proteins differ from one another by the arrangement of amino acids intent on nucleotide sequence of their genes which gives them different functionalities. Depending on their functions proteins are grouped under different families. Thus able to identify the family of a protein sequence will give us the information of its functions. In our work we have classified protein sequences into 30 families only using the primary structure information. We gathered family information of about 40433 protein sequences in Swiss-Prot from Protein family database(Pfam), which consists of 30 distinct families. Swiss-Prot is a curated database of primary protein sequences which is manually annotated and reviewed. There is no redundancy of protein sequences in the database and is evaluated based on obtained through experiments. We have divided the obtained protein sequences into 12000 sequences for test data and the rest 28433 protein sequences for training the model. The details of the 30 family names can be found in Table 1. The proposed architecture typically called as DeepProteomics which composed of Character Embedding, Feature representation, Regularization and Classification sections. Each section is discussed in detail below. By using the aforementioned approach, a matrix is constructed for training(28433*3988) and testing(12000*3988) for the given dataset. These matrices are then passed to an embedding layer with batch size 128. An embedding layer maps each character onto a 128 length real valued vector. This can be considered as one hyper parameter, we choose 128 to provide further level of freedom to the deep learning architectures. This collaboratively works with other layers in the deep network during backpropagation. This facilitates sequence character clustering and similar characters cluster together. This kind of character clustering facilitates other layers to easily detect the semantics and contextual similarity structures of protein sequences. For comparative study, trigram representation is constructed for protein sequence and using feature hashing approach, the protein sequence lengths are set to 1000. We adopt deep layer RNN for feature representation. Recurrent layer extract sequential information of the protein sequences. We have used RNN, LSTM and GRU as the recurrent structures. In all the experiments, 1 layer of any of the algorithms like RNN, LSTM and GRU is used. The number of units used is 128. The recurrent structuress is followed by a dropout of 0.2 while training. This in turn is followed by fully connected layer with 30 neurons in the output layer. A Dropout layer with 0.2 is used in each models between recurrent structures and fully connected layer that acts as a regularization parameter to prevent from overfitting. A Dropout is a method for removing the neurons randomly along with their connections during training a deep learning model. The embedded character level vectors coordinately works with the recurrent structures to obtain optimal feature representation. This kind of feature representation learns the similarity among the sequences. Finally, the feature representations of recurrent structures is passed to the fully-connected network to compute the probability that the sequence belongs to a particular family. The non-linear activation function in the fully connected layer facilitates in classifying the feature vectors to the respective families. The 1000 length protein sequence vectors are passed as input to shallow DNN, deep DNN and other traditional machine learning classifiers for comparative study. In fully-connected layer, each neuron in the previous layer has connection to every other neuron in the next layer. It has two layers, a fully connected layer with 128 units followed by fully connected layer with 30 units. In categorizing the proteins to 30 families, the prediction loss of deep learning models is computed using categorical-cross entropy,Where p is true probability distribution and q is predicted probability distribution. To minimize the loss of categorical-cross entropy we used Adam optimization algorithm BID31. The detailed architecture details of GRU, LSTM and RNN are placed in TAB2, Table 3 and Table 4 respectively. The detailed architecture of shallow and deep DNN module is given in TAB3 and Table 6 respectively. Table 6 Configuration details of proposed DNN Architecture All the experiments are run on GPU enabled TensorFlow BID39 and Keras BID40 higher level API. The detailed statistical measures for the protein sequence dataset for the various algorithms used is reported in TAB3. The overall performance of the neural network models are better than the traditional machine learning techniques. Thus, we claim that the character level URL embedding with deep learning layers can be a powerful method for automatic feature extraction in the case of protein family classification. In our work we have analyzed the performance of different recurrent models like RNN, LSTM and GRU after applying word embedding to the sequence data to classify the protein sequences to their respective families. We have also compared the by applying trigram with deep neural network and shallow neural network. Neural networks are preferred over traditional machine learning models because they capture optimal feature representation by themselves taking the primary protein sequences as input and give considerably high family classification accuracy of about 96%.Deep neural networks architecture is very complex therefore, understanding the mechanics of a neural network model remain as a black box and thus the internal operation of the network is only partially demonstrated. In the future work, the internal working of the network can be explored by examining the Eigenvalues and Eigenvectors across several time steps obtained by transforming the state of the network to linearized dynamics BID32.

Represent the TLDR for finding the corresponding scientific article

Proteins, amino-acid sequences, machine learning, deep learning, recurrent neural network(RNN), long short term memory(LSTM), gated recurrent unit(GRU), deep neural networks

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We proposed a universal method which can be used in the data preprocessing stage to generate the more meaningful topic that better represents the given document