Question: Represent the input

Question: In this paper, we propose an end-to-end deep learning model, called E2Efold, for RNA secondary structure prediction which can effectively take into account the inherent constraints in the problem. The key idea of E2Efold is to directly predict the RNA base-pairing matrix, and use an unrolled constrained programming algorithm as a building block in the architecture to enforce constraints. With comprehensive experiments on benchmark datasets, we demonstrate the superior performance of E2Efold: it predicts significantly better structures compared to previous SOTA (29.7% improvement in some cases in F1 scores and even larger improvement for pseudoknotted structures) and runs as efficient as the fastest algorithms in terms of inference time. Ribonucleic acid (RNA) is a molecule playing essential roles in numerous cellular processes and regulating expression of genes . It consists of an ordered sequence of nucleotides, with each nucleotide containing one of four bases: Adenine (A), Guanine (G), Cytosine (C) and Uracile (U). This sequence of bases can be represented as x:= (x 1, . . ., x L) where x i ∈ {A, G, C, U}, which is known as the primary structure of RNA. The bases can bond with one another to form a set of base-pairs, which defines the secondary structure. A secondary structure can be represented by a binary matrix A * where A * ij = 1 if the i, j-th bases are paired (Fig 1). Discovering the secondary structure of RNA is important for understanding functions of RNA since the structure essentially affects the interaction and reaction between RNA and other cellular components. Although secondary structure can be determined by experimental assays (e.g. X-ray diffraction), it is slow, expensive and technically challenging. Therefore, computational prediction of RNA secondary structure becomes an important task in RNA research and is useful in many applications such as drug design . (ii) Pseudo-knot (i) Nested Structure Research on computational prediction of RNA secondary structure from knowledge of primary structure has been carried out for decades. Most existing methods assume the secondary structure is a of energy minimization, i.e., A * = arg min A E x (A). The energy function is either estimated by physics-based thermodynamic experiments or learned from data . These approaches are faced with a common problem that the search space of all valid secondary structures is exponentially-large with respect to the length L of the sequence. To make the minimization tractable, it is often assumed the base-pairing has a nested structure (Fig 2 left), and the energy function factorizes pairwisely. With this assumption, dynamic programming (DP) based algorithms can iteratively find the optimal structure for subsequences and thus consider an enormous number of structures in time O(L 3). Although DP-based algorithms have dominated RNA structure prediction, it is notable that they restrict the search space to nested structures, which excludes some valid yet biologically important RNA secondary structures that contain'pseudoknots', i.e., elements with at least two non-nested base-pairs (Fig 2 right). Pseudoknots make up roughly 1.4% of base-pairs , and are overrepresented in functionally important regions . Furthermore, pseudoknots are present in around 40% of the RNAs. They also assist folding into 3D structures and thus should not be ignored. To predict RNA structures with pseudoknots, energy-based methods need to run more computationally intensive algorithms to decode the structures. In summary, in the presence of more complex structured output (i.e., pseudoknots), it is challenging for energy-based approaches to simultaneously take into account the complex constraints while being efficient. In this paper, we adopt a different viewpoint by assuming that the secondary structure is the output of a feed-forward function, i.e., A * = F θ (x), and propose to learn θ from data in an end-to-end fashion. It avoids the second minimization step needed in energy function based approach, and does not require the output structure to be nested. Furthermore, the feed-forward model can be fitted by directly optimizing the loss that one is interested in. Despite the above advantages of using a feed-forward model, the architecture design is challenging. To be more concrete, in the RNA case, F θ is difficult to design for the following reasons: (i) RNA secondary structure needs to obey certain hard constraints (see details in Section 3), which means certain kinds of pairings cannot occur at all . Ideally, the output of F θ needs to satisfy these constraints. (ii) The number of RNA data points is limited, so we cannot expect that a naive fully connected network can learn the predictive information and constraints directly from data. Thus, inductive biases need to be encoded into the network architecture. (iii) One may take a two-step approach, where a post-processing step can be carried out to enforce the constraints when F θ predicts an invalid structure. However, in this design, the deep network trained in the first stage is unaware of the post-processing stage, making less effective use of the potential prior knowledge encoded in the constraints. In this paper, we present an end-to-end deep learning solution which integrates the two stages. The first part of the architecture is a transformer-based deep model called Deep Score Network which represents sequence information useful for structure prediction. The second part is a multilayer network called Post-Processing Network which gradually enforces the constraints and restrict the output space. It is designed based on an unrolled algorithm for solving a constrained optimization. These two networks are coupled together and learned jointly in an end-to-end fashion. Therefore, we call our model E2Efold. By using an unrolled algorithm as the inductive bias to design Post-Processing Network, the output space of E2Efold is constrained (see Fig 3 for an illustration), which makes it easier to learn a good model in the case of limited data and also reduces the overfitting issue. Yet, the constraints encoded in E2Efold are flexible enough such that pseudoknots are included in the output space. In summary, E2Efold strikes a nice balance between model biases for learning and expressiveness for valid RNA structures. We conduct extensive experiments to compare E2Efold with state-of-the-art (SOTA) methods on several RNA benchmark datasets, showing superior performance of E2Efold including: • being able to predict valid RNA secondary structures including pseudoknots; • running as efficient as the fastest algorithm in terms of inference time; • producing structures that are visually close to the true structure; • better than previous SOTA in terms of F1 score, precision and recall. Although in this paper we focus on RNA secondary structure prediction, which presents an important and concrete problem where E2Efold leads to significant improvements, our method is generic and can be applied to other problems where constraints need to be enforced or prior knowledge is provided. We imagine that our design idea of learning unrolled algorithm to enforce constraints can also be transferred to problems such as protein folding and natural language understanding problems (e.g., building correspondence structure between different parts in a document). Classical RNA folding methods identify candidate structures for an RNA sequence energy minimization through DP and rely on thousands of experimentally-measured thermodynamic parameters. A few widely used methods such as RNAstructure, Vienna RNAfold and UNAFold achieved linear run time O(L) by applying beam search, but it can not handle pseudoknots in RNA structures. The prediction of lowest free energy structures with pseudoknots is NP-complete (Lyngsø &), so pseudoknots are not considered in most algorithms. Heuristic algorithms such as HotKnots and Probknots have been made to predict structures with pseudoknots, but the predictive accuracy and efficiency still need to be improved. Learning-based RNA folding methods such as ContraFold and ContextFold have been proposed for energy parameters estimation due to the increasing availability of known RNA structures, ing in higher prediction accuracies, but these methods still rely on the above DP-based algorithms for energy minimization. A recent deep learning model, CDPfold, applied convolutional neural networks to predict base-pairings, but it adopts the dot-bracket representation for RNA secondary structure, which can not represent pseudoknotted structures. Moreover, it requires a DP-based post-processing step whose computational complexity is prohibitive for sequences longer than a few hundreds. Learning with differentiable algorithms is a useful idea that inspires a series of works (; ; ; ;), which shared similar idea of using differentiable unrolled algorithms as a building block in neural architectures. Some models are also applied to structured prediction problems (; ;), but they did not consider the challenging RNA secondary structure problem or discuss how to properly incorporating constraints into the architecture. OptNet integrates constraints by differentiating KKT conditions, but it has cubic complexity in the number of variables and constraints, which is prohibitive for the RNA case. Dependency parsing in NLP is a different but related problem to RNA folding. It predicts the dependency between the words in a sentence. Similar to nested/non-nested structures, the corresponding terms in NLP are projective/non-projective parsing, where most works focus on the former and DP-based inference algorithms are commonly used . Deep learning models are proposed to proposed to score the dependency between words, which has a similar flavor to the Deep Score Network in our work. In the RNA secondary structure prediction problem, the input is the ordered sequence of bases x = (x 1, . . ., x L) and the output is the RNA secondary structure represented by a matrix A * ∈ {0, 1} L×L. Hard constraints on the forming of an RNA secondary structure dictate that certain kinds of pairings cannot occur at all . Formally, these constraints are: (i) Only three types of nucleotides combinations, B:= {AU, U A}∪ {GC, CG} ∪ {GU, U G}, can form base-pairs. (ii) No sharp loops are allowed. ∀|i − j| < 4, A ij = 0. (iii) There is no overlap of pairs, i.e., it is a matching. ∀i, The space of all valid secondary structures contains all symmetric matrices A ∈ {0, 1} L×L that satisfy the above three constraints. This space is much smaller than the space of all binary matrices {0, 1} L×L. Therefore, if we could incorporate these constraints in our deep model, the reduced output space could help us train a better predictive model with less training data. We do this by using an unrolled algorithm as the inductive bias to design deep architecture. In the literature on feed-forward networks for structured prediction, most models are designed using traditional deep learning architectures. However, for RNA secondary structure prediction, directly using these architectures does not work well due to the limited amount of RNA data points and the hard constraints on forming an RNA secondary structure. These challenges motivate the design of our E2Efold deep model, which combines a Deep Score Network with a Post-Processing Network based on an unrolled algorithm for solving a constrained optimization problem. The first part of E2Efold is a Deep Score Network U θ (x) whose output is an L × L symmetric matrix. Each entry of this matrix, i.e., U θ (x) ij, indicates the score of nucleotides x i and x j being paired. The x input to the network here is the L × 4 dimensional one-hot embedding. The specific architecture of U θ is shown in Fig 4. It mainly consists of by their exact and relative positions: where {ψ j} is a set of n feature maps such as sin(·), poly(·), sigmoid(·), etc, and MLP(·) denotes multi-layer perceptions. Such position embedding idea has been used in natural language modeling such as BERT , but we adapted for RNA sequence representation; • a stack of Transformer Encoders which encode the sequence information and the global dependency between nucleotides; • a 2D Convolution layers for outputting the pairwise scores. With the representation power of neural networks, the hope is that we can learn an informative U θ such that higher scoring entries in U θ (x) correspond well to actual paired bases in RNA structure. Once the score matrix U θ (x) is computed, a naive approach to use it is to choose an offset term s ∈ R (e.g., s = 0) and let A ij = 1 if U θ (x) ij > s. However, such entry-wise independent predictions of A ij may in a matrix A that violates the constraints for a valid RNA secondary structure. Therefore, a post-processing step is needed to make sure the predicted A is valid. This step could be carried out separately after U θ is learned. But such decoupling of base-pair scoring and post-processing for constraints may lead to sub-optimal , where the errors in these two stages can not be considered together and tuned together. Instead, we will introduce a Post-Processing Network which can be trained end-to-end together with U θ to enforce the constraints. The second part of E2Efold is a Post-Processing Network PP φ which is an unrolled and parameterized algorithm for solving a constrained optimization problem. We first present how we formulate the post-processing step as a constrained optimization problem and the algorithm for solving it. After that, we show how we use the algorithm as a template to design deep architecture PP φ. Formulation of constrained optimization. Given the scores predicted by U θ (x), we define the total score 1 2 i,j (U θ (x) ij − s)A ij as the objective to maximize, where s is an offset term. Clearly, without structure constraints, the optimal solution is to take A ij = 1 when U θ (x) ij > s. Intuitively, the objective measures the covariation between the entries in the scoring matrix and the A matrix. With constraints, the exact maximization becomes intractable. To make it tractable, we consider a convex relaxation of this discrete optimization to a continuous one by allowing A ij ∈. Consequently, the solution space that we consider to optimize over is A(x):= A ∈ L×L | A is symmetric and satisfies constraints (i)-(iii) in Section 3. To further simplify the search space, we define a nonlinear transformation, where • denotes element-wise multiplication. Matrix M is defined as M (x) ij:= 1 if x i x j ∈ B and also |i − j| ≥ 4, and M (x) ij:= 0 otherwise. From this definition we can see that M (x) encodes both constraint (i) and (ii). With transformation T, the ing matrix is non-negative, symmetric, and satisfies constraint (i) and (ii). Hence, by defining A:= T (Â), the solution space is simplified as Finally, we introduce a 1 penalty term Â 1:= i,j |Â ij | to make A sparse and formulate the post-processing step as: (·, · denotes matrix inner product, i.e., sum of entry-wise multiplication) The advantages of this formulation are that the variablesÂ ij are free variables in R and there are only L inequality constraints A1 ≤ 1. This system of linear inequalities can be replaced by a set of nonlinear equalities relu(A1 − 1) = 0 so that the constrained problem can be easily transformed into an unconstrained problem by introducing a Lagrange multiplier λ ∈ R Algorithm for solving it. We use proximal gradient (derived in Appendix B) for maximization and gradient descent for minimization. In each iteration,Â and λ are updated alternatively by: where soft threshold: gradient step: where α, β are step sizes and γ α, γ β are decaying coefficients. When it converges at T, an approximate solution Round A T = T (Â T) is obtained. With this algorithm operated on the learned U θ (x), even if this step is disconnected to the training phase of U θ (x), the final prediction works much better than many other existing methods (as reported in Section 6). Next, we introduce how to couple this post-processing step with the training of U θ (x) to further improve the performance. We design a Post-Processing Network, denoted by PP φ, based on the above algorithm. After it is defined, we can connect it with the deep score network U θ and train them jointly in an end-to-end fashion, so that the training phase of U θ (x) is aware of the post-processing step. Algorithm 1: Post-Processing Network PP φ (U, M) Algorithm 2: The specific computation graph of PP φ is given in Algorithm 1, whose main component is a recurrent cell which we call PPcell φ. The computation graph is almost the same as the iterative update from Eq. 3 to Eq. 6, except for several modifications: • (learnable hyperparameters) The hyperparameters including step sizes α, β, decaying rate γ α, γ β, sparsity coefficient ρ and the offset term s are treated as learnable parameters in φ, so that there is no need to tune the hyperparameters by hand but automatically learn them from data instead. • (fixed # iterations) Instead of running the iterative updates until convergence, PPcell φ is applied recursively for T iterations where T is a manually fixed number. This is why in Fig 3 the output space of E2Efold is slightly larger than the true solution space. • (smoothed sign function) Resulted from the gradient of relu(·), the update step in Eq. 4 contains a sign(·) function. However, to push gradient through PP φ, we require a differentiable update step. Therefore, we use a smoothed sign function defined as softsign(c):= 1/(1 + exp(−kc)), where k is a temperature. • (clipÂ) An additional step,Â ← min(Â, 1), is included to make the output A t at each iteration stay in the range L×L. This is useful for computing the loss over intermediate , for which we will explain more in Section 5. With these modifications, the Post-Processing Network PP φ is a tuning-free and differentiable unrolled algorithm with meaningful intermediate outputs. Combining it with the deep score network, the final deep model is 5 END-TO-END TRAINING ALGORITHM Given a dataset D containing examples of input-output pairs (x, A *), the training procedure of E2Efold is similar to standard gradient-based supervised learning. However, for RNA secondary structure prediction problems, commonly used metrics for evaluating predictive performances are F1 score, precision and recall, which are non-differentiable. Since F1 = 2TP/(2TP + FP + FN), we define a loss function to mimic the negative of F1 score as: Assuming that ij A * ij = 0, this loss is well-defined and differentiable on L×L. Precision and recall losses can be defined in a similar way, but we optimize F1 score in this paper. It is notable that this F1 loss takes advantages over other differentiable losses including 2 and cross-entropy losses, because there are much more negative samples (i.e. A ij = 0) than positive samples (i.e. A ij = 1). A hand-tuned weight is needed to balance them while using 2 or crossentropy losses, but F1 loss handles this issue automatically, which can be useful for a number of problems (; . L×L in each iteration. This allows us to add auxiliary losses to regularize the intermediate , guiding it to learn parameters which can generate a smooth solution trajectory. More specifically, we use an objective that depends on the entire trajectory of optimization: where) and γ ≤ 1 is a discounting factor. Empirically, we find it very useful to pre-train U θ using logistic regression loss. Also, it is helpful to add this additional loss to Eq. 9 as a regularization. We compare E2Efold with the SOTA and also the most commonly used methods in the RNA secondary structure prediction field on two benchmark datasets. It is revealed from the experimental that E2Efold achieves 29.7% improvement in terms of F1 score on RNAstralign dataset and it infers the RNA secondary structure as fast as the most efficient algorithm (LinearFold) among existing ones. An ablation study is also conducted to show the necessity of pushing gradient through the post-processing step. Experiments On RNAStralign. We divide RNAStralign dataset into training, testing and validation sets by stratified sampling (see details in Table 7 and Fig 6), so that each set contains all RNA types. We compare the performance of E2Efold to six methods including CDPfold, LinearFold, Mfold, RNAstructure (ProbKnot), RNAfold and CONTRAfold. Both E2Efold and CDPfold are learned from the same training/validation sets. For other methods, we directly use the provided packages or web-servers to generate predicted structures. We evaluate the F1 score, Precision and Recall for each sequence in the test set. Averaged values are reported in Table 2. As suggested by , for a base pair (i, j), the following predictions are also considered as correct:, so we also reported the metrics when one-position shift is allowed. As shown in Table 2, traditional methods can achieve a F1 score ranging from 0.433 to 0.624, which is consistent with the performance reported with their original papers. The two learning-based methods, CONTRAfold and CDPfold, can outperform classical methods with reasonable margin on some criteria. E2Efold, on the other hand, significantly outperforms all previous methods across all criteria, with at least 20% improvement. Notice that, for almost all the other methods, the recall is usually higher than precision, while for E2Efold, the precision is higher than recall. That can be the of incorporating constraints during neural network training. Fig 5 shows the distributions of F1 scores for each method. It suggests that E2Efold has consistently good performance. To estimate the performance of E2Efold on long sequences, we also compute the F1 scores weighted by the length of sequences, such that the are more dominated by longer sequences. Detailed are given in Appendix D.3. Test On ArchiveII Without Re-training. To mimic the real world scenario where the users want to predict newly discovered RNA's structures which may have a distribution different from the training dataset, we directly test the model learned from RNAStralign training set on the ArchiveII dataset, without re-training the model. To make the comparison fair, we exclude sequences that are overlapped with the RNAStralign dataset. We then test the model on sequences in ArchiveII that have overlapping RNA types (5SrRNA, 16SrRNA, etc) with the RNAStralign dataset. Results are shown in Table 3. It is understandable that the performances of classical methods which are not learning- based are consistent with that on RNAStralign. The performance of E2Efold, though is not as good as that on RNAStralign, is still better than all the other methods across different evaluation criteria. In addition, since the original ArchiveII dataset contains domain sequences (subsequences), we remove the domains and report the in Appendix D.4, which are similar to in Table 3. Inference Time Comparison. We record the running time of all algorithms for predicting RNA secondary structures on the RNAStralign test set, which is summarized in Table 4. LinearFold is the most efficient among baselines because it uses beam pruning heuristic to accelerate DP. CDPfold, which achieves higher F1 score than other baselines, however, is extremely slow due to its DP post-processing step. Since we use a gradient-based algorithm which is simple to design the PostProcessing Network, E2Efold is fast. On GPU, E2Efold has similar inference time as LinearFold. Pseudoknot Prediction. Even though E2Efold does not exclude pseudoknots, it is not sure whether it actually generates pseudoknotted structures. Therefore, we pick all sequences containing pseudoknots and compute the averaged F1 score only on this set. Besides, we count the number of pseudoknotted sequences that are predicted as pseudoknotted and report this count as true positive (TP). Similarly we report TN, FP and FN in Table 5 along with the F1 score. Most tools exclude pseudoknots while RNAstructure is the most famous one that can predict pseudoknots, so we choose it for comparison. RNAstructure CONTRAfold true structure RNAstructure CONTRAfold E2Efold true structure true structure E2Efold true structure E2Efold Visualization. We visualize predicted structures of three RNA sequences in the main text. More examples are provided in appendix (Fig 8 to 14). In these figures, purple lines indicate edges of pseudoknotted elements. Although CDPfold has higher F1 score than other baselines, its predictions are visually far from the ground-truth. Instead, RNAstructure and CONTRAfold produce comparatively more reasonable visualizations among all baselines, so we compare with them. These two methods can capture a rough sketch of the structure, but not good enough. For most cases, E2Efold produces structures most similar to the ground-truths. Moreover, it works surprisingly well for some RNA sequences that are long and very difficult to predict. Ablation Study. To exam whether integrating the two stages by pushing gradient through the post-process is necessary for performance of E2Efold, we conduct an ablation study (Table 6). We test the performance when the post-processing step is disconnected with the training of Deep Score Network U θ. We apply the post-processing step (i.e., for solving augmented Lagrangian) after U θ is learned (thus the notation "U θ + PP" in Table 6). Although "U θ + PP" performs decently well, with constraints incorporated into training, E2Efold still has significant advantages over it. Discussion. To better estimate the performance of E2Efold on different RNA types, we include the per-family F1 scores in Appendix D.5. E2Efold performs significantly better than other methods in 16S rRNA, tRNA, 5S RNA, tmRNA, and telomerase. These are from a single model. In the future, we can view it as multi-task learning and further improve the performance by learning multiple models for different RNA families and learning an additional classifier to predict which model to use for the input sequence. We propose a novel DL model, E2Efold, for RNA secondary structure prediction, which incorporates hard constraints in its architecture design. Comprehensive experiments are conducted to show the superior performance of E2Efold, no matter on quantitative criteria, running time, or visualization. Further studies need to be conducted to deal with the RNA types with less samples. Finally, we believe the idea of unrolling constrained programming and pushing gradient through post-processing can be generic and useful for other constrained structured prediction problems. Here we explain the difference between our approach and other works on unrolling optimization problems. First, our view of incorporating constraints to reduce output space and to reduce sample complexity is novel. Previous works (; ;) did not discuss these aspects. The most related work which also integrates constraints is OptNet , but its very expensive and can not scale to the RNA problem. Therefore, our proposed approach is a simple and effective one. Second, compared to , our approach has a different purpose of using the algorithm. Their goal is to learn a better algorithm, so they commonly make their architecture more flexible than the original algorithm for the room of improvement. However, we aim at enforcing constraints. To ensure that constraints are nicely incorporated, we keep the original structure of the algorithm and only make the hyperparameters learnable. Finally, although all works consider end-to-end training, none of them can directly optimize the F1 score. We proposed a differentiable loss function to mimic the F1 score/precision/recall, which is effective and also very useful when negative samples are much fewer than positive samples (or the inverse). The maximization step in Eq. 2 can be written as the following minimization: Consider the quadratic approximation of −f (Â) centered atÂ t: and rewrite the optimization in Eq. 10 as Next, we define proximal mapping as a function depending on α as follows: Since we always useÂ •Â instead ofÂ in our problem, we can take the absolute value |prox α (Ȧ t+1)| = relu(|Ȧ t+1 | − αρ) without loss of generality. Therefore, the proximal gradient step isȦ A t+1 ← relu(|Ȧ t+1 | − αρ) (correspond to Eq. 5). More specifically, in the main text, we write ∂f ∂Ât The last equation holds sinceÂ t will remain symmetric in our algorithm if the initialÂ 0 is symmetric. Moreover, in the main text, α is replaced by α · γ t α. We used Pytorch to implement the whole package of E2Efold. Deep Score Network. In the deep score network, we used a hyper-parameter, d, which was set as 10 in the final model, to control the model capacity. In the transformer encoder layers, we set the number of heads as 2, the dimension of the feed-forward network as 2048, the dropout rate as 0.1. As for the position encoding, we used 58 base functions to form the position feature map, which goes through a 3-layer fully-connected neural network (the number of hidden neurons is 5 * d) to generate the final position embedding, whose dimension is L by d. In the final output layer, the pairwise concatenation is carried out in the following way: Let X ∈ R L×3d be the input to the final output layers in Figure 4 (which is the concatenation of the sequence embedding and position embedding). The pairwise concatenation in a tensor Y ∈ R L×L×6d defined as where, and X(j, :) ∈ R 3d. In the 2D convolution layers, the the channel of the feature map gradually change from 6 * d to d, and finally to 1. We set the kernel size as 1 to translate the feature map into the final score matrix. Each 2D convolution layer is followed by a batch normalization layer. We used ReLU as the activation function within the whole score network. Post-Processing Network. In the PP network, we initialized w as 1, s as log, α as 0.01, β as 0.1, γ α as 0.99, γ β as 0.99, and ρ as 1. We set T as 20. Training details. During training, we first pre-trained a deep score network and then fine-tuned the score network and the PP network together. To pre-train the score network, we used binary crossentropy loss and Adam optimizer. Since, in the contact map, most entries are 0, we used weighted loss and set the positive sample weight as 300. The batch size was set to fully use the GPU memory, which was 20 for the Titan Xp card. We pre-train the score network for 100 epochs. As for the fine-tuning, we used binary cross-entropy loss for the score network and F1 loss for the PP network and summed up these two losses as the final loss. The user can also choose to only use the F1 loss or use another coefficient to weight the loss estimated on the score network U θ. Due to the limitation of the GPU memory, we set the batch size as 8. However, we updated the model's parameters every 30 steps to stabilize the training process. We fine-tuned the whole model for 20 epochs. Also, since the data for different RNA families are imbalanced, we up-sampled the data in the small RNA families based on their size. For the training of the score network U θ in the ablation study, it is exactly the same as the training of the above mentioned process. Except that during the fine-tune process, there is the unrolled number of iterations is set to be 0. D.1 DATASET STATISTICS Figure 6: The RNAStralign length distribution. To compare the differences among these data distributions, we can test the following hypothesis: The approach that we adopted is the permutation test on the unbiased empirical Maximum Mean Discrepancy (MMD) estimator: where contains M i.i.d. samples from a distribution P 2, and k(·, ·) is a string kernel. Since we conduct stratified sampling to split the training and testing dataset, when we perform permutation test, we use stratified re-sampling as well (for both Hypothese (a) and (b)). The of the permutation test (permuted 1000 times) is reported in Figure 7. The shows (a) Hypothesis P(RNAStr train) = P(RNAStr test) can be accepted with significance level 0.1. (b) Hypothesis P(RNAStr train) = P(ArchiveII) is rejected since the p-value is 0. Therefore, the data distribution in ArchiveII is very different from the RNAStralign training set. A good performance on ArchiveII shows a significant generalization power of E2Efold. For long sequences, E2Efold still performs better than other methods. We compute F1 scores weighted by the length of sequences (Table 8), such that the are more dominated by longer sequences. The third row reports how much F1 score drops after reweighting. Since domain sequence (subsequences) in ArchiveII are explicitly labeled, we filter them out in ArchiveII and recompute the F1 scores (Table 9). The do not change too much before or after filtering out subsequences. To balance the performance among different families, during the training phase we conducted weighted sampling of the data based on their family size. With weighted sampling, the overall F1 score (S) is 0.83, which is the same as when we did equal-weighted sampling. The per-family are shown in Table 10. RNAstructure CDPfold true structure LinearFold Mfold CONTRAfold RNAfold

represent the short summary for finding the corresponding scientific article (long)
------
Example:
Provided: "We introduce CrescendoNet, a deep CNN architecture by stacking simple building blocks without residual connections." Match: "We introduce a new deep convolutional neural network, CrescendoNet, by stacking simple building blocks without residual connections. Each Crescendo block contains independent convolution paths with increased depths. The numbers of convolution layers and parameters are only increased linearly in Crescendo blocks. In experiments, CrescendoNet with only 15 layers outperforms almost all networks without residual connections on benchmark datasets, CIFAR10, CIFAR100, and SVHN. Given sufficient amount of data as in SVHN dataset, CrescendoNet with 15 layers and 4.1M parameters can match the performance of DenseNet-BC with 250 layers and 15.3M parameters. CrescendoNet provides a new way to construct high performance deep convolutional neural networks without residual connections. Moreover, through investigating the behavior and performance of subnetworks in CrescendoNet, we note that the high performance of CrescendoNet may come from its implicit ensemble behavior, which differs from the FractalNet that is also a deep convolutional neural network without residual connections. Furthermore, the independence between paths in CrescendoNet allows us to introduce a new path-wise training procedure, which can reduce the memory needed for training. Deep convolutional neural networks (CNNs) have significantly improved the performance of image classification BID3 BID25. However, training a CNN also becomes increasingly difficult with the network deepening. One of important research efforts to overcome this difficulty is to develop new neural network architectures BID6 BID14.Recently, residual network BID3 and its variants BID8 have used residual connections among layers to train very deep CNN. The residual connections promote the feature reuse, help the gradient flow, and reduce the need for massive parameters. The ResNet BID3 and DenseNet BID8 achieved state-of-the-art accuracy on benchmark datasets. Alternatively, FractalNet BID14 expanded the convolutional layers in a fractal form to generate deep CNNs. Without residual connections BID3 and manual deep supervision BID15, FractalNet achieved high performance on image classification based on network structural design only. Many studies tried to understand reasons behind the representation view of deep CNNs. BID27 showed that residual network can be seen as an ensemble of relatively shallow effective paths. However, BID2 argued that ensembles of shallow networks cannot explain the experimental of lesioning, layer dropout, and layer reshuffling on ResNet. They proposed that residual connections have led to unrolled iterative estimation in ResNet. Meanwhile, BID14 speculated that the high performance of FractalNet was due to the unrolled iterative estimation of features of the longest path using features of shorter paths. Although unrolled iterative estimation model can explain many experimental , it is unclear how it helps improve the classification performance of ResNet and FractalNet. On the other hand, the ensemble model can explain the performance improvement easily. In this work, we propose CrescendoNet, a new deep convolutional neural network with ensemble behavior. Same as other deep CNNs, CrescendoNet is created by stacking simple building blocks, called Crescendo blocks FIG0 ). Each Crescendo block comprises a set of independent feed-forward paths with increased number of convolution and batch-norm layers (a). We only use the identical size, 3 × 3, for all convolutional filters in the entire network. Despite its simplicity, CrescendoNet shows competitive performance on benchmark CIFAR10, CI-FAR100, and SVHN datasets. Similar to FractalNet, CrescendoNet does not include residual connections. The high performance of CrescendoNet also comes completely from its network structural design. Unlike the FractalNet, in which the numbers of convolutional layers and associated parameters are increased exponentially, the numbers of convolutional layers and parameters in Crescendo blocks are increased linearly. CrescendoNet shows clear ensemble behavior (Section 3.4). In CrescendoNet, although the longer paths have better performances than those of shorter paths, the combination of different length paths have even better performance. A set of paths generally outperform its subsets. This is different from FractalNet, in which the longest path alone achieves the similar performance as the entire network does, far better than other paths do. Furthermore, the independence between paths in CrescendoNet allows us to introduce a new pathwise training procedure, in which paths in each building block are trained independently and sequentially. The path-wise procedure can reduce the memory needed for training. Especially, we can reduce the amortized memory used for training CrescendoNet to about one fourth. We summarize our contribution as follows:• We propose the Crescendo block with linearly increased convolutional and batch-norm layers. The CrescendoNet generated by stacking Crescendo blocks further demonstrates that the high performance of deep CNNs can be achieved without explicit residual learning.• Through our analysis and experiments, we discovered an emergent behavior which is significantly different from which of FractalNet. The entire CrescendoNet outperforms any subset of it can provide an insight of improving the model performance by increasing the number of paths by a pattern.• We introduce a path-wise training approach for CrescendoNet, which can lower the memory requirements without significant loss of accuracy given sufficient data. Crescendo Block The Crescendo block is built by two layers, the convolution layer with the activation function and the following batch normalization layer BID10. The convolutional layers have the identical size, 3 × 3. The Conv-Activation-BatchNorm unit f 1, defined in the Eq.1 is the base branch of the Crescendo block. We use ReLU as the activation function to avoid the problem of vanishing gradients BID17. DISPLAYFORM0 The variable z denotes the input feature maps. We use two hyper-parameters, the scale S and the interval I to define the structure of the Crescendo block H S. The interval I specifies the depth difference between every two adjacent branches and the scale S sets the number of branches per block. The structure of the n th branch is defined by the following equation: DISPLAYFORM1 where the superscript nI is the number of recursion time of the function f 1. The structure of Crescendo block H S can be obtained below: DISPLAYFORM2 where ⊕ denotes an element-wise averaging operation. Note that the feature maps from each path are averaged element-wise, leaving the width of the channel unchanged. A Crescendo block with S = 4 and I = 1 is shown in FIG0.The structure of Crescendo block is designed for exploiting more feature expressiveness. The different depths of parallel paths lead to different receptive fields and therefore generate features in different abstract levels. In addition, such an incremental and parallel form explicitly supports the ensemble effects, which shows excellent characteristics for efficient training and anytime classification. We will explain and demonstrate this in the following sections. CrescendoNet Architecture The main body of CrescendoNet is composed of stacked Crescendo blocks with max-pooling layers between adjacent blocks (FIG0). Following the main body, like most deep CNNs, we use two fully connected layers and a soft-max layer as the classifier. In all experiments, the two fully connected layers have 384 hidden units and 192 hidden units respectively. The overall structure of CrescendoNet is simple and we only need to tune the Crescendo block to modify the entire network. To reduce the memory consumption during training CrescendoNet, we propose a path-wise training procedure, leveraging the independent multi-path structure of our model. We denote stacked Conv-BatchNorm layers in one Crescendo block as one path. We train each path individually, from the shortest to the longest repetitively. When we are training one path, we freeze the parameters of other paths. In other words, these frozen layers only provide learned features to support the training. FIG1 illustrates the procedure of path-wise training within a CrescendoNet block containing four paths. There are two advantages of path-wise training. First, path-wise training procedure significantly reduces the memory requirements for convolutional layers, which constitutes the major memory cost for training CNNs. For example, the higher bound of the memory required for computation and storage of gradients using momentum stochastic gradient descent algorithms can be reduced to about 40% for a Crescendo block with 4 paths where interval = 1. Second, path-wise training works well with various optimizers and regularizations. Even dropout and drop-path can be applied to the model during the training. Dropout and drop-connect BID28, which randomly set a selected subset of activations or weights to zero respectively, are effective regularization techniques for deep neural networks. Their variant, drop-path BID14, shows further performance improvement by dropping paths when training FractalNet. We use both dropout and drop-path for regularizing the Crescendo block. We drop the branches in each block with a predefined probability. For example, given drop-path rate, p = 0.3, the expectation of the number of dropped branches is 1.2 for a Crescendo block with four branches. For the fully connected layers, we use L2 norm of their weights as an additional term to the loss. We evaluate our models with three benchmark datasets: CIFAR10, CIFAR100, and Street View House Numbers (SVHN) BID19. CIFAR10 and CIFAR100 each have 50,000 training images and 10,000 test images, belonging to 10 and 100 classes respectively. All the images are in RGB format with the size of 32 × 32-pixel. SVHN are color images, with the same size of 32 × 32-pixel, containing 604,388 and 26,032 images for training and testing respectively. Note that these digits are cropped from a series of numbers. Thus, there may be more than one digit in an image, but only the one in the center is used as the label. For data augmentation, we use a widely adopted scheme BID16 BID14 BID6 b; BID24 BID21 BID3. We first pad images with 4 zero pixels on each side, then crop padded images to 32 × 32-pixel randomly and horizontally flipping with a 50% probability. We preprocess each image in all three datasets by subtracting off the mean and dividing the variance of the pixels. We use Mini-batch gradient descent to train all our models. We implement our models using TensorFlow distributed computation framework BID0 and run them on NVidia P100 GPU. We also optimize our models by adaptive momentum estimation (Adam) optimization BID11 and Nesterov Momentum optimization BID18 respectively. For Adam optimization, we set the learning rate hyper-parameter to 0.001 and let Adam adaptively tune the learning rate during the training. We choose the momentum decay hyper-parameter β 1 = 0.9 and β 2 = 0.999. And we set the smoothing term = 10 −8. This configuration is the default setting for the AdamOptimizer class in TensorFlow. For Nesterov Momentum optimization, we set the hyper-parameter momentum = 0.9. We decay the learning rate from 0.1 to 0.01 after 512 epochs for CIFAR and from 0.05 to 0.005, then to 0.0005, after 42 epochs and 63 epochs respectively for SVHN. We use truncated normal distribution for parameter initialization. The standard deviation of hyper-parameters is 0.05 for convolutional weights and 0.04 for fully connected layer weights. For all datasets, we use the batch size of 128 on each training replica. For the whole net training, we run 700 epochs on CIFAR and 70 epochs on SVHN. For the path-wise training, we run 1400 epochs on CIFAR and 100 epochs on SVHN.Using a CrescendoNet model with three blocks each contains four branches as illustrated in FIG0, we investigate the following preliminary aspects: the model performance under different block widths, the ensemble effect, and the path-wise training performance. We study the Crescendo block with three different width configurations: equal width globally, equal width within the block, and increasing width. All the three configurations have the same fully connected layers. For the first one, we set the number of feature maps to 128 for all the convolutional layers. For the second, the numbers of feature maps are for convolutional layers in each block. For the last, we gradually increase the feature maps for each branch in three blocks to correspondingly. For example, the number of feature maps for the second and fourth branches in the second block is and. The exact number of maps for each layer is defined by the following equation: DISPLAYFORM0 where n maps denotes the number of feature maps for a layer, n inmaps and n outmaps are number of input and output maps respectively, n layers is the number of layers in the block, and i layer is the index of the layer in the branch, starting from 1.To inspect the ensemble behavior of CrescendoNet, we compare the performance of models with and without drop-path technique and subnets composed by different combinations of branches in each block. For the simplicity, we denote the branch combination as a set P containing the index of the branch. For example, P = {1, 3} means the blocks in the subnet only contains the first and third branches. The same notation is used in TAB0 and FIG2. Table 1 gives a comparison among CrescendoNet and other representative models on CIFAR and SVHN benchmark datasets. For five datasets, CrescendoNet with only 15 layers outperforms almost all networks without residual connections, plus original ResNet and ResNet with Stochastic Depth. For CIFAR10 and CIFAR100 without data augmentation, CrescendoNet also performs better than all the given models except DenseNet with bottleneck layers and compression (DenseNet-BC) with 250 layers. However, CrescendoNet's error rate 1.76% matches the 1.74% error rate of given DenseNet-BC, on SVHN dataset which has plentiful data for each class. Comparing with FractalNet, another outstanding model without residual connection, CrescendoNet has a simpler structure, fewer parameters, but higher accuracies. The lower rows in Table 1 compare the performance of our model given different configuration. In three different widths, the performance simultaneously grows with the number of feature maps. In other words, there is no over-fitting when we increase the capacity of CrescendoNet in an appropriate scope. Thus, CrescendoNet demonstrates a potential to further improve its performance by scaling up. In addition, the drop-path technique shows its benefits to our models on all the datasets, just as it does to FractalNet. Another interesting from Table 1 is the performance comparison between Adam and Nesterov Momentum optimization methods. Comparing with Nesterov Momentum method, Adam performs similarly on CIFAR10 and SVHN, but worse on CIFAR100. Note that there are roughly 60000, 5000, and 500 training images for each class in SVHN, CIFAR10, and CIFAR100 respectively. This implies that Adam may be a better option for training CrescendoNet when the training data is abundant, due to the convenience of its adaptive learning rate scheduling. The last row of Table 1 gives the from path-wise training. Training the model with less memory requirement can be achieved at the cost of some performance degradation. However, Pathwise trained CrescendoNet still outperform many of networks without residual connections on given datasets.. The show the ensemble behavior of our model. Specifically, the more paths contained in the network, the better the Table 1: Whole net classification error (%) on CIFAR10/CIFAR100/SVHN. We highlight the top three accuracies in each column with the bold font. The three numbers in the parentheses denote the number of output feature maps of each block. The plus sign (+) denotes the data augmentation. The sign (-W) means that the feature maps of layers in each branch increase as explained in the model configuration section. The compared models include: Network in Network BID24, ALL-CNN , Deeply Supervised Net BID15, Highway Network BID24, FractalNet BID14, ResNet BID3, ResNet with Stochastic Depth BID7, Wide ResNet BID30, and DenseNet BID6 performance. And the whole net outperforms any single path network with a large margin. For example, the whole net and the net based on the longest path show the inference error rate of 6.90% and 10.69% respectively, for CIFAR10 without data augmentation. This implicit ensemble behavior differentiates CrescendoNet from FractalNet, which shows a student-teacher effect. Specifically, the longest path in FractalNet can achieve a similar or even lower error rate compared to the whole net. To investigate the dynamic behavior of subnets, we test the error rate changes of subnets during the training. We use Adam to train the CrescendoNet with the structure shown in FIG0 on CIFAR10 for 450 epochs. FIG2 illustrates the behavior of different path combinations during the training. It shows that the inference accuracy of the whole net grows simultaneously with all the subnets, which demonstrates the ensemble effect. Second, for any single path network, the performance grows with the depth. This behavior of the anytime classifier is also shown by FractalNet. In other words, we could use the short path network to give a rough but quick inference, then use more paths to gradually increase the accuracy. This may be useful for time-critical applications, like integrated recognition system for autonomous driving tasks. Conventional deep CNNs, such as AlexNet VGG-19 (Simonyan & BID20, directly stacked the convolutional layers. However, the vanishing gradient problem makes it difficult to train and tune very deep CNN of conventional structures. Recently, stacking small convolutional blocks has become an important method to build deep CNNs. Introducing new building blocks becomes the key to improve the performance of deep CNN. BID16 first introduced the NetworkInNetwork module which is a micro neural network using a multiple layer perceptron (MLP) for local modeling. Then, they piled the micro neural networks into a deep macro neural network. BID25 introduced a new building block called Inception, based on which they built GoogLeNet. Each Inception block has four branches of shallow CNNs, building by convolutional kernels with size 1 × 1, 3 × 3, 5 × 5, and max-pooling with kernel size 3 × 3. Such a multiple-branch scheme is used to extract diversified features while reducing the need for tuning the convolutional sizes. The main body of GoogLeNet has 9 Inception blocks stacked each other. Stacking multiplebranch blocks can create an exponential combination of feed-forward paths. Such a structure com-bined with the dropout technique can show an implicit ensemble effect BID27 BID22. GoogLeNet was further improved with new blocks to more powerful models, such as Xception BID1 and Inception-v4 BID26. To improve the scalability of GoogLeNet, BID26 used convolution factorization and label-smoothing regularization in Inception-v4. In addition, BID1 explicitly defined a depth-wise separable convolution module replacing Inception module. Recently, BID14 introduced FractalNet built by stacked Fractal blocks, which are the combination of identical convolutional layers in a fractal expansion fashion. FractalNet showed that it is possible to train very deep neural network through the network architecture design. FractalNet implicitly also achieved deep supervision and student-teacher learning by the fractal architecture. However, the fractal expansion form increases the number of convolution layers and associated parameters exponentially. For example, the original FractalNet model with 21 layers has 38.6 million parameters, while a ResNet of depth 1001 with similar accuracy has only 10.2 million parameters BID6. Thus, the exponential expansion reduced the scalability of FractalNet. Another successful idea in network architecture design is the use of skip-connections BID3 b; BID6 BID30 BID29. ResNet BID3 used the identity mapping to short connect stacked convolutional layers, which allows the data to pass from a layer to its subsequent layers. With the identity mapping, it is possible to train a 1000-layer convolutional neural network. BID6 recently proposed DenseNet with extremely residual connections. They connected each layer in the Dense block to every subsequent layer. DenseNet achieved the best performance on benchmark datasets so far. On the other hand, Highway networks BID23 used skip-connections to adaptively infuse the input and output of traditional stacked neural network layers. Highway networks have helped to achieve high performance in language modeling and translation. CNN has shown excellent performance on image recognition tasks. However, it is still challenging to tune, modify, and design an CNN. We propose CrescendoNet, which has a simple convolutional neural network architecture without residual connections BID3. Crescendo block uses convolutional layers with same size 3 × 3 and joins feature maps from each branch by the averaging operation. The number of convolutional layers grows linearly in CrescendoNet while exponentially in FractalNet BID14. This leads to a significant reduction of computational complexity. Even with much fewer layers and a simpler structure, CrescendoNet matches the performance of the original and most of the variants of ResNet on CIFAR10 and CIFAR100 classification tasks. Like FractalNet BID14, we use dropout and drop-path as regularization mechanisms, which can train CrescendoNet to be an anytime classifier, namely, CrescendoNet can perform inference with any combination of the branches according to the latency requirements. Our experiments also demonstrated that CrescendoNet synergized well with Adam optimization, especially when the training data is sufficient. In other words, we can avoid scheduling the learning rate which is usually performed empirically for training existing CNN architectures. CrescendoNet shows a different behavior from FractalNet in experiments on CIFAR10/100 and SVHN. In FractalNet BID14, the longest path alone achieves the similar performance as the entire network, far better than other paths, which shows the student-teacher effect. The whole FractalNet except the longest path acts as a scaffold for the training and becomes dispensable later. On the other hand, CrescendoNet shows that the whole network significantly outperforms any set of it. This fact sheds the light on exploring the mechanism which can improve the performance of deep CNNs by increasing the number of paths."

A DL model for RNA secondary structure prediction, which uses an unrolled algorithm in the architecture to enforce constraints.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.


Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: Learning in recurrent neural networks (RNNs) is most often implemented by gradient descent using backpropagation through time (BPTT), but BPTT does not model accurately how the brain learns. Instead, many experimental on synaptic plasticity can be summarized as three-factor learning rules involving eligibility traces of the local neural activity and a third factor. We present here eligibility propagation (e-prop), a new factorization of the loss gradients in RNNs that fits the framework of three factor learning rules when derived for biophysical spiking neuron models. When tested on the TIMIT speech recognition benchmark, it is competitive with BPTT both for training artificial LSTM networks and spiking RNNs. Further analysis suggests that the diversity of learning signals and the consideration of slow internal neural dynamics are decisive to the learning efficiency of e-prop. The brain seems to be able to solve tasks such as counting, memorizing and reasoning which require efficient temporal processing capabilities. It is natural to model this with recurrent neural networks (RNNs), but their canonical training algorithm called backpropagation through time (BPTT) does not appear to be compatible with learning mechanisms observed in the brain. There, long-term changes of synaptic efficacies depend on the local neural activity. It was found that the precise timing of the electric pulses (i.e. spikes) emitted by the pre-and post-synaptic neurons matters, and these spike-timing dependent plasticity (STDP) changes can be conditioned or modulated by a third factor that is often thought to be a neuromodulator (see for reviews). Looking closely at the relative timing, the third factor affects the plasticity even if it arrives with a delay. This suggests the existence of local mechanisms that retain traces of the recent neural activity during this temporal gap and they are often referred to as eligibility traces. To verify whether three factor learning rules can implement functional learning algorithms, researchers have simulated how interesting learnt behaviours can emerge from them. The third factor is often considered as a global signal emitted when a reward is received or predicted, and this alone can solve learning tasks of moderate difficulty, even in RNNs. Yet in feed-forward networks, it was already shown that plausible learning algorithms inspired by backpropagation and ing in neuron-specific learning signals largely outperform the rules based on a global third factor. This suggests that backpropagation provides important details that are not captured by all three factor learning rules. Here we aim at a learning algorithm for RNNs that is general and efficient like BPTT but remains plausible. A major plausibility issue of BPTT is that it requires to propagate errors backwards in time or to store the entire state space trajectory raising questions on how and where this is performed in the brain. We suggest instead a rigorous re-analysis of gradient descent in RNNs that leads to a gradient computation relying on a diversity of learning signals (i.e. neuron-specific third factors) and a few eligibility traces per synapse. We refer to this algorithm as eligibility propagation (e-prop). When derived with spiking neurons, e-prop fits under the three factor learning rule framework and is qualitatively compatible with experimental data. To test its learning efficiency, we applied e-prop to artificial Long Short-Term Memory (LSTM) networks, and Long short-term memory Spiking Neural Networks (LSNNs) (spiking RNNs combining short and long realistic time constants). We found that it is competitive with BPTT on the TIMIT speech recognition benchmark, and it can solve nontrivial temporal credit assignment problems with long delays. We are not aware of any comparable achievements with previous three factor learning rules. Real-time recurrent learning (RTRL) computes the same loss gradients as BPTT in an online fashion but requires many more operations. Eventhough the method is online, one may wonder where can it be implemented in the brain if it requires a machinery bigger than the network itself. Recent works have suggested that eligibility traces can be used to approximate RTRL. This was shown to be feasible if the neurons do not have recurrent connections, if the recurrent connections are ignored during learning or if the network dynamics are approximated with a trained estimator. However these algorithms were derived for specific neuron models without long-short term memory, making it harder to tackle challenging RNN benchmark tasks (no machine learning benchmarks were considered in ). Other mathematical methods, have suggested approximations to RTRL which are compatible with complex neuron models. Yet those methods lead to gradient estimates with a high variance or requiring heavier computations when the network becomes large. This issue was solved in e-prop, as the computational and memory costs are the same (up to constant factor) as for running any computation with the RNN. This reduction of the computational load arises from an essential difference between e-prop and RTRL: e-prop computes the same loss gradients but only propagates forward in time the terms that can be computed locally. This provides a new interpretation of eligibility traces that is mathematically grounded and generalizes to a broad class of RNNs. Our empirical show that such traces are sufficient to approach the performance of BPTT despite a simplification of the non-local learning signal, but we believe that more complex strategies for computing a learning signals can be combined with e-prop to yield even more powerful online algorithms. A separate paper presents one such example to enable one-shot learning in recurrent spiking neural networks. The mathematical basis for e-prop E-prop applies for a general class of recurrent network models that includes LSTMs and LSNNs, where each neuron j has a hidden state h j P R d where d is typically 1 or 2 (e.g. the memory cell content of an LSTM unit or the membrane potential for a spiking neuron), and an observable state z t j P R (e.g. the LSTM outputs or the spikes). The performance of a network on a specific task is usually expressed using a loss function E, and learning by gradient descent learning amounts to changing the network weights W such that E is minimized. Much like BPTT, e-prop computes the gradient dE dWji with respect to the weights from i to j where the neurons i and j are potentially connected. Here, this gradients depends on learning signals L t j specific to the neuron j and eligibility traces e t ji such that (see proof in The eligibility traces e This allows the loss-independent eligibility traces to be defined for any RNN model. For equation to hold, the ideal learning signal is required to be L are the output targets. E-prop can be implemented online by accumulating the products L j e t ji or applying them directly at each time step and does not require to backpropagate through time or to store the past neural activity. This solves a major plausibility issue raised by BPTT. To also avoid the implausible weight sharing between the feedback and feedforward pathways, one can replace the feedback weights in L t j by fixed random values as done in leading to the two variants: symmetric and random e-prop. E-prop for spiking neurons and data on synaptic plasticity To link the theory to data on STDP and three factor learning rules, we applied e-prop to a recurrent network of spiking neurons. We use leaky-integrate and fire (LIF) neurons for which the dynamics of the membrane voltage is modelled as a leaky integrator and a spike is emitted when the voltage crosses a firing threshold from below. When simulated in discrete time with a time step of one millisecond, a recurrent network of LIF neurons fits into the general class of RNNs described above such that the hidden state h t j is the membrane voltage and the spikes are modelled by a binary observable state z t j P t0, 1u. Non-differentiability of the binary output of spiking neurons is solved as in, by using a pseudo-derivative in-place of. Remarkably, the eligibility trace e t ji that emerges for a LIF neuron is a product of the pre-and post-synaptic activity. It is non-zero only if pre-synaptic spikes have preceded a depolarization of the post-synaptic neuron in a time window of about 20 ms which is reminiscent of STDP. Moreover it was verified in that the replacement of e t ji in equation by a form of voltage dependent STDP used to fit data accurately, does not strongly impair the performance of e-prop on a pattern generation task. Forward and stable propagation of RNN gradients with eligibility traces To enhance the working memory capabilities of the spiking network model, we model slower neural dynamics by introducing a model of firing rate adaptation: after each spike the threshold increases by a constant amount and reduces back to its resting value after hundreds of milliseconds or few seconds. This type of adaptive LIF (ALIF) neuron also includes its current firing threshold in its hidden state h recurrent network of ALIF and LIF neurons connected in an all-to-all fashion is termed LSNN. For an ALIF neuron j each eligibility vector t ji include a slow component t ji,a that decays with the time constants of adaptation, i.e. much slower than for non-adaptive LIF neurons. We tested the ability of e-prop and LSNNs to learn temporal dependencies on a task that was used to study working memory in rodents and that requires to memorize over a delay of hundreds of milliseconds. It requires the rodent to run along a linear track in a virtual environment, where it encounters a number of visual cues to its left and its right, see Figure 1A. After arrival at a T-junction, it has to decide whether it had observed more cues to the left or on the right side and turn towards that direction. This task requires to establish a connection between errors in the decision and the processing of cues that happened a long time ago. We found that LSNNs with 100 neurons can be trained by e-prop to solve this task (Figure 1B), but a similar network of LIF neurons without adaptation cannot solve the task (Figure 1C). The key feature arising with adaptive neurons is the slow component t ji,a of the eligibility vector associated with the threshold adaptation and sharing its slow dynamics (see Figure 1B bottom). As the learning signal L t j is only non-zero during the decision period at the last time steps of the episode, the eligibility traces must hold the information about the relevant cues for hundreds of time steps during the delay (see Figure 1A). In this way e-prop alleviates the need to propagate signals backwards in time. Approaching the performance of BPTT We compare E-prop and BPTT on two benchmarks for RNNs based on the TIMIT dataset: phoneme classification of each audio frame in a spoken sentence, and transcription of the entire sequence of phonemes spoken in a sentence. The LSTM and BPTT baselines were obtained by reproducing the experiments from solving framewise-phoneme classification with 400 LSTM units and solving sentence transcription with three layers of 500 LSTM units. For the LSNN we used 800 spiking neurons in the first task and three layers of 800 spiking neurons in the second. Remarkably, the error rate obtained with e-prop is only a few percents larger than the BPTT baseline in all cases, even if the feedback weights are replaced by random ones (Figure 1D). In contrast, the loss performance of e-prop when using uniform feedback matrices was significantly worse: the error rate jumped from 34.6 to 52% for frame-wise classification and from 24.7 to 60% for the speech transcription. Beyond supervised learning task it is also shown in that e-prop can be applied on reinforcement learning tasks. Discussion E-prop is a novel learning algorithm that qualitatively fits experimental data on synaptic plasticity and maintains a performance that is competitive with BPTT. Our analysis shows that e-prop can take advantage of neuron models with enhanced memory capabilities to solve non-trivial temporal credit assignment problems, and the diversity of the learning signals is decisive for the learning efficiency of three factor learning rules. Interestingly, it was found recently that dopaminergic neurons encode more diverse information than a global reward prediction error, and performance monitoring neurons providing potential learning signals are found prominently in cortices.

Represent the TLDR for finding the corresponding scientific article

We present eligibility propagation an alternative to BPTT that is compatible with experimental data on synaptic plasticity and competes with BPTT on machine learning benchmarks.

represent the tldr for finding the corresponding scientific article:

To give you a sense - "Evaluating pixel-level out-of-distribution detection methods on two new real world datasets using PSPNet and DeeplabV3+." should be close to "The detection of out of distribution samples for image classification has been widely researched. Safety critical applications, such as autonomous driving, would benefit from the ability to localise the unusual objects causing the image to be out of distribution. This paper adapts state-of-the-art methods for detecting out of distribution images for image classification to the new task of detecting out of distribution pixels, which can localise the unusual objects. It further experimentally compares the adapted methods on two new datasets derived from existing semantic segmentation datasets using PSPNet and DeeplabV3+ architectures, as well as proposing a new metric for the task. The evaluation shows that the performance ranking of the compared methods does not transfer to the new task and every method performs significantly worse than their image-level counterparts. Figure 1: Image from the LostAndFound dataset , where two unlikely objects (storage crates) are almost entirely incorrectly predicted to be road. The Max Softmax method clearly highlights these crates as OOD. (best viewed in colour) Many applications using machine learning (ML) may benefit from out of distribution (OOD) detection to improve safety. When inputs are determined to be out of distribution, the output of an ML algorithm should not be trusted. A large body of research exists for detecting entire images as OOD for the task of image classification. Image-level OOD detection outputs a classification for the entire image; this coarse level of detection may be inadequate for many safety critical applications, including autonomous driving. Most of the pixels in an image taken from an onboard camera will be in distribution (ID), i.e. an image of a road scene with cars, people, and roadway-but an unusual object that was not part of the training set may cause only a small number of OOD pixels. Extending the framework to semantic segmentation networks will allow each pixel to have an "in" or "out of" distribution classification. Applied to autonomous driving, groups of pixels classified as OOD would be considered as unknown objects. Depending on the location of the unknown objects, a planner would then proceed with caution or hand over control to a safety driver. Another application is automatic tagging of images with OOD objects, which would then be sent for human labelling. Figure 1 shows a failure case where OOD detection is beneficial. The two crates are predicted as road. The right image of this figure shows the of pixel-level OOD detection using one of the proposed methods, which clearly identifies the unusual objects. This paper adapts existing state-of-the-art image-level OOD detection methods to the new task of pixel-level OOD classification and compares their performance on a new dataset designed for this task. In addition to adapting the methods, we address the question of whether the best-performing image-level methods maintain their performance when adapted to the new task. In order to answer this question, we also propose pixel-level OOD detection performance metrics, drawing both on existing image-level OOD detection and semantic segmentation performance metrics. Further, we design two new datasets for pixel-level OOD detection with test images that contain both pixels that are in distribution and pixels that are out of distribution, evaluated with two different network architectures-PSPNet and DeeplabV3+ . Somewhat surprisingly, our evaluation shows that the best performing pixel-level OOD detection methods were derived from image-level OOD detection methods that were not necessarily the best performing on the image-level OOD detection task. In summary, the contributions of this paper are the following: • adaptation of image-level OOD detection methods to pixel-level OOD detection and their evaluation; • training and evaluation datasets for pixel-level OOD detection evaluation derived from existing segmentation datasets; and • a new metric for pixel-level OOD detection, called MaxIoU. We use two criteria to select existing image-level OOD detection methods to adapt to pixellevel OOD detection. First, the candidate methods are top performers on image classification datasets. Second, they must be computationally feasible for semantic segmentation. Max Softmax , ODIN , Mahalanobis and Confidence fit both criteria. The Entropy , Sum of Variances (VarSum) , and Mutual Information methods do not meet the first criterion, but are included as an existing uncertainty baseline for the pixel-level OOD classification. An example method that does not meet the second criterion is an ensemble method by , with an ensemble of K leave-out classifiers. In general, images and architectures for semantic segmentation are larger than for image classification, and therefore an ensemble method is much less feasible for segmentation than classification due to GPU memory limitations. Generative Adversarial Networks (GANs) and Auto-Encoders (AEs) are often used in the literature as a method for OOD detection. GANs and AEs are excluded to limit the scope of this work. Table 1 briefly describes the selected image-level OOD detection methods and any architecture modifications necessary to adapt them to OOD detection. 1 The Dim reduction modification is an additional penultimate layer that is a 1 × 1 convolution reducing the depth to 32. DeeplabV3+ has a much larger feature extractor than PSPNet, therefore due to hardware limitation, the Mahalanobis method is not evaluated on this architecture. Each original/adapted method produces a value that can be thresholded to predict whether an image/pixel is OOD. All metrics used in our evaluation are threshold independent, therefore no thresholds are discussed in this section. See Appendix A for more detailed description of each OOD detection method. to OOD data prior to evaluation. This unfair comparison has been criticised in the past . Most methods have a input perturbation step; however, this is removed as it also requires access to OOD data before evaluation. Previous work on image-level OOD detection designates a dataset used for training as the ID dataset, and the dataset for testing as the OOD dataset. This framework is not easily extended to semantic segmentation, as any two datasets may share features with the ID dataset. For example, people may exist in both datasets, but one is indoor scenes and the other is outdoor scenes. attempt to get around this issue by inserting animals from the COCO dataset or internet images into the Cityscapes dataset. The lack of realism of the inserted images (improper scale and hard boundaries) make the created dataset insufficient for OOD detection. The remainder of this section describes the datasets used in this work and any modifications required. The dataset used for training the weights of all networks is the unmodified Cityscapes dataset . It is the most prominent dataset for training and benchmarking semantic segmentation for road scene images. The diverse SUN dataset and the India Driving Dataset (IDD) are used as the main evaluation datasets. The main motivation for using the SUN dataset is that it has a large variety of scenes (e.g. street, forest, and conference room), as well as a large variety of labels (e.g. road, door, and vase). In total there are 908 scene categories and 3819 label categories. Anonymous label submissions are ignored, as their validity is not confirmed. However, the SUN dataset is not very realistic in terms of what a vehicle camera would see. The IDD dataset is an autonomous vehicle dataset that has all the same labels of Cityscapes with the addition of the autorickshaw class. Although the classes are the same, the instances tend to be OOD, due to different building architecture, road infrastructure, etc. This shift in features leads to higher average values OOD predictions as shown in Figure 2. Therefore only the car class as ID and auto-rickshaw class as OOD are used for evaluation. The SUN dataset labels have to be modified before the dataset can be used for evaluating OOD detection. 2 The approach for modifying this dataset is similar to the open world dataset created by. Let C = {person, car, ignore, ...} be the set of labels available in Cityscapes. Let S = {person, roof, chair, ...} be the set of labels available in the SUN dataset. Ambiguous classes such as "wall" (can appear indoor and outdoor) or "path" (a sidewalk or a dirt path in the woods) are sorted into a third set A ⊂ S. The map M: S → C ∪ {OOD} is defined as: For every image in the SUN dataset, each pixel label l i gets a new label l i = M(l i). Pixels with the "ignore" class are given a weight of 0 during evaluation. All the images in the SUN dataset have various dimensions. To prevent artefacts in the input after resizing, only images that have more than 640 · 640 = 409, 600 pixels are used. All images are resized to the same size as Cityscapes (1024 × 2048). The SUN dataset is split into a train set and an evaluation set (25%/75% split). It is stressed that the training set is used only to select the best hyperparameters of the ODIN method. Following previous work on image-level OOD detection, a synthetic dataset of random normally distributed images are used as well. The random normal noise is usually very easily detected by all methods, therefore Perlin noise images are used as well. Perlin noise is a smooth form of noise that has large "blobs" of colour that is much harder to detect and filter. Each of these datasets have the entire image labelled as OOD. All OOD datasets used are mixed with Cityscapes evaluation sets. The ing ratio of ID to OOD pixels for IDD/Cityscapes and SUN/Cityscapes is about 7.5:1 and 3:1 respectively. Since ODIN requires ODD data for hyperperameter tuning, Cityscapes training set is mixed with a held-out training set of OOD data in the same manner as the evaluation sets. There are five metrics used to evaluate the performance of each model. The first four listed below are the same metrics usually used by previous works on OOD detection (; ; ;). Since the output of semantic segmentation is so much larger than image classification, the below metrics must be approximated. This is done by using 400 linearly spaced thresholds between 0 and 1 and tracking all true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) for each threshold. Each pixel contributes to one of TP, TN, FP, FN for a given threshold, based on the ground truth and OOD prediction value -accumulated across all images. • AUROC -Area under the receiver operating characteristic (ROC) curve. The ROC curve shows the false positive rate (FPR) FP FP+TN against the true positive rate (TPR) TP TP+FN. The area under this curve is the AUROC metric. • AUPRC -Area under the precision recall (PR) curve. The PR curve shows the precision TP TP+FP against the TPR (or recall). The area under this curve is the AUPRC. • FPRatTPR -FPR at 95% TPR. Extracted from the ROC curve, this metric is the FPR value when the TPR is 95%. • MaxIoU -Max intersection over union (IoU). IoU is calculated by TP TP+FP+FN. MaxIoU is the maximum IoU over all thresholds. A common metric in the literature, coined detection error , is excluded as it is a linear transformation of the FPRatTPR metric. Therefore, it adds no useful information. The inspiration for MaxIoU issued from the semantic segmentation community. Mean intersection over union (mIoU) is the canonical performance metric for semantic segmentation. MaxIoU is similar, but targets tasks that are threshold dependant. Thresholds selected by MaxIoU punish false positives more than AUROC. The optimal threshold is generally greater, ing in fewer positive predictions than for AUROC. To verify that the MaxIoU is complimentary to AUROC, the optimal thresholds selected by each were experimentally compared. The mean absolute difference between the threshold selected via Youden index and that chosen for MaxIoU was found to be 0.039. PSPNet and DeeplabV3+ network architectures are used with the Resnet and Xception feature extractors respectively. The two driving factors for these networks are: near top performance on the Cityscapes benchmark (2.6 and 1.7 less mIoU than the top scoring algorithm) and the final operation before the softmax classification layer being a bi-linear upsample. The upsample ensures that any method's OOD prediction will be directly correlated with the pixel prediction from the softmax layer, as most methods rely on the softmax layer. There is a clear relationship between any spatial location in the penultimate layer and an output pixel. The Xception feature extractor is much larger than the Resnet -therefore due to hardware limitations, the space intensive Mahalanobis method is only evaluated on PSPNet. There are two main research questions that this paper focuses on. Each question has an associated experiment. • RQ1: Do the required modifications to architecture and loss functions negatively affect the semantic segmentation performance of the network? • RQ2: Which OOD detection method performs the best? To answer RQ1, we evaluate the semantic segmentation performance on Cityscapes using the standard class mean intersection over union (mIoU) metric. As long as the performance drop of modified networks is not too large, the modifications do not interfere with the original task. RQ2 is answered by evaluating each method on the datasets described in Section 3. Figure 3 shows the comparison of the performance of the different methods using the PSPNet architecture. Each graph shows a different metric (c.f . Section 4) for each dataset. Max Softmax, ODIN, and Mahalanobis follow the same trend as their image classification counterparts with the SUN dataset, increasing in performance in that order. However, for the IDD dataset the order is Mahalanobis, Max Softmax, then ODIN, in increasing performance. For image-level OOD detection, the Confidence method outperforms the Max Softmax baseline. However, for pixel-level OOD detection it is worse. For real datasets, VarSum is worse than Mutual Information, but better on the synthetic datasets. Across each metric, VarSum has the biggest performance increase from the modified SUN and IDD datasets to the random normal dataset, moving from worst to near top performance. This however is not the same for the Perlin noise dataset, as the low frequency noise is not easily filtered. The Confidence method seems to mostly learn to highlight class boundaries, as that is where prediction errors are likely to occur. Therefore the prediction loss force lower confidence levels. This makes it less suitable for the random normal and Perlin noise datasets, where the network predicts one single class for the majority of the input. Figure 4 shows the comparison of the performance of the different methods using the DeeplabV3+ architecture. With respect to AUROC, AUPRC, and MaxIoU on the IDD dataset Mutual Information Figure 5: Comparison of the Mutual Information method on an IDD dataset image, successfully predicted. The top row is using the PSPNet architecture, the bottom row is using the DeeplabV3+ architecture. The columns from left to right are: input image, ground truth, class prediction (Cityscapes colours), ODD prediction. The OOD prediction is masked to only cars and auto-rickshaws. Best viewed in colour. has the best performance, followed by ODIN. On all datasets and metrics, VarSum has the worst or is near worst performance. Confidence has much better performance with the DeeplabV3+ architecture than the PSPNet architecture, especially on the random datasets. The for DeeplabV3+ are much less conclusive than PSPNet as the relative ordering across the different metrics changes significantly. The metrics values reported in the original works for the image-level OOD detection for all methods with image-level OOD detection counterparts are very close to 1 (∼0.95 and above). However, in our experiments, the are much less than 1. One peculiar is the significant difference in performance between the Confidence method on DeeplabV3+ and PSPNet on the random normal and Perlin noise datasets. ODIN performs well on all datasets and architectures. The performance of this method is aided by its access to OOD data during hyperparameter tuning, however. Figure 5 shows a comparison of the Mutual Information method using both the PSPNet and DeeplabV3+ architecture. Mutual information is one of the worst performing on PSPNet and the best performing on DeeplabV3+. There are two major observations to note. The first is that both networks label different parts of the auto-rickshaw as different classes. The second is the increase in OOD prediction of all cars from DeeplabV3+ to PSPNet, while the OOD prediction of the autorickshaw remains approximately the same. Both architectures successfully predict the majority of the pixels correctly; however, the separation is better for DeeplabV3+. Figure 6 shows a comparison of the Entropy method using both the PSPNet and DeeplabV3+ architecture. The auto-rickshaw is mistaken by both architectures as a car, therefore the ing OOD prediction is similar to other cars in the scene. This failure highlights a more general failure of pixel-level OOD detection methods. When the network is confident about a wrong prediction, the OOD methods are fooled as well. The drop in performance for pixel-level OOD detection is likely due to features that cause large disruptions at the pixel-level, but would not affect an entire image; for example, shadows, occlusion, and far away objects. Figure 7 shows an example of shadows and far away objects in the bottom row. At the end of the road, most pixels are high OOD values as well as the right side of the scene, which is in the shade of a building. The top row of Figure 7 shows an interesting failure case of a flooded road being predicted as road with a low OOD value. As can be seen in all example outputs, class boundaries are highlighted. A classical computer vision algorithm was developed, using a series of erosion, dilation and other filters to remove these boundaries. In general performance was increased; however, the increase was on the order of 10 −3. To our knowledge, there are very few works researching pixel-level OOD detection. create a dataset by overlaying animals and objects from the COCO dataset on top of the Cityscapes dataset , this however lacks realism. This new dataset is tested with various OOD detection methods. train a segmentation network with two datasets-one ID and one OOD dataset. The network learns to classify between the two on a per pixel-level. This is compared to the Max Softmax baseline. The major flaw in this method is that the network learns its weights from OOD samples. There are some commonalities to Active Learning for semantic segmentation . These studies attempt to use some form of output uncertainty to choose which images are best for training/labelling in order to reduce the number of training examples needed. They produce a heat map similar to the OOD detection output. These heat maps are then aggregated across a whole image to produce a score for the entire image. create an open world dataset. Known object labels are drawn from the COCO dataset , and labels drawn from the NYU dataset are relabelled as unknown if the class doesn't exist in COCO. This methodology is very similar to the modified SUN dataset in Section 3. A generic object instance level segmentation algorithm is developed, based on a class specific object detector, a boundary detector and simulated annealing, and is evaluated on the new dataset. This approach splits the image into visually distinct connected regions, but is too slow for real-time applications. Several methods for detecting OOD pixels were adapted from image-level OOD detection, as well as a pixel uncertainty estimation. These methods were compared using metrics previously established by OOD detection works, as well as a new metric that has roots in the semantic segmentation task. This paper also contributed two new datasets for pixel-level OOD classification derived from semantic segmentation datasets that have common classes but also unique ones. There is great room for improvement for pixel-level OOD detection. One shortcoming for all the methods compared in this paper is the ability to distinguish between class boundary pixels and OOD pixels. We tested classical computer vision techniques that could be used to visually fix this problem, but the performance increase was negligible. The ODIN and Mahalanobis methods have the best performance with PSPNet and SUN dataset, beating the VarSum, Mutual Information, and Confidence methods by a significant margin. However, Mutual Information has the best performance with DeeplabV3+ and the IDD dataset, with the other methods following closely. Therefore the ODIN, Mahalanobis, and Mutual Information methods should be considered the baseline for further research in pixel-level OOD detection. Understanding the faults of pixel-level OOD detectors is crucial for progress. This would include categorising the failure cases of a detector. For example, understanding why a flooded road is not highlighted, and what makes that different to shadows falsely being highlighted. Each subsection uses notation close to the original works, so that readers can refer to said works for more detail. The appendix also provides additional detail. Notation should not be carried between sections unless explicitly stated. Assume the neural network is a function f, and input image is x. The set of pixels P will be used with typical subscripts i, j ∈ P. For example f (x) i is the i th pixel of the of f (x). Note that most methods have a input perturbation step. They are included here for completeness, however, they are not used in any evaluation as it requires access to OOD data. This access to OOD data has been criticised as an unfair comparison to methods that do not require access to OOD data. At test time, dropout can be used as an estimate for model uncertainty. A multiplier of 0 or 1 is randomly chosen for each neuron at test time. use the variation in predictions of a model that uses dropout at test time to compute an estimate of model uncertainty. The output is a variance per class, the sum of these variances is the estimate of model uncertainty. This is performed per pixel to get the value s i for pixel i ∈ P. A.2 show that the max softmax value can be used to detect OOD examples as a baseline for image classification. The softmax function is defined as: Where Sŷ(x) is the max softmax value andŷ is used to signify the chosen index. Given a prediction p i = f (x) i, the max softmax value for each pixel i is v i = Sŷ i (p i), and that value is used to determine if that pixel is OOD. A.3 create a similar method to the max softmax value, dubbed ODIN. This method adds temperature scaling and input preprocessing. The softmax function in Equation 3 is modified to include a temperature value T: Liang et al. found that perturbing the input in the direction of the gradient influences ID samples more than OOD samples, thus separating ID and OOD examples more. The input preprocessing step is: where is a hyperparameter chosen from a set of 21 evenly spaced values starting at 0 and ending at 0.004. The best temperature value is chosen from a predefined set of temperatures {1, 2, 5, 10, 20, 50, 100, 200, 500, 1000}. The temperature-scaled and preprocessed max softmax score v i = Sŷ(x; T) i for pixel i is used to predict if that pixel is OOD. A.4 use the Mahalanobis distance for detecting OOD samples. The Mahalanobis distance is the number of standard deviations a vector is away from the mean, generalised to many dimensions. Each feature vector at a spatial location in the penultimate layer is assumed to be normally distributed. Each distribution is parameterised by the pixel class mean µ ci and global class covariance Σ c for pixel i and class c. The global class covariance is computed for all pixels of a given class, independent of their location. Initial tests showed that using pixel class means and a global class covariance has better performance than global or pixel class mean and pixel class covariance, therefore they are used throughout. The labels are resized to match the height and width of the penultimate layer using nearest neighbour interpolation. The two quantities µ ci and Σ c are computed as follows: where N c is the number of examples of class c, and X j is the j th example of the training dataset X. f l is the output of the l th layer of the network f. Here l is the second to last layer. Each spatial location has a class distance and minimum distance computed by: This increases the number of matrix multiplications required to compute each pixel distance. Due to hardware memory limitations, a dimensionality reduction layer is needed after the penultimate layer, reducing the depth from 512 to 32 via a 1 × 1 convolution. An input preprocessing step is also performed. The new inputx is computed by: Instead of a logistic regression layer, the minimum distance is normalised to have zero mean and unit variance. The sigmoid function σ is applied to clamp to the interval: where µ and s are the mean and standard deviation of all M (x) computed over the whole training dataset. v i is used to determine if pixel i is OOD. The prediction values are resized, with bi-linear interpolation, to the original input size. train a secondary branch of an image classification network to output a single confidence value. This confidence value is learned via a regularization loss. The loss gives "hints" to the network when the confidence is low, and it penalizes low confidence. Similar to the image classification method, a secondary branch is added to the network f. Therefore c i, p i = f (x) i, where c i is the confidence value and p i is the prediction vector for pixel i ∈ P. The new branch is trained by creating a new prediction p i as: where B is a Bernoulli distribution and y i is the one hot ground truth vector for pixel i. The mean negative log likelihood is then applied to the new p i as well as a regularization term is added to force each c i to 1 (i.e. high confidence): where L is the total loss that is used to train the network, and λ is a hyperparameter. λ is 0.5 in all experiments. At test time a preprocessing step is applied to each pixel, and is computed using the gradients of the L c loss. Note that the adapted L c sums over all the pixel confidence predictions c i, therefore the gradient implicitly sums over output pixels as well. Letp i,c i = f i (x), then v i =c i is used to determine if a pixel is OOD. A.6 ENTROPY Shannon entropy is an information theoretic concept, that is used to determine how much information a source contains. The entropy equation is H: R n → R: Since H was developed for probabilities, x must behave like a probability distribution meaning: ∀i, x i >= 0 train an image-level classifier with a third outlier dataset that is disjoint from both the training and OOD dataset. An auxiliary loss is added minimising the entropy of predictions of outlier images. The network learns to predict uniform values for all classes. The max softmax value is then used at test time to determine if a sample is OOD. The entropy function is applied to the softmax output, which satisfies the properties in Equations 21 and 22. Let P = f (x) then v i is used to determine if pixel i is OOD. develop Bayesian uncertainty estimation methods and evaluation metrics to evaluate if higher uncertainty is correlated with higher error rates. The estimation can be used for OOD detection as well, similar to the Entropy and VarSum methods. There are two quantities required. Predictive entropy: and Aleatoric Entropy: In both equations, x is the input and y is the prediction. ω i is the set of weights selected by dropout at iteration i. Mutual information is then MI(y|x) is used to determine if a pixel is OOD or ID."

We propose a continual version of Equilibrium Propagation, where neuron and synapse dynamics occur simultaneously throughout the second phase, with theoretical guarantees and numerical simulations.