Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Convolutional networks are not aware of an object's geometric variations, which leads to inefficient utilization of model and data capacity. To overcome this issue, recent works on deformation modeling seek to spatially reconfigure the data towards a common arrangement such that semantic recognition suffers less from deformation. This is typically done by augmenting static operators with learned free-form sampling grids in the image space, dynamically tuned to the data and task for adapting the receptive field. Yet adapting the receptive field does not quite reach the actual goal -- what really matters to the network is the *effective* receptive field (ERF), which reflects how much each pixel contributes. It is thus natural to design other approaches to adapt the ERF directly during runtime. In this work, we instantiate one possible solution as Deformable Kernels (DKs), a family of novel and generic convolutional operators for handling object deformations by directly adapting the ERF while leaving the receptive field untouched. At the heart of our method is the ability to resample the original kernel space towards recovering the deformation of objects. This approach is justified with theoretical insights that the ERF is strictly determined by data sampling locations and kernel values. We implement DKs as generic drop-in replacements of rigid kernels and conduct a series of empirical studies whose conform with our theories. Over several tasks and standard base models, our approach compares favorably against prior works that adapt during runtime. In addition, further experiments suggest a working mechanism orthogonal and complementary to previous works. The rich diversity of object appearance in images arises from variations in object semantics and deformation. Semantics describe the high-level abstraction of what we perceive, and deformation defines the geometric transformation tied to specific data . Humans are remarkably adept at making abstractions of the world ; we see in raw visual signals, abstract semantics away from deformation, and form concepts. Interestingly, modern convolutional networks follow an analogous process by making abstractions through local connectivity and weight sharing . However, such a mechanism is an inefficient one, as the emergent representations encode semantics and deformation together, instead of as disjoint notions. Though a convolution responds accordingly to each input, how it responds is primarily programmed by its rigid kernels, as in Figure 1(a, b). In effect, this consumes large model capacity and data modes. We argue that the awareness of deformations emerges from adaptivity -the ability to adapt at runtime (; ;). Modeling of geometric transformations has been a constant pursuit for vision researchers over decades (; ; ;). A basic idea is to spatially recompose data towards a common mode such that semantic recognition suffers less from deformation. A recent work that is representative of this direction is Deformable Convolution . As shown in Figure 1 (c), it augments the convolutions with free-form sampling grids in the data space. It is previously justified as adapting receptive field, or what we phrase as the "theoretical receptive field", that defines which input pixels can contribute to the final output. However, theoretical receptive field does not measure how much impact an input pixel actually has. On the other hand, reconfigure data towards common arrangement to counter the effects of geometric deformation. (d) Our Deformable Kernels (DKs) instead resample kernels and, in effect, adapt kernel spaces while leaving the data untouched. Note that (b) and (c) share kernel values but sample different data locations, while (b) and (d) share data locations but sample different kernel values. propose to measure the effective receptive field (ERF), i.e. the partial derivative of the output with respect to the input data, to quantify the exact contribution of each raw pixel to the convolution. Since adapting the theoretical receptive field is not the goal but a means to adapt the ERF, why not directly tune the ERF to specific data and tasks at runtime? Toward this end, we introduce Deformable Kernels (DKs), a family of novel and generic convolutional operators for deformation modeling. We aim to augment rigid kernels with the expressiveness to directly interact with the ERF of the computation during inference. Illustrated in Figure 1 (d), DKs learn free-form offsets on kernel coordinates to deform the original kernel space towards specific data modality, rather than recomposing data. This can directly adapt ERF while leaving receptive field untouched. The design of DKs that is agnostic to data coordinates naturally leads to two variants -the global DK and the local DK, which behave differently in practice as we later investigate. We justify our approach with theoretical which show that ERF is strictly determined by data sampling locations and kernel values. Used as a generic drop-in replacement of rigid kernels, DKs achieve empirical coherent with our developed theory. Concretely, we evaluate our operator with standard base models on image classification and object detection. DKs perform favorably against prior works that adapt during runtime. With both quantitative and qualitative analysis, we further show that DKs can work orthogonally and complementarily with previous techniques. We distinguish our work within the context of deformation modeling as our goal, and dynamic inference as our means. Deformation Modeling: We refer to deformation modeling as learning geometric transformations in 2D image space without regard to 3D. One angle to attack deformation modeling is to craft certain geometric invariances into networks. However, this usually requires designs specific to certain kinds of deformation, such as shift, rotation, reflection and scaling (; ; ; ; ;). Another line of work on this topic learns to recompose data by either semi-parameterized or completely free-form sampling in image space: Spatial Transformers learns 2D affine transformations, Deep Geometric Matchers learns thin-plate spline transformations, Deformable Convolutions learns free-form transformations. We interpret sampling data space as an effective approach to adapt effective receptive fields (ERF) by directly changing receptive field. At a high-level, our Deformable Kernels (DKs) share intuitions with this line of works for learning geometric transformations, yet are instantiated by learning to sample in kernel space which directly adapt ERF while leaving theoretical receptive fields untouched. While kernel space sampling is also studied in Deformable Filter and KPConv , but in their contexts, sampling grids are computed from input point clouds rather than learned from data corpora. Dynamic Inference: Dynamic inference adapts the model or individual operators to the observed data. The computation of our approach differs from self-attention in which linear or convolution modules are augmented with subsequent queries that extract from the same input. We consider our closest related works in terms of implementation as those approaches that adapt convolutional kernels at run time. It includes but is not limited to Dynamic Filters , Selective Kernels and Conditional Convolutions. All of these approaches can learn and infer customized kernel spaces with respect to the data, but are either less inefficient or are loosely formulated. Dynamic Filters generate new filters from scratch, while Conditional Convolutions extend this idea to linear combinations of a set of synthesized filters. Selective Kernels are, on the other hand, comparably lightweight, but aggregating activations from kernels of different size is not as compact as directly sampling the original kernel space. Another line of works contemporary to ours is to compose free-form filters with structured Gaussian filters, which essentially transforms kernel spaces by data. Our DKs also differ from these works with the emphasize of direct adaptation the ERF rather than the theoretical receptive field. As mentioned previously, the true goal should be to adapt the ERF, and to our knowledge, our work is the first to study dynamic inference of ERFs. We start by covering preliminaries on convolutions, including the definition of effective receptive field (ERF). We then formulate a theoretical framework for analyzing ERFs, from which we gain insights to motivate our Deformable Kernels (DKs). We then elaborate two different instantiations of DKs, namely the global and local DK. Finally, we distinguish DKs from Deformable Convolutions and present a unified approach together with them. Our analysis suggests compatibility between DKs and the prior work. 2D Convolution: Let us first consider an input image I ∈ R D×D. By convolving it with a kernel W ∈ R K×K of stride 1, we have an output image O whose pixel values at each coordinate j ∈ R 2 can be expressed as by enumerating discrete kernel positions k within the support This defines a rigid grid for sampling data and kernels. Theoretical Receptive Field: The same kernel W can be stacked repeatedly to form a linear convolutional network with n layers. The theoretical receptive field can then be imagined as the "accumulative coverage" of kernels at each given output unit on the input image by deconvolving back through the network. This property characterizes a set of input fields that could fire percepts onto corresponding output pixels. The size of a theoretical receptive field scales linearly with respect to the network depth n and kernel size K . Effective Receptive Field: Intuitively, not all pixels within a theoretical receptive field contribute equally. The influence of different fields varies from region to region thanks to the central emphasis of stacked convolutions and also to the non-linearity induced by activations. The notion of effective receptive field (ERF) is thus introduced to measure the impact of each input pixel on the output at given locations. It is defined as a partial derivative field of the output with respect to the input data. With the numerical approximations in linear convolution networks, the ERF was previously identified as a Gaussian-like soft attention map over input images whose size grows fractionally with respect to the network depth n and linearly to the kernel size K. Empirical validate this idea under more complex and realistic cases when networks exploit non-linearities, striding, padding, skip connections, and subsampling. We aim to revisit and complement the previous analysis on ERFs by. While the previous analysis concentrates on studying the expectation of an ERF, i.e., when network depth n approaches infinity or all kernels are randomly distributed without learning in general, our analysis focuses on how we can perturb the computation such that the change in ERF is predictable, given an input and a set of kernel spaces. We start our analysis by considering a linear convolutional network, without any unit activations, as defined in Section 3.1. For consistency, superscripts are introduced to image I, kernel W, and subscripts to kernel positions k to denote the index s ∈ [1, n] of each layer. Formally, given an input image I and a set of K × K kernels {W (s) } n s=1 of stride 1, we can roll out the final output O ≡ I (n) by unfolding Equation 1 as By definition 1, the effective receptive field value of output coordinate j that takes input coordinate i can be computed by where 1[·] denotes the indicator function. This indicates that ERF is related only to the data sampling location j, kernel sampling location k, and kernel matrices {W (s) }. If we replace the m th kernel W (m) with a 1 × 1 kernel of a single parameter W (m) km sampled from it, the value of ERF becomes to where S = [1, n] \ {m}. Since a K × K kernel can be deemed as a composition of K 2 1 × 1 kernels distributed on a square grid, Equation 3 can thus be reformulated as For the case of complex non-linearities, where we here consider post ReLU activations in Equation 1, We can follow a similar analysis and derive corresponding ERF as 1 The original definition of ERF in focuses on the central coordinate of the output, i.e. j =, to partially avoid the effects of zero padding. In this work, we will keep j in favor of generality while explicitly assuming input size D → ∞. Here we can see that the ERF becomes data-dependent due to the coefficient C, which is tied to input coordinates, kernel sampling locations, and input data I. The more detailed analysis of this coefficient is beyond the scope of this paper. However, it should be noted that this coefficient only "gates" the contribution of the input pixels to the output. So in practice, ERF is "porous" -there are inactive (or gated) pixel units irregularly distributed around the ones that fire. This phenomenon also appeared in previous studies (such as in , Figure 1). The maximal size of an ERF is still controlled by the data sampling location and kernel values as in the linear cases in Equation 5. is that all computations are linear, making it compatible with any linear sampling operators for querying kernel values of fractional coordinates. In other words, sampling kernels in effect samples the ERF on the data in the linear case, but also roughly generalizes to non-linear cases as well. This finding motivates our design of Deformable Kernels (DKs) in Section 3.3. In the context of Equation 1, we resample the kernel W with a group of learned kernel offsets denoted as {∆k} that correspond to each discrete kernel position k. This defines our DK as and the value of ERF as Note that this operation leads to sub-pixel sampling in the kernel space. In practice, we use bilinear sampling to interpolate within the discrete kernel grid. Intuitively, the size (resolution) of the original kernel space can affect sampling performance. Concretely, suppose we want to sample a 3 × 3 kernel. DKs do not have any constraint on the size of the original kernel space, which we call the "scope size" of DKs. That said, we can use a W of any size K even though the number of sampling locations is fixed as K 2. We can thus exploit large kernels -the largest ones can reach 9×9 in our experiments with nearly no overhead in computation since bilinear interpolations are extremely lightweight compared to the cost of convolutions. This can also increase the number of learning parameters, which in practice might become intractable if not handled properly. In our implementation, we will exploit depthwise convolutions such that increasing scope size induces a negligible amount of extra parameters. As previously discussed, sampling the kernel space in effect transforms into sampling the ERF. On the design of locality and spatial granularity of our learned offsets, DK naturally delivers two variants -the global DK and the local DKs, as illustrated in Figure 2. In both operators, we learn a kernel offset generator G that maps an input patch into a set of kernel offsets that are later applied to rigid kernels. In practice, we implement G global as a stack of one global average pooling layer, which reduces feature maps into a vector, and another fully-connected layer without non-linearities, which projects the reduced vector into an offset vector of 2K 2 dimensions. Then, we apply these offsets to all convolutions for the input image following Equation 7. For local DKs, we implement G local as an extra convolution that has the same configuration as the target kernel, except that it only has 2K 2 output channels. This produces kernel sampling offsets {∆k} that are additionally indexed by output locations j. It should be noted that similar designs were also discussed in , in which filters are generated given either an image or individual patches from scratch rather than by resampling. Intuitively, we expect the global DK to adapt kernel space between different images but not within a single input. The local DK can further adpat to specific image patches: for smaller objects, it is better to have shaper kernels and thus denser ERF; for larger objects, flatter kernels can be more beneficial for accumulating a wider ERF. On a high level, local DKs can preserve better locality and have larger freedom to adapt kernel spaces comparing to its global counterpart. We later compare these operators in our experiments. The core idea of DKs is to learn adaptive offsets to sample the kernel space for modeling deformation, which makes them similar to Deformable Convolutions , at both the conceptual and implementation levels. Here, we distinguish DKs from Deformable Convolutions and show how they can be unified. where they aim to learn a group of data offsets {∆j} with respect to discrete data positions j. For consistency for analysis, the value of effective receptive field becomes This approach essentially recomposes the input image towards common modes such that semantic recognition suffers less from deformation. Moreover, according to our previous analysis in Equation 5, sampling data is another way of sampling the ERF. This, to a certain extent, also explains why Deformable Convolutions are well suited for learning deformation-agnostic representations. Moreover, we can learn both data and kernel offsets in one convolutional operator. Conceptually, this can be done by merging Equation 7 with Equation 9, which leads to We also investigate this operator in our experiments. Although the two techniques may be viewed as serving a similar purpose, we find the collaboration between Deformable Kernels and Deformable Convolutions to be powerful in practice, suggesting strong compatibility. We evaluate our Deformable Kernels (DKs) on image classification using ILSVRC and object detection using the COCO benchmark. Necessary details are provided to reproduce our , together with descriptions on base models and strong baselines for all experiments and ablations. For taskspecific considerations, we refer to each corresponding section. Implementation Details: We implement our operators in PyTorch and CUDA. We exploit depthwise convolutions when designing our operator for better computational efficiency 2. We initialize kernel grids to be uniformly distributed within the scope size. For the kernel offset generator, we set its learning rate to be a fraction of that of the main network, which we cross-validate for each base model. We also find it important to clip sampling locations inside the original kernel space, such that k + ∆k ∈ K in Equation 7. Base Models: We choose our base models to be ResNet-50 and MobileNet-V2 , following the standard practice for most vision applications. As mentioned, we exploit depthwise convolution and thus make changes to the ResNet model. Concretely, we define our ResNet-50-DW base model by replacing all 3 × 3 convolutions by its depthwise counterpart while doubling the dimension of intermediate channels in all residual blocks. We find it to be a reasonable base model compared to the original ResNet-50, with comparable performance on both tasks. During training, we set the weight decay to be 4 × 10 −5 rather than the common 10 −4 for both models since depthwise models usually underfit rather than overfit (; ;). We set the learning rate multiplier of DK operators as 10 −2 for ResNet-50-DW and 10 −1 for MobileNet-V2 in all of our experiments. Strong Baselines: We develop our comparison with two previous works: Conditional Convolutions for dynamics inference, and Deformable Convolutions for deformation modeling. We choose Conditional Convolutions due to similar computation forms -sampling can be deemed as an elementewise "expert voting" mechanism. For fair comparisons, We reimplement and reproduce their . We also combine our operator with these previous approach to show both quantitative evidence and qualitative insight that our working mechanisms are compatible. We first train our networks on the ImageNet 2012 training set . Similar to; , training is performed by SGD for 90 epochs with momentum 0.9 and batch size 256. We set our learning rate of 10 −1 so that it linearly warms up from zero within first 5 epochs. A cosine training schedule is applied over the training epochs. We use scale and aspect ratio augmentation with color perturbation as standard data augmentations. We evaluate the performance of trained models on the ImageNet 2012 validation set. The images are resized so that the shorter side is of 256 pixels. We then centrally crop 224 × 224 windows from the images as input to measure recognition accuracy. We first ablate the scope size of kernels for our DKs and study how it can affect model performance using ResNet-50-DW. As shown in Table 1, our DKs are sensitive to the choice of the scope size. We shown that when only applied to 3 × 3 convolutions inside residual bottlenecks, local DKs induce a +0.7 performance gain within the original scope. By further enlarging the scope size, performance increases yet quickly plateaus at scope 4 × 4, yielding largest +1.4 gain for top-1 accuracy. Our speculation is that, although increasing scope size theoretically means better interpolation, it also makes the optimization space exponentially larger for each convolutional layer. And since number of entries for updating is fixed, this also leads to relatively sparse gradient flows. In principle, we set default scope size at 4 × 4 for our DKs. We next move on and ablate our designs by comparing the global DK with the local DK, shown in the table. Both operators helps while the local variants consistently performs better than their global counterparts, bringing a +0.5 gap on both base models. We also study the effect of using more DKs in the models -the 1 × 1 convolutions are replaced by global DKs 3 with scope 2 × 2. Note that all 1 × 1 convolutions are not depthwise, and therefore this operation induces nearly 4 times of parameters. We refer their only for ablation and show that adding more DKs still helps - especially for MobileNet-V2 since it is under-parameterized. This finding also holds for previous models as well. We further compare and combine DKs with Conditional Convolutions and Deformable Convolutions. Results are recorded in Table 2. We can see that DKs perform comparably on ResNet-V2 and compare favorably on MobileNet-V2 -improve +0.9 from Deformable Convolutions and achieve comparable with less than a quarter number of parameters compared to Conditional Convolutions. Remarkably, we also show that if combined together, even larger performance gains are in reach. We see consistent boost in top-1 accuracy compared to strong baselines: +1.3/+1.0 on ResNet-50-DW, and +1.2/+1.2 on MobileNet-V2. These gaps are bigger than those from our own ablation, suggesting the working mechanisms across the operators to be orthogonal and compatible. We examine DKs on the COCO benchmark . For all experiments, we use Faster R-CNN with FPN as the base detector, plugging in the backbones we previously trained on ImageNet. For MobileNet-V2, we last feature maps of the each resolution for FPN post aggregation. Following the standard protocol, training and evaluation are performed on the 120k images in the train-val split and the 20k images in the test-dev split, respectively. For evaluation, we measure the standard mean average precision (mAP) and shattered scores for small, medium and large objects. Table 4: Comparisons to strong baselines for object detection DKs perform fall short to Deformable Convolution, but combination still improves performance. Table 3 and Table 4 follow the same style of analysis as in image classification. While the baseline methods of ResNet achieve 36.6 mAP, indicating a strong baseline detector, applying local DKs brings a +1.2 mAP improvement when replacing 3x3 rigid kernels alone and a +1.8 mAP improvement when replacing both 1x1 and 3x3 rigid kernels. This trend magnifies on MobileNet-v2 models, where we see an improvement of +1.6 mAP and +2.4 mAP, respectively. Results also confirm the effectiveness of local DKs against global DKs, which is again in line with our expectation that local DKs can model locality better. For the comparisons with strong baselines, an interesting phenomenon worth noting is that though DKs perform better than Deformable Convolutions on image classification, they fall noticeably short for object detection measured by mAP. We speculate that even though both techniques can adapt ERF in theory (as justified in Section 3.2), directly shifting sampling locations on data is easier to optimize. Yet after combining DKs with previous approaches, we can consistently boost performance for all the methods -+0.7/+1.2 for Deformable Convolutions on each base models, and +1.7/+1.1 for Conditional Convolutions. These findings align with the from image classification. We next investigate what DKs learn and why they are compatible with previous methods in general. Awareness of Object Scale: Since deformation is hard to quantify, we use object scale as a rough proxy to understand what DKs learn. In Figure 3, we show the t-SNE of learned model dynamics by the last convolutional layers in MobileNet-V2 using Conditional Convolution and our DKs. We validate the finding as claimed by that the experts of Conditional Convolutions have better correlation with object semantics than their scales (in reference to Figure 6 from their paper). Instead, our DKs learn kernel sampling offsets that strongly correlate to scales rather than semantics. This sheds light on why the two operators are complementary in our previous experiments. deformations. We compare the of rigid kernels, Deformable Convolutions, our DKs, and the combination of the two operators. For all examples, note that the theoretical receptive field covers every pixel in the image but ERFs contain only a central portion of it. Deformable Convolutions and DKs perform similarly in terms of adapting ERFs, but Deformable Convolutions tend to spread out and have sparse responses while DKs tend to concentrate and densely activate within an object region. Combining both operators yields more consistent ERFs that exploit both of their merits. In this paper, we introduced Deformable Kernels (DKs) to adapt effective receptive fields (ERFs) of convolutional networks for object deformation. We proposed to sample kernel values from the original kernel space. This in effect samples the ERF in linear networks and also roughly generalizes to non-linear cases. We instantiated two variants of DKs and validate our designs, showing connections to previous works. Consistent improvements over them and compatibility with them were found, as illustrated in visualizations. image patch kernel patch Figure 5: Illustration of feed-forwarding through a 3×3 local Deformable Kernel from a 4×4 scope. For each input patch, local DK first generates a group of kernel offsets {∆k} from input feature patch using the light-weight generator G (a 3×3 convolution of rigid kernel). Given the original kernel weights W and the offset group {∆k}, DK samples a new set of kernel W using a bilinear sampler B. Finally, DK convolves the input feature map and the sampled kernels to complete the whole computation. We now cover more details on implementing DKs by elaborating the computation flow of their forward and backward passes. We will focus on the local DK given its superior performance in practice. The extension to global DK implementation is straight-forward. In Section 3.3, we introduce a kernel offset generator G and a bilinear sampler B. Figure 5 illustrates an example of the forward pass. Concretely, given a kernel W and a learned group of kernel offsets {∆k} on top of a regular 2D grid {k}, we can resample a new kernel W by a bilinear operator B as where B(k + ∆k, k) = max(0, 1 − |k x + ∆k x − k x |) · max(0, 1 − |k y + ∆k y − k y |). Given this resampled kernel, DK convolves it with the input image just as in normal convolutions using rigid kernels, characterized by Equation 1. The backward pass of local DK consists of three types of gradients: the gradient to the data of the previous layer, the gradient to the full scope kernel of the current layer and the additional gradient to the kernel offset generator of the current layer. The first two types of gradients share same forms of the computation comparing to the normal convolutions. We now cover the computation for the third flow of gradient that directs where to sample kernel values. In the context of Equation 7, the partial derivative of a output item O j w.r.t. x component of a given kernel offset ∆k x (similar for its y component ∆k y) can be computed as where ∂B(k + ∆k, k) ∂∆k x = max(0, 1 − |k y + ∆k y − k y |) ·    0 |k x + ∆k x − k x | ≥ 1 1 k x + ∆k x < k x −1 k x + ∆k x ≥ k x. Table 5: Network architecture of our ResNet-50-DW comparing to the original ResNet-50 Inside the brackets are the general shape of a residual block, including filter sizes and feature dimensionalities. The number of stacked blocks on each stage is presented outside the brackets. "G = 128" suggests the depthwise convolution with 128 input channels. Two models have similar numbers of parameters and FLOPs. At the same time, depthwise convolutions facilitate the computation efficiency of our Deformable Kernels. B NETWORK ARCHITECTURES Table 5 shows the comparison between the original ResNet-50 and our modified ResNet-50-DW. The motivation of introducing depthwise convolutions to ResNet is to accelerate the computation of local DKs based on our current implementations. The ResNet-50-DW model has similar model capacity/complexity and performance (see Table 1) compared to its non-depthwise counterpart, making it an ideal base architecture for our experiments. On the other hand, in all of our experiments, MobileNet-V2 base model is left untouched. We here show additional comparison of ERFs when objects have different kinds of deformations in Figure 6. Comparing to baseline, our method can adapt ERFs to be more persistent to object's semantic rather than its geometric configuration.

Don't deform your convolutions -- deform your kernels.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question:
represent the text to find a very short summary (1-2 sentences max) Examples:

Given Stochastic gradient descent (SGD) is the workhorse of modern machine learning. Sometimes, there are many different potential gradient estimators that can be used. When so, choosing the one with the best tradeoff between cost and variance is important. This paper analyzes the convergence rates of SGD as a function of time, rather than iterations. This in a simple rule to select the estimator that leads to the best optimization convergence guarantee. This choice is the same for different variants of SGD, and with different assumptions about the objective (e.g. convexity or smoothness). Inspired by this principle, we propose a technique to automatically select an estimator when a finite pool of estimators is given. Then, we extend to infinite pools of estimators, where each one is indexed by control variate weights. This is enabled by a reduction to a mixed-integer quadratic program. Empirically, automatically choosing an estimator performs comparably to the best estimator chosen with hindsight. In stochastic gradient variational inference (SGVI) there are multiple gradient estimators with varying costs and variances. Estimators may be obtained using the reparameterization trick (; ; Titsias and Lázaro-), the score function method , or other techniques (Titsias and Lázaro- ; ;). Also, many control variates can be added to an estimator to reduce variance . The cost and variance of an estimator significantly affects optimization convergence speed . The use of different estimators leads to different optimization performances, and the estimator with optimal cost-variance tradeoff is often situationdependent (for an example see Fig. 1). In settings where multiple estimators with varying costs and variances are available, selecting the optimal one is important. Rather than rely on the user to manually select one, we propose that estimator selection could be done adaptively. This paper investigates how, given a pool of gradient estimators, automatically choose one to get the best convergence guarantee for stochastic optimization. We study cost-variance tradeoffs by analyzing the convergence rates of several variants of SGD. We express convergence rates in terms of time rather than iterations. This leads to what we call the "G 2 T principle": A simple rule that predicts, given a pool of gradient estimators, which one in the best convergence guarantees for optimization. We use the principle to propose two gradient estimator selection algorithms: One for the case in which a finite pool of estimators is available, and other when the pool contains an infinite number of estimators, each indexed by control variate weights (i.e. control variate selection). Notation: We use g(w, ξ), where ξ is a random variable, to denote an unbiased estimator of target's gradient, G 2 (g) to denote a bound on g's expected squared norm, and T (g) to denote the computational cost of computing estimator g(w, ξ), measured in seconds. 2 T Principle and Gradient Estimator Selection Given a set of gradient estimators with varying costs and variances, our goal is to find the one that gives the best convergence guarantee for optimization algorithms. Convergence guarantees for several variants of SGD are shown in Table 1. Given a pool of estimators with known G 2 and T, the one with minimum G 2 T should be used. In practice, however, G 2 and T are typically not known. We propose to use estimates. Assuming that the cost of an estimator g(w, ξ) is independent of w, an estimateT (g) of T (g) can be obtained for each g ∈ G through a single initial profiling phase. Dealing with G 2 (g) is more involved. Convergence guarantees assume that E ||g(w, ξ)|| 2 ≤ G 2 (g) for all w. Often (e.g. when w is unbounded) this is not true for any finite G. We propose an approach that is justified under two assumptions: (i) Optimization starting from a point w 0 will only visit a restricted part of parameter space. It is sufficient to bound E ||g(w, ξ)|| 2 for the set of w that may actually be encountered. (ii) E g(w, ξ) 2 tends to decrease slowly over time. When these are true, it makes sense to also form an estimatê G 2 (g) through an initial profiling phase, and to update these estimates a small number of times as optimization proceeds. The approach is summarized in Alg. 1. It is possible to use multiple control variates to reduce a gradient estimator's variance. However, some control variates might be computationally expensive but only in a small reduction in variance. It may be better to remove them and accept a noisier but cheaper estimator. How can we select what control variates to use? When an unbiased gradient estimator g base and control variates c 1,...c J are given, a gradient estimator can be expressed as The available estimators are G = {g a : a ∈ R J}. The number of estimators g a ∈ G is infinite, and Alg. 1 cannot be used (cannot measureT andĜ 2 for each estimator g a individually). We show that, despite having an infinite number of estimators, when estimators are indexed by control variate weights finding the one with minimumĜ 2T can be done efficiently. This is because two properties hold: (i) EstimatesT (g a) andĜ 2 (g a) can be efficiently obtained for all estimators g a ∈ G through the use of shared statistics (a finite number of evaluations of the base estimator and control variates); and (ii) The ing (combinatorial) optimization problem a * = arg min aĜ 2 (g a)T (g a) can be reduced to a Mixed Integer Quadratically Constrained Program (MIQCP), which can be solved quickly in practice. For all g ∈ G measure timeT (g). for k = 1, 2, · · · do if time to re-select estimator then for each estimator g dô Algorithm 2 SGD with automatically selected control variates. Require: Set of estimators, G. Require: Times to re-select estimator. Require: Number of MC samples M. For g base measure time t 0. This section presents an overview of the experiments. Full details are in the appendix. We tackle inference problems using SGVI. We consider three models: Logistic regression, a hierarchical regression model, and a Bayesian neural network. For the simple logistic regression model we use a Gaussian with a full rank covariance as variational distribution q w (z). For the other more complex models we use a factorized Gaussian. We use SGD with momentum to optimize, and five samples z ∼ q w (z) to form Monte Carlo gradient estimates. For both Algs. 1 and 2 we update the estimator used (by minimizingĜ 2T) three times during training. We first present an empirical validation for Alg. 1. We compare the achieved by using three different gradient estimators: (Rep) the plain reparameterization estimator, (Miller) the estimator proposed by Miller et al. , and (STL) the "sticking the landing" estimator . We also run Alg. 1 with the set of estimators G = {Rep, Miller, STL}, which uses the estimator g ∈ G with minimumĜ 2T. We now present an empirical validation for Alg. 2 (control variate selection). We consider the same three models as above. The set of candidate estimators is G Auto = {g a : a ∈ R 3}, where g a is as defined in eq.. The base estimator is plain reparameterization, and there are three candidate control variates (c 1, c 2, c 3). The goal is to check if Alg. 2 successfully navigates cost/variance tradeoffs. We thus compare against using each possible fixed subsets of control variates S ⊆ {c 1, c 2, c 3}, with the weights that minimize the estimator's variance (which can be estimated efficiently ( Log. Regression (a1a) We compare against using different fixed subsets of control variates with the weights that minimize the estimator's variance. Lines are identified as follows: "Auto" stands for using Alg. 2 to select what control variates to use and their weights, "Base" stands for optimizing using the base gradient alone, "1" stands for using the fixed set of control variates {c 1} with the minimum variance weights, "12" stands for using the fixed set of control variates {c 1, c 2}, and so on. Appendix A. Appendix Three different models were considered: a Bayesian neural network, a hierarchical Poisson model, and Bayesian logistic regression. Bayesian logistic regression: We use the dataset a1a. The training set is given by, where y i is binary. The model is specified by Hierarchical Poisson model:. The model measures the relative stop-and-frisk events in different precincts in New York city, for different ethnicities. The model is specified by In this case, e stands for ethnicity and p for precinct, Y ep for the number of stops in precinct p within ethnicity group e (observed), and N ep for the total number of arrests in precinct p within ethnicity group e (observed). BNN: As done by Miller et al. we use a subset of 100 rows from the "Red-wine" dataset (regression). We implement a neural network with one hidden layer with 50 units and Relu activations. Let D = {x i, y i} N i=1 be the training set. The model is specified by log α ∼ N (0, 10 2), log τ ∼ N (0, 10 2), W ∼ N (0, α 2 I), (weights and biases) A.2. Details on the simulations Control variates used: c 1: Difference between the entropy term computed exactly and estimated using reparameterization: c(w, ξ) = ∇ w log q w (T w (ξ)) − ∇ w Eq w log q w (Z). c 2: Control variate by Miller et al. based on a second order Taylor expansion of log p(x, z). c 3: Difference between the prior term computed exactly and estimated using reparameterization: c(w, ξ) = ∇ w log p(T w (ξ)) − ∇ w Eq w log p(Z). Algorithmic details: For Alg. 2 we use M = 400 to estimateĜ 2 (except for Logistic regression, where we use M = 200). We re-select the optimal estimator three times during training, initially, after 10% of training is done, and after 50% of training is done. Optimization details: We use SGD with momentum (β = 0.9) with 5 samples z ∼ q w (z) to form the Monte Carlo gradient estimates. For all models we find an initial set of parameters by optimizing with the base gradient for 300 steps and a fixed learning rate of 10 −5. This initialization was helpful in practice because w tends to change rapidly at the beginning of optimization. After this brief initialization, E ||g(w, ξ)|| 2 tends to change much more slowly, meaning our technique is more helpful. The performance of all algorithms depends on the step-size. To give a fair comparison, Figs. 1 and 2 summarize by showing the with the best step-size for each estimator. (12 stepsizes between 10 −6 and 10 −3 were considered.) Estimators with control variates can be expressed as An expression forT (g a) can be obtained by noticing that computing g a only requires computing the base gradient and the control variates with non-zero weights. Then, for all 2. Using Tw(ξ) = µ + D 1/2 ξ, where ξ ∼ N (0, I), µ is the mean of qw and D 1/2 is the Cholesky factorization of the covariance of qw. Thus, we can computeT (g a) for all g a ∈ G only by profiling the base gradient and each control variate individually. Similarly,Ĝ 2 (g a, w) is determined by the same set of base gradient and control variate evaluations, regardless of the value of a. Suppose that, at iteration k, we sample ξ k1,..., ξ kM. Then, for all g a ∈ G, Thus, we can computeĜ 2 (g a, w k) for all g a ∈ G using only M evaluations of the base gradient g base and each control variate c i. Equations and characterize the (estimated) cost and variance of the gradient estimator with weights a. We find the weights that in the optimal cost-variance tradeoff by solving a * (w) = arg min whereT (g a) andĜ 2 (g a, w) are as in equations and. The solution a * (w) indicates what control variates to use (those with a * i = 0), and their weights. Solving the (combinatorial) minimization problem in equation may be challenging. However, theorem 1 states that it can be reduced to a MIQCP, which can be solved fast using solvers such as. Theorem 1 When different gradient estimators are indexed by a set of J control variate weights, the problem of finding a * (w) as in equation can be reduced to solving a mixed integer quadratically constrained program with 2J + 2 variables, one quadratic constraint, and one linear constraint. A mixed integer quadratic program is an optimization problem in which the objective function and constraints are quadratic (or linear), and some (or all) variables are restricted to be integers: Ax + b = 0, where x ∈ R n, Q 0,..., Q m ∈ R n×n, and some components of x are restricted to be integers. We now prove the theorem 1. Proof GivenT The final minimization problem shown in equation 14 has the form of a general MIQCP, shown in equation 6, with the exception of the last constraint b i = 1[a i = 0]. Despite not being in the original definition of a MIQCP, several solver accept constraints of this type (, the solver used in our simulation, does). it matches with We propose a gradient estimator selection algorithm with the aim on improving optimization efficiency. but not with This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.

Question: Inferring the structural properties of a protein from its amino acid sequence is a challenging yet important problem in biology. Structures are not known for the vast majority of protein sequences, but structure is critical for understanding function. Existing approaches for detecting structural similarity between proteins from sequence are unable to recognize and exploit structural patterns when sequences have diverged too far, limiting our ability to transfer knowledge between structurally related proteins. We newly approach this problem through the lens of representation learning. We introduce a framework that maps any protein sequence to a sequence of vector embeddings --- one per amino acid position --- that encode structural information. We train bidirectional long short-term memory (LSTM) models on protein sequences with a two-part feedback mechanism that incorporates information from (i) global structural similarity between proteins and (ii) pairwise residue contact maps for individual proteins. To enable learning from structural similarity information, we define a novel similarity measure between arbitrary-length sequences of vector embeddings based on a soft symmetric alignment (SSA) between them. Our method is able to learn useful position-specific embeddings despite lacking direct observations of position-level correspondence between sequences. We show empirically that our multi-task framework outperforms other sequence-based methods and even a top-performing structure-based alignment method when predicting structural similarity, our goal. Finally, we demonstrate that our learned embeddings can be transferred to other protein sequence problems, improving the state-of-the-art in transmembrane domain prediction. Proteins are linear chains of amino acid residues that fold into specific 3D conformations as a of the physical properties of the amino acid sequence. These structures, in turn, determine the wide array of protein functions, from binding specificity to catalytic activity to localization within the cell. Information about structure is vital for studying the mechanisms of these molecular machines in health and disease, and for development of new therapeutics. However, experimental structure determination is costly and atomic structures have only been determined for a tiny fraction of known proteins. Methods for finding proteins with related structure directly from sequence are of considerable interest, but the problem is challenging, because sequence similarity and structural similarity are only loosely related BID0 BID1 BID2 BID3, e.g. similar structural folds can be formed by diverse sequences. As a , our ability to transfer knowledge between proteins with similar structures is limited. In this work, we address this problem by learning protein sequence embeddings using weak supervision from global structural similarity for the first time. Specifically, we aim to learn a bidirectional LSTM (biLSTM) embedding model, mapping sequences of amino acids to sequences of vector representations, such that residues occurring in similar structural contexts will be close in embedding space. This is difficult, because we have not observed position-level correspondences between sequences, only global sequence similarity. We solve this by defining a whole sequence similarity measure from sequences of vector embeddings. The measure decomposes into an alignment of the sequences and pairwise comparison of the aligned positions in embedding space. For the alignment, we propose a soft symmetric alignment (SSA) mechanism -a symmetrization of the directional alignment commonly used in attention mechanisms. Furthermore, in order to take advantage of information about local structural context within proteins, we extend this framework to include position-level supervision from contacts between residues in the individual protein structures. This multitask framework FIG0 ) allows us to newly leverage both global structural similarity between proteins and residue-residue contacts within proteins for training embedding models. The similarity prediction module takes pairs of proteins represented by their sequences of vector embeddings and predicts their shared SCOP level. Sequences are first aligned based on L1 distance between their vector embeddings using SSA. From the alignment, a similarity score is calculated and related to shared SCOP levels by ordinal regression. The contact prediction module uses the sequence of vector embeddings to predict contacts between amino acid positions within each protein. The contact loss is calculated by comparing these predictions with contacts observed in the 3D structure of the protein. Error signal from both tasks is used to fit the parameters of the encoder. We first benchmark our model's ability to correctly predict structural similarity between pairs of sequences using the SCOPe ASTRAL dataset BID4. This dataset contains protein domains manually classified into a hierarchy of structural categories (Appendix Figure 3). We show that our model dramatically outperforms other sequence-based protein comparison methods when predicting comembership in the SCOP hierarchy. Remarkably, our model even outperforms TMalign BID5, which requires structures as input and therefore structures must be known a priori. In contrast, our model uses only sequence as input. Next, we perform an ablation study to evaluate the importance of our modeling components for structural similarity prediction. We also consider an additional task, secondary structure prediction, to assess the model's ability to capture local structure features. We demonstrate that SSA outperforms alternative alignment methods for both of these tasks and that inclusion of the contact prediction training task further improves performance. Finally, we demonstrate that the embeddings learned by our model are generally applicable to other protein machine learning problems by leveraging our embeddings to improve the state-of-the-art in transmembrane prediction. This work presents the first attempt in learning protein sequence embeddings from structure and takes a step towards bridging the sequence-structure divide with representation learning. Current work in protein sequence embeddings has primarily been focused on unsupervised k-mer co-occurence approaches to learn fixed sized vector representations BID6 BID7 based on similar methods in NLP BID8 BID9 BID10 BID11. Melvin et al. BID12 also learn fixed sized semantic embeddings by projecting alignment scores into a low dimensional vector to recapitulate rankings given by existing alignment tools and shared superfamily membership. However, fixed sized vector representations are limited, because they are not usable for any sequence labeling problems (e.g. active site prediction, transmembrane region prediction, etc.). Other methods have focused on manual feature engineering based on biophysical and sequence attributes BID13. These methods rely on expert knowledge and do not capture properties that emerge from interactions between amino acids. Instead, we seek to learn embeddings that encode the full structural context in which each amino acid occurs. This is inspired partly by the recent success of unsupervised contextual embedding models using bidirectional recurrent neural network language models BID14 BID15 where word embeddings, learned as a function of their context, have been successfully transferred to other tasks. In particular, we apply a similar language model for the first time on protein sequences as part of our supervised framework. Supervised embedding models have also been trained for natural language inference (NLI) but produce only fixed sized embeddings BID16 BID17 BID18.At the same time, problems involving word alignment given matched sequences, such as cross lingual word embeddings and document similarity, have also been explored. Cross lingual word embeddings are learned from unaligned parallel text, where sentences are matched between languages but words are not. Kočiský et al. BID19 learn bilingual word embeddings jointly with a FastAlign BID20 word alignment model using expectation maximization. BilBOWA BID21 learns cross lingual word embeddings using parallel sentences without word level alignments by assuming a uniform alignment between words. However, Gouws et al. BID21 assume all pairings between words are equally likely and do not infer them from current values of the embeddings. Related methods have been developed for measuring similarity between documents based on their words. Word Mover's Distance (WMD) and its supervised variant align words between pairs of documents by solving an optimal transport problem given by distance between word vectors. However, these methods are not designed for learning neural network embedding models and the embeddings are not contextual. Furthermore, WMD alignments are prohibitively expensive when alignments must be computed at every optimization step, scaling as O(p 3 logp) where p is the number of unique words. Our SSA solves these problems via an alignment mechanism inspired by previous work using soft alignments and attention mechanisms for sequence modeling BID22 BID23. Further elaborate directional alignments have been used for question answering and reading comprehension models BID24 BID25 BID26 and for natural language inference BID27 BID28 BID28. Unlike these methods, however, our SSA method is both symmetric and memoryless. Furthermore, it is designed for learning interpretable embeddings based on a similarity measure between individual sequence elements. It is also fast and memory efficient -scaling with the product of the sequence lengths. Protein fold recognition is the problem of classifying proteins into folds (for example, as defined by the SCOP database) based on their sequences. Approaches to this problem have largely been based on sequence homology using sequence similarity to classify structures based on close sequence matches BID4 BID29. These methods are either direct sequence alignment tools BID30 BID31 or based on profile HMMs in which multiple sequence alignments are first built by iterative search against a large sequence database, the multiple sequence alignments are converted into profile HMMs, and then sequences are compared using HMM-sequence or HMM-HMM alignments BID32 BID33. However, these methods are only appropriate for matching proteins with high sequence similarity BID1 BID2 BID29. In contrast, we focus on learning protein sequence representations that directly capture structure information in an easily transferable manner. We hypothesize that this approach will improve our ability to detect structural similarity from sequence while also producing useful features for other learning tasks. In this section, we describe the three components of our framework (FIG0) in detail: the specific choice of embedding model, a multi-layer bidirectional LSTM with additional inputs from a pretrained LSTM language model, soft symmetric alignment and ordinal regression components for relating sequences of vector representations for pairs of proteins to their global structural similarity, and the pairwise feature vectors and convolutional neural network design for residue-residue contact prediction. BiLSTM encoder. The encoder takes a sequence of amino acids representing a protein and encodes it into a sequence of vector representations of the same length. To allow the vector representations at each position to be functions of all surrounding amino acids, we structure the encoder as a stack of bidirectional LSTMs followed by a linear layer projecting the outputs of the last biLSTM layer into the final embedding space (Appendix Figure 2).Pretrained language model. The concept of feeding LSTM language model representations as inputs for supervised learning problems as part of a larger neural network model has shown recent success in NLP but has not yet been tried for biological sequences. Inspired partially by the success of ELMo BID15, we consider, in addition to 1-hot representations of the amino acids, the inclusion of the hidden layers of a pretrained bidirectional LSTM language model as inputs to the encoder described above. The language model is pretrained on the raw protein sequences in the protein families database (Pfam) BID34 to predict the amino acid at each position of each protein given the previous amino acids and the following amino acids (see Appendix section A.1 for details).Specifically, given the language model hidden states at each position, i, denoted as h, and the 1-hot representation of the amino acid at those positions, x i, we introduce a learned linear transformation of these representations with ReLU non-linearity, DISPLAYFORM0 which is passed as input to the biLSTM sequence encoder. The parameters W LM, W x, and b are trained together with the parameters of the biLSTM encoder. The parameters of the language model itself are frozen during training. In experiments without the language model, h LM i is set to zero for all positions. The primary task we consider for training the sequence embedding model with structural information is the prediction of global structural similarity between protein sequences as defined by shared membership in the SCOP hierarchy. SCOP is an expertly curated database of protein domain structures in which protein domains are assigned into a hierarchy of structures (Appendix Figure 3). Specifically, we define this as a multiclass classification problem in which a pair of proteins is classified into no similarity, class level similarity, fold level similarity, superfamily level similarity, or family level similarity based on the most specific level of the SCOP hierarchy shared by those proteins. We encode these labels as y ∈ {0, 1, 2, 3, 4} based on the number of levels shared (i.e. y=0 encodes no similarity, y=1 encodes class similarity, etc.). In the following two sections, we describe how protein sequences are compared based on their sequences of vector embeddings using soft symmetric alignment and then how this alignment score is used to predict the specific similarity class by taking advantage of the natural ordering of these classes in an ordinal regression framework. In order to calculate the similarity of two amino acid sequences given that each has been encoded into a sequence of vector representations, z 1...z n and z 1...z m, we develop a soft symmetric alignment mechanism in which the similarity between two sequences is calculated based on their vector embeddings asŝ DISPLAYFORM0 where a ij are entries of the alignment matrix given by DISPLAYFORM1, and DISPLAYFORM2 and A = n i=1 m j=1 a ij is the length of the alignment. Next, to relate this scalar similarity to the ordinal structural similarities defined using SCOP (y ∈ {0, 1, 2, 3, 4}), we adopt an ordinal regression framework. Specifically, we learn a series of binary classifiers to predict whether the structural similarity level is greater than or equal to each level, t, given the alignment score (Equation 1). Given parameters θ 1...θ 4 and b 1...b 4, the probability that two sequences share similarity greater than or equal to t is defined byp(y ≥ t) = sigmoid(θ tŝ + b t), with the constraint that θ t ≥ 0 to enforce thatp increases monotonically withŝ. The structural similarity loss is then given by DISPLAYFORM0 These parameters are fit jointly with the parameters of the sequence encoder by backpropogating through the SSA which is fully differentiable. Furthermore, given these classifiers, the predicted probability that two sequences belong to structural similarity level t isp(y = t) =p(y ≥ t)(1−p(y ≥ t + 1)) withp(y ≥ 0) = 1 by definition. We can augment our SSA framework, in which position-level correspondence is inferred between sequences, with position-level supervision directly in the form of within protein contacts between residues. We introduce a secondary task of within protein residue-residue contact prediction with the hypothesis that the fine-grained structural supervision provided by the observed contacts will improve the quality of the embeddings. Contact prediction is a binary classification problem in which we seek to predict whether residues at positions i and j within an amino acid sequence make contact in the 3D structure. Following common practice in the protein structure field, we define two positions as making contact if the Cα atoms of those residues occur within 8Åin the 3D structure. In order to predict contacts from the sequence of embedding vectors given by the encoder for an arbitrary protein of length N, we define a pairwise features tensor of size (NxNx2D) where D is the dimension of the embedding vector containing pairwise features given by the concatenation of the absolute element-wise differences and the element-wise products of the vector representations for each pair of positions, DISPLAYFORM0 We choose this featurization because it is symmetric, v ij = v ji, and has shown widespread utility for pairwise comparison models in NLP BID35. These vectors are then transformed through a single hidden layer of dimension H (implemented as a width 1 convolutional layer) and ReLU activation giving h ij = ReLU(W v ij + b). Contact predictions are then made by convolving a single 7x7 filter over the ing NxNxH tensor with padding and sigmoid activation to give an NxN matrix containing the predicted probability for each pair of residues forming a contact. Given the observed contacts, we define the contact prediction loss, L contact, to be the expectation of the cross entropy between the observed labels and the predicted contact probabilities taken over all pairs of residues within each protein in the dataset. Complete multitask loss. We define the full multitask objective by DISPLAYFORM1 where λ is a parameter that interpolates between the structural similarity and contact prediction losses. This error signal is backpropogated through the contact prediction specific parameters defined in section 3.3, the similarity prediction specific parameters defined in section 3.2, and the parameters of sequence encoder defined in section 3.1 to train the entire model end-to-end. Our encoder consists of 3 biLSTM layers with 512 hidden units each and a final output embedding dimension of 100 (Appendix Figure 2). Language model hidden states are projected into a 512 dimension vector before being fed into the encoder. In the contact prediction module, we use a hidden layer with dimension 50. These hyperparameters were chosen to be as large as possible while fitting on a single GPU with reasonable minibatch size. While we compare performance with simpler encoder architectures in section 4.2, it is possible that performance could be improved further with careful architecture search. However, that is beyond the scope of this work. Sequence embedding models are trained for 100 epochs using ADAM with a learning rate of 0.001 and otherwise default parameters provided by PyTorch. Each epoch consists of 100,000 examples sampled from the SCOP structural similarity training set with smoothing of the similarity level distribution of 0.5. In other words, the probability of sampling a pair of sequences with similarity level t is proportional to N 0.5 t where N t is the number of sequence pairs with t similarity in the training set. This is to slightly upweight sampling of highly similar pairs of sequences that would otherwise be rare. We choose 0.5 specifically such that a minibatch of size 64 is expected to contain two pairs of sequences with family level similarity. The structural similarity component of the loss is estimated with minibatches of 64 pairs of sequences. When using the full multitask objective, the contact prediction component uses minibatches of 10 sequences and λ = 0.1. Furthermore, during training we apply a small perturbation to the sequences by resampling the amino acid at each position from the uniform distribution with probability 0.05. These hyperparameters were selected using a validaton set as described in Appendix section A.2. All models were implemented in PyTorch and trained on a single NVIDIA Tesla V100 GPU. Each model took roughly 3 days to train and required 16 GB of GPU RAM. Additional runtime and memory details can be found in Appendix A.3.In the following sections, we refer to the 3-layer biLSTM encoder trained with the full framework as "SSA (full)" and the framework without contact prediction (i.e. λ = 1) as "SSA (no contact prediction)." We first evaluate the performance of our full SSA embedding model for predicting structural similarity between amino acid sequences using the SCOP dataset. We benchmark our embedding model against several widely used sequence-based protein comparison methods, Needleman-Wunsch alignment (NW-align), phmmer BID32, an HMM-to-sequence comparison method, and HHalign BID33, an HMMto-HMM comparison method. For HHalign, profle HMMs for each sequence were constructed by iterative search against the uniref30 database. We also benchmark our method agains TMalign, a method for evaluating protein similarity based on alignments of protein structures. Details for each of these baselines can be found in Appendix section A.4. Methods are compared based on accuracy Table 1: Comparison of the full SSA model with three protein sequence alignment methods (NWalign, phmmer, and HHalign) and the structure alignment method TMalign. We measure accuracy, Pearson's correlation (r), Spearman's rank correlation (ρ), and average precision scores for retrieving protein pairs with structural similarity of at least class, fold, superfamily, and family levels. of classifying the shared SCOP level, correlation between the similarity score and shared SCOP level, and average precision scores when considering correct matches at each level of the SCOP hierarchy (i.e. proteins that are in the same class, proteins in the same fold, etc.).The SCOP benchmark datasets are formed by splitting the SCOPe ASTRAL 2.06 dataset, filtered to a maximum sequence identity of 95%, into 22,408 train and 5,602 heldout sequences. From the heldout sequences, we randomly sample 100,000 pairs as the ASTRAL 2.06 structural similarity test set. Furthermore, we define a second test set using the newest release of SCOPe (2.07) by collecting all protein sequences added between the 2.06 and 2.07 ASTRAL releases. This gives a set of 688 protein sequences all pairs of which define the ASTRAL 2.07 new test set. The average percent identity between pairs of sequences within all three datasets is 13%. Sequence length statistics can be found in Appendix TAB4.We find that our full SSA embedding model outperforms all other methods on all metrics on both datasets. On the 2.06 test set, we improve overall prediction accuracy from 0.79 to 0.95, Pearson's correlation from 0.37 to 0.91, and Spearman's rank correlation from 0.23 to 0.69 over the next best sequence comparison method, HHalign, without requiring any database search to construct sequence profiles. Furthermore, our full SSA model is much better for retrieving proteins sharing the same fold -the structural level of most interest for finding distant protein homologues -improving the average precision score by 0.28 on the 2.06 test set and 0.10 on the 2.07 test set over HHalign. We find that our full SSA embedding model even outperforms TMalign, a method for comparing proteins based on their 3D structures, when predicting shared SCOP membership. This is remarkable considering that our model uses only sequence information when making predictions whereas TMalign is provided with the known protein structures. The largest improvement comes at the SCOP class level where TMalign achieves much lower average precision score for retrieving these weak structural matches. We next evaluate the individual model components on two tasks: structure similarity prediction on the ASTRAL 2.06 test set and 8-class secondary structure prediction on a 40% sequence identity filtered dataset containing 22,086 protein sequences from the protein data bank (PDB) BID36, a repository of experimentally determined protein structures. Secondary structure prediction is a sequence labeling problem in which we attempt to classify every position of a protein sequence into one of eight classes describing the local 3D structure at that residue. We use this task to measure the utility of our embeddings for position specific prediction problems. For this problem, we split the secondary Table 2: Study of individual model components. Results of structural similarity prediction on the ASTRAL 2.06 test set and secondary structure prediction are provided for embedding models trained with various components of our multitask framework. The SSA model trained without the language model component of the encoder and without contact prediction (SSA (without language model)), the SSA, UA, and ME models trained without contact prediction (ME, UA, SSA (without contact prediction)), and the full SSA embedding model. structure dataset into 15,461 training and 6,625 testing sequences. We then treat each position of each sequence as an independent datapoint with features either given by the 100-d embedding vector or 1-hot encoding of the k-mer at that position and train a fully connected neural network (2 hidden layers, 1024 units each, ReLU activations) to predict the secondary structure class from the feature vector. These models are trained with cross entropy loss for 10 epochs using ADAM with learning rate 0.001 and a minibatch size of 256.SSA outperforms alternative comparison methods. We first demonstrate the importance of our SSA mechanism when training the contextual embedding model by comparing the performance of biLSTM encoders trained with SSA versus the same encoders trained with uniform alignment and a mean embedding comparison approaches BID21. In uniform alignment (UA), we consider a uniform prior over possible alignments giving the similarity score. For the mean embedding method (ME), we instead calculate the similarity score based on the difference between the average embedding of each sequence. For these baselines, we substitutê DISPLAYFORM0 in for SSA similarity (Equation 1) respectively during model training and prediction. These models are trained without contact prediction (λ = 1) to compare the alignment component in isolation. We find that not only are the SSA embeddings better predictors of secondary structure than k-mer features (accuracy 0.487 vs. 0.444 for 3-mers), but that the SSA mechanism is necessary for achieving best performance on both the structure similarity and local structure prediction tasks. As seen in table 2, the ME model achieves close to SSA performance on secondary structure prediction, but is significantly worse for SCOP similarity prediction. The UA model, on the other hand, is close to SSA on SCOP similarity but much worse when predicting secondary structure. This suggests that our SSA mechanism captures the best of both methods, allowing embeddings to be position specific as in the ME model but also being better predictors of SCOP similarity as in the UA model. Contact prediction improves embeddings. Although the SSA mechanism allows our embedding model to capture position specific information, we wanted to explore whether positional information within sequences in the form of contact prediction could be used to improve the embeddings. We train models with and without the contact prediction task and find that including contact prediction improves both the structural similarity prediction and secondary structure prediction . The accuracy of secondary structure prediction improves from 0.487 without to 0.630 with contact prediction (Table 2). This suggests that the contact prediction task dramatically improves the quality Table 3: Accuracy of transmembrane prediction using structural embeddings in 10-fold cross validation and comparison with other transmembrane prediction methods. BiLSTM+CRF models using either our full SSA model embeddings or 1-hot encodings of the amino acids as features are displayed below the dotted line. We compare with for a variety of transmembrane prediction methods previously reported on the TOPCONS dataset.of the local embeddings on top of the weak supervision provided by whole structure comparison. For reference, we also report contact prediction performance for our full SSA model in Appendix A.5.Encoder architecture and pretrained language model are important. Finally, we show, for the first time with biological sequences, that a language model pretrained on a large unsupervised protein sequence database can be used to transfer information to supervised sequence modeling problems. SCOP similarity classification for SSA embedding models trained with and without LM hidden layer inputs shows that including the LM substantially improves performance, increasing accuracy from 0.898 to 0.938. Furthermore, we examine the extent to which LM hidden states capture all useful structural information by training SSA embedding models with less expressive power than our 3-layer biLSTM architecture (Appendix TAB5). We find that the LM hidden states are not sufficient for high performance on the structural similarity task with linear, fully connected (i.e. width 1 convolution), and single layer biLSTM embedding models having lower accuracy, Pearson, and Spearman correlations than the 3-layer biLSTM on the ASTRAL 2.06 test set. We demonstrate the potential utility of our protein sequence embedding model for transfering structural information to other sequence prediction problems by leveraging our embedding model for transmembrane prediction. In transmembrane prediction, we wish to detect which, if any, segments of the amino acid sequence cross the lipid bilayer for proteins integrated into the cell membrane. This is a well studied problem in protein biology with methods generally consisting of HMMs with sophisticated, manually designed hidden state transition distributions and emission distributions including information about residue identity, amino acid frequencies from multiple sequence alignments, local structure, and chemical properties. Newer methods are also interested in detection of signal peptides, which are short amino acid stretches at the beginning of a protein sequence signaling for this protein to be inserted into the cell membrane. To benchmark our embedding vectors for this problem, we develop a conditional random field (CRF) model in which the propensity of each hidden state given the sequence of embedding vectors is defined by a single layer biLSTM with 150 units. As a baseline, we include an identical biLSTM + CRF model using only 1-hot encodings of the amino acids as features. For the transition probabilities between states, we adopt the same structure as used in TOPCONS BID37 and perform 10-fold cross validation on the TOPCONS transmembrane benchmark dataset. We report for correctly predicting regions in proteins with only transmembrane domains (TM), transmembrane domains and a signal peptide (SP+TM), neither transmembrane nor signal peptide domains (Globular), or a signal peptide but no transmembrane regions (Globular+SP). Transmembrane state labels are predicted with Viterbi decoding. Again following TOPCONS, predictions are counted as correct if, for TM proteins, our model predicts no signal peptide, the same number of transmembrane regions, and those regions overlap with real regions by at least five positions. Correct SP+TM predictions are defined in the same way except that proteins must be predicted to start with a signal peptide. Globular protein predictions are correct if no transmembrane or signal peptides are predicted and Globular+SP predictions are correct if only a leading signal peptide is predicted. We find that our transmembrane predictions rank first or tied for first in 3 out of the 4 categories (SP+TM, Globular, and Globular+SP) and ranks second for the TM category. Overall, our transmembrane predictions are best with prediction accuracy of 0.89 vs 0.87 for TOPCONS. Remarkably, this is by simply replacing the potential function in the CRF with a function of our embedding vectors, the hidden state grammar is the same as that of TOPCONS. Furthermore, the performance cannot be attributed solely to the biLSTM+CRF structure, as the biLSTM+CRF with 1-hot encoding of the amino acids performs poorly, tying MEMSAT-SVM for worst performance. This is particularly noteworthy, because TOPCONS is a meta-predictor. It uses outputs from a wide variety of other transmembrane prediction methods to define the transmembrane state potentials. In this work, we proposed a novel alignment approach to learning contextual sequence embeddings with weak supervision from a global similarity measure. Our SSA model is fully differentiable, fast to compute, and can be augmented with position-level structural information. It outperforms competition in predicting protein structural similarity including, remarkably, structure alignment with TMalign. One consideration of training using SCOP, however, is that we focus exclusively on single-domain protein sequences. This means that the highly contextual embeddings given by the biLSTM encoder to single domains may differ from embeddings for the same domain in a multi-domain sequence. One interesting extension would thus be to modify the encoder architecture or training procedure to better model domains in multi-domain contexts. Nonetheless, the ing embeddings are widely useful, allowing us to improve over the state-of-the-art in transmembrane region prediction, and can easily be applied to other protein prediction tasks such as predicting functional properties, active site locations, protein-protein interactions, etc. Most methods that use HMM sequence profiles or position-specific scoring matrices could be augmented with our embeddings. The broader framework extends to other related (non-biological) tasks. A APPENDIX Figure The bidirectional LSTM language model was trained on the full set of protein domain sequences in the Pfam database, 21,827,419 total sequences. The language model was trained to predict the amino acid at position i given observations of all amino acids before i and all amino acids after i by minimizing the cross entropy loss with log predicted log probabilities given by the sum of the forward and reverse LM direction predictions DISPLAYFORM0 where p F (x i) is the probability given by the forward direction LSTM and p R (x i) is the probability given by the reverse direction LSTM.The language model architecture consisted of a 2-layer LSTM with 1024 units in each layer followed by a linear transformation into the 20-d amino acid prediction. All parameters were shared between the forward and reverse direction components. The model was trained for a single epoch using ADAM with a learning rate of 0.001 and minibatch size of 32. We select the resampling probability and λ hyperparameters based on structural similarity prediction accuracy on a validation set held out from the SCOP ASTRAL 2.06 training set (Section 4.1). For these experiments, we hold out 2,240 random sequences from the 22,408 sequences of the training set. From these held out sequences, we randomly sample 100,000 pairs as the validation set. Resampling Contact Prediction Accuracy Table 6: Evaluation of amino acid resampling probability and contact prediction loss weight, λ, for structural similarity prediction on the validation set. (Top) Resampling probability of 0.05 is compared with no resampling for 3-layer biLSTM encoders with and without language model components that are trained using SSA without contact prediction. (Bottom) Comparison of models trained using the full framework with λ = 0.5, λ = 0.33, and λ = 0.1.Resampling probability. We consider models trained with amino acid resampling probability 0.05 and without amino acid resampling in the simplified framework (SSA without contact prediction) using the 3-layer biLSTM encoder with and without the language model component. We find that the structural similarity are slightly improved when using amino acid resampling regardless of whether the LM component of the encoder is included (Appendix Table 6). Based on this , all models were trained with amino acid resampling of 0.05.Multitask loss weight, λ. We evaluate three values of λ, interpolating between the structural similarity loss and contact prediction loss, for predicting structural similarity on the validation set. Prediction accuracy increased progressively as λ decreased and all models trained with contact prediction outperformed those trained without contact prediction (Appendix Table 6). Because it gave the best prediction accuracy on the validation set, λ = 0.1 was selected for training models with our full framework. For the NW-align method, similarity between protein sequences was computed using the BLOSUM62 substitution matrix with gap open and extend penalties of -11 and -1 respectively. For phmmer, each pair of sequences was compared in both directions (i.e. query->target and target->query) using the'-max' option. The similarity score for each pair was treated as the average off the query->target and target->query scores. For HHalign, multiple sequence alignments were first built for each sequence by using HHblits to search for similar sequences in the uniclust30 database with a maximum of 2 rounds of iteration (-n 2). Sequences pairs were then scored by using HHalign to score the target->query and query->target HMM-HMM alignments. Again, the average of the two scores was treated as the overall HHalign score. Finally, for TMalign, the structures for each pair of proteins were aligned and the scores for the query->target and target->query alignments were averaged to give the overall TMalign score for each pair of proteins. To calculate the classification accuracy from the above scores, thresholds were found to maximize prediction accuracy when binning scores into similarity levels using 100,000 pairs of sequences sampled from the ASTRAL 2.06 training set. Although we use contact prediction as an auxiliary task to provide position-level structural supervision for the purpose of improving embedding quality and structure similarity prediction, we include for predicting contacts using the trained contact prediction module here. We report for contact prediction using the full SSA model on the SCOP ASTRAL 2.06 test set and 2.07 new test set in Appendix Table 7. Precision, recall, and F1 score are calculated using a probability threshold of 0.5 for assigning predicted contacts. We consider performance for predicting all contacts (i.e. contacts between all amino acid positions, excluding neighbors, |i − j| ≥ 2), our training objective, and for distant contacts (|i − j| ≥ 12) which are the focus of co-evolution based methods. We also report the precision of the top L, L/2, and L/5 contact predictions, where L is the length of the protein sequence. Table 7: Contact prediction performance of the full SSA model on the SCOP ASTRAL 2.06 test set and the 2.07 new test set. We report precision, recall, F1 score, and the area under the precision-recall curve (AUPR) for predicting all contacts (|i − j| ≥ 2) and distant contacts (|i − j| ≥ 12) in the test set proteins. We also report precision of the top L, L/2, and L/5 predicted contacts. DISPLAYFORM0 In order to facilitate some comparison with state-of-the-art co-evolution based contact prediction methods, we also report for contact prediction using our full SSA model on the publicly released set of free modelling targets from CASP12 BID39. This dataset consists of 21 protein domains. We include the complete list at the end of this section. We compare with deep convolutional neural network models using co-evolution features, RaptorX-Contact BID40, iFold & Deepfold-Contact BID41, and MetaPSICOV BID42, and with GREMLIN BID43, an entirely co-evolution based approach. We find that our model dramatically outperforms these methods when predicting all contacts but performs worse when predicting only distant contacts (Appendix Table 8). This is unsurprising, as our model is trained to predict all contacts, of which local contacts are much more abundant than distant contacts, whereas the co-evolution methods are designed to predict distant contacts. Furthermore, we wish to emphasize that our model is tuned for structural similarity prediction and that our contact prediction module is extremely simple, being only a single fully connected layer followed by a single convolutional layer. It is possible that much better performance could be achieved using our embeddings with a more sophisticated contact prediction architecture. That said, our model does outperform the pure co-evolution method, GREMLIN, based on AUPR for predicting distant contacts. These suggest that our embeddings may be useful as features, in combination with co-evolution based features, for improving dedicated contact prediction models on both local and distant contacts. Table 8: Contact prediction performance of the full SSA model on the CASP12 free modelling targets with of state-of-the-art co-evolution based methods for comparison. We report precision, recall, F1 score, and the area under the precision-recall curve (AUPR) for predicting all contacts (|i − j| ≥ 2) and distant contacts (|i − j| ≥ 12). We also report precision of the top L, L/2, and L/5 predicted contacts. TAB4

We present a method for learning protein sequence embedding models using structural information in the form of global structural similarity between proteins and within protein residue-residue contacts.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.