Question: Represent the article for finding a single-sentence TLDR

Question: A capsule is a group of neurons whose outputs represent different properties of the same entity. Each layer in a capsule network contains many capsules. We describe a version of capsules in which each capsule has a logistic unit to represent the presence of an entity and a 4x4 matrix which could learn to represent the relationship between that entity and the viewer (the pose). A capsule in one layer votes for the pose matrix of many different capsules in the layer above by multiplying its own pose matrix by trainable viewpoint-invariant transformation matrices that could learn to represent part-whole relationships. Each of these votes is weighted by an assignment coefficient. These coefficients are iteratively updated for each image using the Expectation-Maximization algorithm such that the output of each capsule is routed to a capsule in the layer above that receives a cluster of similar votes. The transformation matrices are trained discriminatively by backpropagating through the unrolled iterations of EM between each pair of adjacent capsule layers. On the smallNORB benchmark, capsules reduce the number of test errors by 45\% compared to the state-of-the-art. Capsules also show far more resistance to white box adversarial attacks than our baseline convolutional neural network. Convolutional neural nets are based on the simple fact that a vision system needs to use the same knowledge at all locations in the image. This is achieved by tying the weights of feature detectors so that features learned at one location are available at other locations. Convolutional capsules extend the sharing of knowledge across locations to include knowledge about the part-whole relationships that characterize a familiar shape. Viewpoint changes have complicated effects on pixel intensities but simple, linear effects on the pose matrix that represents the relationship between an object or object-part and the viewer. The aim of capsules is to make good use of this underlying linearity, both for dealing with viewpoint variations and for improving segmentation decisions. Capsules use high-dimensional coincidence filtering: a familiar object can be detected by looking for agreement between votes for its pose matrix. These votes come from parts that have already been detected. A part produces a vote by multiplying its own pose matrix by a learned transformation matrix that represents the viewpoint invariant relationship between the part and the whole. As the viewpoint changes, the pose matrices of the parts and the whole will change in a coordinated way so that any agreement between votes from different parts will persist. Finding tight clusters of high-dimensional votes that agree in a mist of irrelevant votes is one way of solving the problem of assigning parts to wholes. This is non-trivial because we cannot grid the high-dimensional pose space in the way the low-dimensional translation space is gridded to facilitate convolutions. To solve this challenge, we use a fast iterative process called "routingby-agreement" that updates the probability with which a part is assigned to a whole based on the proximity of the vote coming from that part to the votes coming from other parts that are assigned to that whole. This is a powerful segmentation principle that allows knowledge of familiar shapes to derive segmentation, rather than just using low-level cues such as proximity or agreement in color or velocity. An important difference between capsules and standard neural nets is that the activation of a capsule is based on a comparison between multiple incoming pose predictions whereas in a standard neural net it is based on a comparison between a single incoming activity vector and a learned weight vector. Neural nets typically use simple non-linearities in which a non-linear function is applied to the scalar output of a linear filter. They may also use softmax non-linearities that convert a whole vector of logits into a vector of probabilities. Capsules use a much more complicated non-linearity that converts the whole set of activation probabilities and poses of the capsules in one layer into the activation probabilities and poses of capsules in the next layer. A capsule network consists of several layers of capsules. The set of capsules in layer L is denoted as Ω L. Each capsule has a 4x4 pose matrix, M, and an activation probability, a. These are like the activities in a standard neural net: they depend on the current input and are not stored. In between each capsule i in layer L and each capsule j in layer L + 1 is a 4x4 trainable transformation matrix, W ij. These W ij s (and two learned biases per capsule) are the only stored parameters and they are learned discriminatively. The pose matrix of capsule i is transformed by W ij to cast a vote V ij = M i W ij for the pose matrix of capsule j. The poses and activations of all the capsules in layer L + 1 are calculated by using a non-linear routing procedure which gets as input V ij and a i for all i ∈ Ω L, j ∈ Ω L+1.The non-linear procedure is a version of the Expectation-Maximization procedure. It iteratively adjusts the means, variances, and activation probabilities of the capsules in layer L + 1 and the assignment probabilities between all i ∈ Ω L, j ∈ Ω L+1. In appendix 1, we give a gentle intuitive introduction to routing-by-agreement and describe in detail how it relates to the EM algorithm for fitting a mixture of Gaussians. Let us suppose that we have already decided on the poses and activation probabilities of all the capsules in a layer and we now want to decide which capsules to activate in the layer above and how to assign each active lower-level capsule to one active higher-level capsule. Each capsule in the higher-layer corresponds to a Gaussian and the pose of each active capsule in the lower-layer (converted to a vector) corresponds to a data-point (or a fraction of a data-point if the capsule is partially active).Using the minimum description length principle we have a choice when deciding whether or not to activate a higher-level capsule. Choice 0: if we do not activate it, we must pay a fixed cost of −β u per data-point for describing the poses of all the lower-level capsules that are assigned to the higher-level capsule. This cost is the negative log probability density of the data-point under an improper uniform prior. For fractional assignments we pay that fraction of the fixed cost. Choice 1: if we do activate the higher-level capsule we must pay a fixed cost of −β a for coding its mean and variance and the fact that it is active and then pay additional costs, pro-rated by the assignment probabilities, for describing the discrepancies between the lower-level means and the values predicted for them when the mean of the higher-level capsule is used to predict them via the inverse of the transformation matrix. A much simpler way to compute the cost of describing a datapoint is to use the negative log probability density of that datapoint's vote under the Gaussian distribution fitted by whatever higher-level capsule it gets assigned to. This is incorrect for reasons explained in appendix 1, but we use it because it requires much less computation (also explained in the appendix). The difference in cost between choice 0 and choice 1, is then put through the logistic function on each iteration to determine the higher-level capsule's activation probability. Appendix 1 explains why the logistic is the correct function to use. Using our efficient approximation for choice 1 above, the incremental cost of explaining a whole data-point i by using an active capsule j that has an axis-aligned covariance matrix is simply the sum over all dimensions of the cost of explaining each dimension, h, of the vote V ij. This is simply −ln(P h i|j) where P h i|j is the probability density of the h th component of the vectorized vote V ij under j's Gaussian model for dimension h which has variance (σ h j) 2 and mean µ h j where µ j is the vectorized version of j's pose matrix M j. DISPLAYFORM0 Summing over all lower-level capsules for a single dimension, h, of j we get: DISPLAYFORM1 where i r ij is the amount of data assigned to j and V h ij is the value on dimension h of V ij. Turningon capsule j increases the description length for the means of the lower-level capsules assigned to j from −β u per lower-level capsule to −β a plus the sum of the cost over all dimensions so we define the activation function of capsule j to be: DISPLAYFORM2 where β a is the same for all capsules and λ is an inverse temperature parameter. We learn β a and β u discriminatively and set a fixed schedule for λ as a hyper-parameter. For finalizing the pose parameters and activations of the capsules in layer L + 1 we run the EM algorithm for few iterations (normally 3) after the pose parameters and activations have already been finalized in layer L. The non-linearity implemented by a whole capsule layer is a form of cluster finding using the EM algorithm, so we call it EM Routing. Procedure 1 Routing algorithm returns activation and pose of the capsules in layer L + 1 given the activations and votes of capsules in layer L. V h ij is the h th dimension of the vote from capsule i with activation a i in layer L to capsule j in layer L + 1. β a, β u are learned discriminatively and the inverse temperature λ increases at each iteration with a fixed schedule. DISPLAYFORM3 for t iterations do 4: DISPLAYFORM4 The general architecture of our model is shown in FIG0. The model starts with a 5x5 convolutional layer with 32 channels (A=32) and a stride of 2 with a ReLU non-linearity. All the other layers are capsule layers starting with the primary capsule layer. The 4x4 pose of each of the B=32 primary capsule types is a learned linear transformation of the output of all the lower-layer ReLUs centered at that location. The activations of the primary capsules are produced by applying the sigmoid function to the weighted sums of the same set of lower-layer ReLUs. The primary capsules are followed by two 3x3 convolutional capsule layers (K=3), each with 32 capsule types (C=D=32) with strides of 2 and one, respectively. The last layer of convolutional capsules is connected to the final capsule layer which has one capsule per output class. When connecting the last convolutional capsule layer to the final layer we do not want to throw away information about the location of the convolutional capsules but we also want to make use of the fact that all capsules of the same type are extracting the same entity at different positions. We therefore share the transformation matrices between different positions of the same capsule type and add the scaled coordinate (row, column) of the center of the receptive field of each capsule to the first two elements of the right-hand column of its vote matrix. We refer to this technique as Coordinate Addition. This should encourage the shared final transformations to produce values for those two elements that represent the fine position of the entity relative to the center of the capsule's receptive field. The routing procedure is used between each adjacent pair of capsule layers. For convolutional capsules, each capsule in layer L + 1 sends feedback only to capsules within its receptive field in layer L. Therefore each convolutional instance of a capsule in layer L receives at most kernel size X kernel size feedback from each capsule type in layer L + 1. The instances closer to the border of the image receive fewer feedbacks with corner ones receiving only one feedback per capsule type in layer L + 1. In order to make the training less sensitive to the initialization and hyper-parameters of the model, we use "spread loss" to directly maximize the gap between the activation of the target class (a t) and the activation of the other classes. If the activation of a wrong class, a i, is closer than the margin, m, to a t then it is penalized by the squared distance to the margin: DISPLAYFORM0 By starting with a small margin of 0.2 and linearly increasing it during training to 0.9, we avoid dead capsules in the earlier layers. Spread loss is equivalent to squared Hinge loss with m = 1. studies a variant of this loss in the context of multi class SVMs. The smallNORB dataset BID18 ) has gray-level stereo images of 5 classes of toys: airplanes, cars, trucks, humans and animals. There are 10 physical instances of each class which are painted matte green. 5 physical instances of a class are selected for the training data and the other 5 for the test data. Every individual toy is pictured at 18 different azimuths, 9 elevations and 6 lighting conditions, so the training and test sets each contain 24,300 stereo pairs of 96x96 images. We selected smallNORB as a benchmark for developing our capsules system because it is carefully designed to be a pure shape recognition task that is not confounded by context and color, but it is much closer to natural images than MNIST. We downsample smallNORB to 48 × 48 pixels and normalize each image to have zero mean and unit variance. During training, we randomly crop 32 × 32 patches and add random brightness and contrast to the cropped images. During test, we crop a 32 × 32 patch from the center of the image and achieve 1.8% test error on smallNORB. If we average the class activations over multiple crops at test time we achieve 1.4%. The best reported on smallNORB without using meta data is 2.56% (Cireşan et al. FORMULA1). To achieve this, they added two additional stereo pairs of input images that are created by using an on-center off-surround filter and an off-center on-surround filter. They also applied affine distortions to the images. Our work also beats the BID21 capsule work which achieves 2.7% on smallNORB. We also tested our model on NORB which is a jittered version of smallNORB with added and we achieved a 2.6% error rate which is on par with the state-of-the-art of 2.7% BID1 ).As the baseline for our experiments on generalization to novel viewpoints we train a CNN which has two convolutional layers with 32 and 64 channels respectively. Both layers have a kernel size of 5 and a stride of 1 with a 2 × 2 max pooling. The third layer is a 1024 unit fully connected layer with dropout and connects to the 5-way softmax output layer. All hidden units use the ReLU non-linearity. We use the same image preparation for the CNN baseline as described above for the capsule network. Our baseline CNN was the of an extensive hyperparameter search over filter sizes, numbers of channels and learning rates. The CNN baseline achieves 5.2% test error rate on smallNORB and has 4.2M parameters. We deduce that the Cireşan et al. FORMULA1 network has 2.7M parameters. By using small matrix multiplies, we reduced the number of parameters by a factor of 15 to 310K compared with our baseline CNN (and a factor of 9 w.r.t Cireşan et al. FORMULA1). A smaller capsule network of A = 64, B = 8, C = D = 16 with only 68K trainable parameters achieves 2.2% test error rate which also beats the prior state-of-the-art. FIG2 shows how EM routing adjusts the vote assignments and the capsule means to find the tight clusters in the votes. The histograms show the distribution of vote distances to the mean (pose) of each class capsule during routing iterations. At the first iteration, votes are distributed equally between 5 final layer capsules. Therefore, all capsules receive votes closer than 0.05 to their calculated mean. In the second iteration, the assignment probability for agreeing votes increases. Therefore, most of the votes are assigned to the detected clusters, the animal and human class in the middle row, and the other capsules only receive scattered votes which are further than 0.05 from the calculated mean. The zoomed-out version of FIG2 in the Appendix shows the full distribution of vote distances at each routing iteration. Instead of using our MDL-derived capsule activation term which computes a separate activation probability per capsule, we could view the capsule activations like the mixing proportions in a mixture of Gaussians and set them to be proportional to the sum of the assignment probabilities of a capsule and to sum to 1 over all the capsules in a layer. This increases the test error rate on FIG2: Histogram of distances of votes to the mean of each of the 5 final capsules after each routing iteration. Each distance point is weighted by its assignment probability. All three images are selected from the smallNORB test set. The routing procedure correctly routes the votes in the truck and the human example. The plane example shows a rare failure case of the model where the plane is confused with a car in the third routing iteration. The histograms are zoomed-in to visualize only votes with distances less than 0.05. Fig. B.2 shows the complete histograms for the "human" capsule without clipping the x-axis or fixing the scale of the y-axis. smallNORB to 4.5%. Tab. 1 summarizes the effects of the number of routing iterations, the type of loss, and the use of matrices rather than vectors for the poses. The same capsules architecture as FIG0 achieves 0.44% test error rate on MNIST. If the number of channels in the first hidden layer is increased to 256, it achieves 11.9% test error rate on Cifar10 BID14 ). A more severe test of generalization is to use a limited range of viewpoints for training and to test on a much wider range. We trained both our convolutional baseline and our capsule model on one-third of the training data containing azimuths of and tested on the two-thirds of the test data that contained azimuths from 60 to 280. In a separate experiment, we trained on the 3 smaller elevations and tested on the 6 larger elevations. It is hard to decide if the capsules model is better at generalizing to novel viewpoints because it achieves better test accuracy on all viewpoints. To eliminate this confounding factor, we stopped training the capsule model when its performance matched the baseline CNN on the third of the test set that used the training viewpoints. Then, we compared these matched models on the twothirds of the test set with novel viewpoints. Results in Tab. 2 show that compared with the baseline CNN capsules with matched performance on familiar viewpoints reduce the test error rate on novel viewpoints by about 30% for both novel azimuths and novel elevations. There is growing interest in the vulnerability of neural networks to adversarial examples; inputs that have been slightly changed by an attacker to trick a neural net classifier into making the wrong classification. These inputs can be created in a variety of ways, but straightforward strategies such as FGSM BID9 ) have been shown to drastically decrease accuracy in convolutional neural networks on image classification tasks. We compare our capsule model and a traditional convolutional model on their ability to withstand such attacks. FGSM computes the gradient of the loss w.r.t. each pixel intensity and then changes the pixel intensity by a fixed amount in the direction that increases the loss. So the changes only depend on the sign of the gradient at each pixel. This can be extended to a targeted attack by updating the input to maximize the classification probability of a particular wrong class. We generated adversarial attacks using FGSM because it has only one hyper-parameter and it is easy to compare models that have very different gradient magnitudes. To test the robustness of our model, we generated adversarial images from the test set using a fully trained model. We then reported the accuracy of the model on these images. We found that our model is significantly less vulnerable to both general and targeted FGSM adversarial attacks; a small can be used to reduce a convolutional model's accuracy much more than an equivalent can on the capsule model FIG1. It should also be noted that the capsule model's accuracy after the untargeted attack never drops below chance (20%) whereas the convolutional model's accuracy is reduced to significantly below chance with an as small as 0.2.We also tested our model on the slightly more sophisticated adversarial attack of the Basic Iterative Method BID15 ), which is simply the aforementioned attack except it takes multiple smaller steps when creating the adversarial image. Here too we find that our model is much more robust to the attack than the traditional convolutional model. It has been shown that some robustness to adversarial attacks in models can be due to simple numerical instability in the calculation of the gradient BID0 Finally we tested our model's robustness to black box attacks by generating adversarial examples with a CNN and testing them on both our capsule model and a different CNN. We found that the capsule model did not perform noticeably better at this task than the CNN. Among the multiple recent attempts at improving the ability of neural networks to deal with viewpoint variations, there are two main streams. One stream attempts to achieve viewpoint invariance and the other aims for viewpoint equivariance. The work presented by BID13 ), Spatial Transformer Networks, seeks viewpoint invariance by changing the sampling of CNNs according to a selection of affine transformations. De Brabandere et al. FORMULA1 extends spatial transformer networks where the filters are adapted during inference depending on the input. They generate different filters for each locality in the feature map rather than applying the same transformation to all filters. Their approach is a step toward input covariance detection from traditional pattern matching frameworks like standard CNNs BID17 ). BID4 improves upon spatial transformer networks by generalizing the sampling method of filters. Our work differs substantially in that a unit is not activated based on the matching score with a filter (either fixed or dynamically changing during inference). In our case, a capsule is activated only if the transformed poses coming from the layer below match each other. This is a more effective way to capture covariance and leads to models with many fewer parameters that generalize better. The success of CNNs has motivated many researchers to extend the translational equivariance built in to CNNs to include rotational equivariance BID3, BID6, BID20 ). The recent approach in Harmonic Networks BID23 ) achieves rotation equivariant feature maps by using circular harmonic filters and returning both the maximal response and orientation using complex numbers. This shares the basic representational idea of capsules: By assuming that there is only one instance of the entity at a location, we can use several different numbers to represent its properties. They use a fixed number of streams of rotation orders. By enforcing the equality of the sum of rotation orders along any path, they achieve patch-wise rotation equivariance. This approach is more parameter-efficient than data augmentation approaches, duplicating feature maps, or duplicating filters BID7, BID16 ). Our approach encodes general viewpoint equivariance rather than only affine 2D rotations. Symmetry networks BID8 ) use iterative Lucas-Kanade optimization to find poses that are supported by the most low-level features. Their key weakness is that the iterative algorithm always starts at the same pose, rather than the mean of the bottom-up votes. BID19 proposes a feature detection mechanism (DetNet) that is equivariant to affine transformations. DetNet is designed to detect the same points in the image under different viewpoint variations. This effort is orthogonal to our work but DetNet might be a good way to implement the de-rendering first-stage that activates the layer of primary capsules. Our routing algorithm can be seen as an attention mechanism. In this view, it is related to the work of BID10, where they improved the decoder performance in a generative model by using Gaussian kernels to attend to different parts of the feature map generated by the encoder. BID22 uses a softmax attention mechanism to match parts of the query sequence to parts of the input sequence for the translation task and when generating an encoding for the query. They show improvement upon previous translation efforts using recurrent architectures. Our algorithm has attention in the opposite direction. The competition is not between the lower-level capsules that a higher-level capsule might attend to. It is between the higher-level capsules that a lower-level capsule might send its vote to. Hinton et al. FORMULA1 used a transformation matrix in a transforming autoencoder that learned to transform a stereo pair of images into a stereo pair from a slightly different viewpoint. However, that system requires the transformation matrix to be supplied externally. More recently, routing-byagreement was shown to be effective for segmenting highly overlapping digits BID21 ), but that system has several deficiencies that we have overcome in this paper:1. It uses the length of the pose vector to represent the probability that the entity represented by a capsule is present. To keep the length less than 1, requires an unprincipled non-linearity and this prevents the existence of any sensible objective function that is minimized by the iterative routing procedure. 2. It uses the cosine of the angle between two pose vectors to measure their agreement. Unlike the negative log variance of a Gaussian cluster, the cosine saturates at 1, which makes it insensitive to the difference between a quite good agreement and a very good agreement. 3. It uses a vector of length n rather than a matrix with n elements to represent a pose, so its transformation matrices have n 2 parameters rather than just n. Building on the work of BID21, we have proposed a new type of capsule system in which each capsule has a logistic unit to represent the presence of an entity and a 4x4 pose matrix to represent the pose of that entity. We also introduced a new iterative routing procedure between capsule layers, based on the EM algorithm, which allows the output of each lower-level capsule to be routed to a capsule in the layer above in such a way that active capsules receive a cluster of similar pose votes. This new system achieves significantly better accuracy on the smallNORB data set than the state-of-the-art CNN, reducing the number of errors by 45%. We have also shown it to be significantly more robust to white box adversarial attacks than a baseline CNN.SmallNORB is an ideal data-set for developing new shape-recognition models precisely because it lacks many of the additional features of images in the wild. Now that our capsules model works well on NORB, we plan to implement an efficient version to test much larger models on much larger data-sets such as ImageNet. Dynamic routing is performed between two adjacent layers of capsules. We will refer to these layers as the higher-level and the lower-level. We complete the routing between one pair of layers before starting the routing between the next pair of layers. The routing process has a strong resemblance to fitting a mixture of Gaussians using EM, where the higher-level capsules play the role of the Gaussians and the means of the activated lower-level capsules for a single input image play the role of the datapoints. We start by explaining the cost function that is minimized when using the EM procedure to fit a mixture of Gaussians. We then derive our dynamic routing procedure by making two modifications to the procedure for fitting a mixture of Gaussians. The EM algorithm for fitting a mixture of Gaussians alternates between an E-step and an M-step. The E-step is used to determine, for each datapoint, the probability with which it is assigned to each of the Gaussians. These assignment probabilities act as weights and the M-step for each Gaussian consists of finding the mean of these weighted datapoints and the variance about that mean. If we are also fitting mixing proportions for each Gaussian, they are set to the fraction of the data assigned to the Gaussian. The M-step holds the assignment probabilities constant and adjusts each Gaussian to maximize the sum of the weighted log probabilities that the Gaussian would generate the datapoints assigned to it. The negative log probability density of a datapoint under a Gaussian can be treated like the energy of a physical system and the M-step is minimizing the expected energy where the expectations are taken using the assignment probabilities. The E-step adjusts the assignment probabilities for each datapoint to minimize a quantity called "free energy" which is the expected energy minus the entropy. We can minimize the expected energy by assigning each datapoint with probabilty 1 to whichever Gaussian gives it the lowest energy (i. e. the highest probability density). We can maximize the entropy by assigning each datapoint with equal probability to every Gaussian ignoring the energy. The best trade-off is to make the assignment probabilities be proportional to exp(−E). This is known as the Boltzmann distribution in physics or the posterior distribution in statistics. Since the E-step minimizes the free energy w.r.t. the assignment distribution and the M-step leaves the entropy term unchanged and minimizes the expected energy w.r.t. the parameters of the Gaussians, the free energy is an objective function for both steps. The softmax function computes the distribution that minimizes free energy when the logits are viewed as negative energies. So when we use a softmax in our routing procedure to recompute assignment probabilities we are minimizing a free energy. When we refit the Gaussian model of each capsule we are minimizing the same free energy provided the logits of the softmax are based on the same energies as are optimized when refitting the Gaussians. The energies we use are the negative log probabilities of the votes coming from a lower-level capsule under the Gaussian model of a higher-level capsule. These are not the correct energies for maximizing the log probability of the data (see the discussion of determinants below) but this does not matter for convergence so long as we use the same energies for fitting the Gaussians and for revising the assignment probabilities. The objective function minimizes Eq. 4 which consists of:• MDL cost −β a scaled by the probability of presence of capsules in layer L + 1 (a j, j ∈ Ω L+1).• Negative entropy of activations a j, j ∈ Ω L+1.• The expected energy minimized in M-step: sum of the weighted log probabilities (cost h j).• Negative entropy of routing softmax assignments (R ij) scaled by the probability of presence of the datapoint (a i, i ∈ Ω L). DISPLAYFORM0 In a standard mixture of Gaussians, each Gaussian only has a subset of the datapoints assigned to it but all of the Gaussians see the same data. If we view the capsules in the higher-layer as the Gaussians and the means of the active capsules in the lower-layer as the dataset, each Gaussian sees a dataset in which the datapoints have been transformed by transformation matrices and these matrices are different for different Gaussians. For one higher-level capsule, two transformed datapoints may be close together and for another higher-level capsule the same two datapoints may be transformed into points that are far apart. Every Gaussian has a different view of the data. This is a far more effective way to break symmetry than simply initializing the Gaussians with different means and it generally leads to much faster convergence. If the fitting procedure is allowed to modify the transformation matrices, there is a trivial solution in which the transformation matrices all collapse to zero and the transformed data points are all identical. We avoid this problem by learning the transformation matrices discriminatively in an outer loop and we restrict the dynamic routing to modifying the means and variances of the Gaussians and the probabilities with which the datapoints are assigned to the Gaussians. There is a more subtle version of the collapse problem that arises when different transformation matrices have different determinants. Suppose that the datapoints in a particular subset are transformed into a cluster of points in the pose space of higher-level capsule j and they are transformed into a different but equally tight cluster of points in the pose space of higher-level capsule k. It may seem that j and k provide equally good models of this subset of the datapoints, but this is not correct from a generative modeling perspective. If the transformation matrices that map the datapoints into the pose space used by capsule j have bigger determinants, then j provides a better model. This is because the probability density of a point in the pose space of a lower-level capsule gets diluted by the determinant of the relevant transformation matrix when it is mapped to the pose of a higherlevel capsule. This would be a serious issue if we wanted to learn the transformation matrices by maximizing the probability of the observed datapoints, but we are learning the transformation matrices discriminatively so it does not matter. It does, however, mean that when the dynamic routing maximizes the probability of the transformed datapoints it cannot be viewed as also maximizing the probability of the untransformed points. The obvious way to avoid the determinant issue is to take the mean in pose space of a higher-level capsule and to map this mean back into the pose space of each lower-level capsule using the inverses of the transformation matrices. A mean in a higher-level pose space will generally map to different points in the pose spaces of different lower-level capsules because the pose of a whole will generally make different predictions for the poses of the different parts of that whole. If we use the lowerlevel pose space when measuring the misfit between the actual pose of a lower-level capsule and the top-down prediction of that pose obtained by applying the inverse transformation matrix to the mean of the higher-level capsule, the collapse problem disappears and we can base decisions about routing on a fair comparison of how well two different top-down predictions fit the actual pose of the lower-level capsule. We do not use this correct method for two reasons. First, it involves inverting the transformation matrices. Second, it requires a new multiplication by the inverse transformation matrices every time the higher-level mean is modified during the dynamic routing. By measuring misfits in the higher-level pose space we avoid matrix inversions and, more importantly, we avoid having to multiply by the inverses in each iteration of the dynamic routing. This allows us to do many iterations of dynamic routing for the same computational cost as one forward propagation through the transformation matrices. In a standard mixture of Gaussians, the modifiable parameters are the means, (co)variances, and mixing proportions and the only thing that distinguishes different Gaussians is the values of these parameters. In a mixture of transforming Gaussians, however, Gaussians also differ in the transformation matrices they use. If these transformation matrices are fixed during the fitting of the other parameters, it makes sense to have a large set of transforming Gaussians available but to only use the small subset of them that have appropriate transformation matrices for explaining the data at hand. Fitting to a dataset will then involve deciding which of the transforming Gaussians should be "switched on". We therefore give each transforming Gaussian an additional activation parameter which is its probability of being switched on for the current dataset. The activation parameters are not mixing proportions because they do not sum to 1.To set the activation probability for a particular higher-level capsule, j, we compare the description lengths of two different ways of coding the poses of the activated lower-level capsules assigned to j by the routing, as described in section 3. "Description length" is just another term for energy. The difference in the two description lengths (in nats) is put through a logistic function to determine the activation probability of capsule j. The logistic function computes the distribution (p, 1 − p) that minimizes free energy when the difference in the energies of the two alternatives is the argument of the logistic function. The energies we use for determining the activation probabilities are the same energies as we use for fitting the Gaussians and computing the assignment probabilities. So all three steps minimize the same free energy but with respect to different parameters for each step. In some of the explanations above we have implicitly assumed that the lower-level capsules have activities of 1 or 0 and the assignment probabilities computed during the dynamic routing are also 1 or 0. In fact, these numbers are both probabilities and we use the product of these two probabilities as a multiplier on both the baseline description length of each lower-level mean and its alternative description length obtained by making use of the Gaussian fitted by a higher-level capsule. B SUPPLEMENTARY FIGURES Figure B.1: Sample smallNORB images at different viewpoints. All images in first row are at azimuth 0 and elevation 0. The second row shows a set of images at a higher-elevation and different azimuth. FIG2 the histograms are independently log scaled so that small and large counts can both be seen. Also, the considered distance range is 60 and the number of bins is much larger.

Capsule networks with learned pose matrices and EM routing improves state of the art classification on smallNORB, improves generalizability to new view points, and white box adversarial robustness.

Represent the TLDR for finding the corresponding scientific article

Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We study a general formulation of program synthesis called syntax-guided synthesis(SyGuS) that concerns synthesizing a program that follows a given grammar and satisfies a given logical specification. Both the logical specification and the grammar have complex structures and can vary from task to task, posing significant challenges for learning across different tasks. Furthermore, training data is often unavailable for domain specific synthesis tasks. To address these challenges, we propose a meta-learning framework that learns a transferable policy from only weak supervision. Our framework consists of three components: 1) an encoder, which embeds both the logical specification and grammar at the same time using a graph neural network; 2) a grammar adaptive policy network which enables learning a transferable policy; and 3) a reinforcement learning algorithm that jointly trains the embedding and adaptive policy. We evaluate the framework on 214 cryptographic circuit synthesis tasks. It solves 141 of them in the out-of-box solver setting, significantly outperforming a similar search-based approach but without learning, which solves only 31. The is comparable to two state-of-the-art classical synthesis engines, which solve 129 and 153 respectively. In the meta-solver setting, the framework can efficiently adapt to unseen tasks and achieves speedup ranging from 2x up to 100x. Program synthesis concerns automatically generating a program that satisfies desired functional requirements. Promising have been demonstrated by applying this approach to problems in diverse domains, such as spreadsheet data manipulation for end-users BID21, intelligent tutoring for students, and code auto-completion for programmers BID19, among many others. In a common formulation posed by BID3 called syntax-guided synthesis (SyGuS), the program synthesizer takes as input a logical formula φ and a grammar G, and produces as output a program in G that satisfies φ. In this formulation, φ constitutes a semantic specification that describes the desired functional requirements, and G is a syntactic specification that constrains the space of possible programs. The SyGuS formulation has been targeted by a variety of program synthesizers based on discrete techniques such as constraint solving BID36, enumerative search BID5, and stochastic search BID37. A key limitation of these synthesizers is that they do not bias their search towards likely programs. This in turn hinders their efficiency and limits the kinds of programs they are able to synthesize. It is well known that likely programs have predictable patterns BID23 BID1. As a , recent works have leveraged neural networks for program synthesis. However, they are limited in two aspects. First, they do not target general SyGuS tasks; more specifically:• They assume a fixed grammar (i.e., syntactic specification G) across tasks. For example, BID39 learn loop invariants for program verification, but the grammar of loop invariants is fixed across different programs to be verified. • The functional requirements (i.e., semantic specification φ) are omitted, in applications that concern identifying semantically similar programs BID34 BID0 BID2, or presumed to be input-output examples BID33 BID7 BID16 BID11 BID13 BID43 BID42 BID35.In contrast, the SyGuS formulation allows the grammar G to vary across tasks, thereby affording flexibility to enforce different syntactic requirements in each task. It also allows to specify functional requirements in a manner more general than input-output examples, by allowing the semantic specification φ to be a logical formula (e.g., f (x) = 2x instead of f = 2 ∧ f = 6). As a , the general SyGuS setting necessitates the ability to capture common patterns across different specifications and grammars. A second limitation of existing approaches is that they rely on strong supervision on the generated program BID33 BID7 BID11. However, in SyGuS tasks, ground truth programs f are not readily available; instead, a checker is provided that verifies whether f satisfies φ. In this paper, we propose a framework that is general in that it makes few assumptions on specific grammars or constraints, and has meta-learning capability that can be utilized in solving unseen tasks more efficiently. The key contributions we make are a joint graph representation of both syntactic and semantic constraints in each task that is learned by a graph neural network model; a grammar adaptive policy network that generalizes across different grammars and guides the search for the desired program; and a reinforcement learning training method that enables learning transferable representation and policy with weak supervision. We demonstrate our meta-learning framework on a challenging and practical instance of the SyGuS problem that concerns synthesizing cryptographic circuits that are provably free of side-channel attacks BID17. In our experiments, we first compare the framework in an out-of-box solver setting against a similar search-based approach and two state-of-the-art classical solvers developed in the formal methods community. Then we demonstrate its capability as a meta-solver that can efficiently adapt to unseen tasks, and compare it to the out-of-box version. The Syntax-Guided Synthesis (SyGuS) problem is to synthesize a function f that satisfies two kinds of constraints:• a syntactic constraint specified by a context-free grammar (CFG) G, and • a semantic constraint specified by a formula φ built from symbols in a theory T along with f.One example of the SyGuS problem is cryptographic circuit synthesis BID17. The goal is to synthesize a side-channel free cryptographic circuit by following the given CFG (syntactic constraint) while ensuring that the synthesized circuit is equivalent to the original circuit (semantic constraint). In this example, the grammar is designed to avoid side-channel attacks, whereas the original circuit is created only for functional correctness and thus is vulnerable to such attacks. We henceforth use this problem as an illustrative example but note that our proposed method is not limited to this specific SyGuS problem. We investigate how to efficiently synthesize the function f. Specifically, given a dataset of N tasks DISPLAYFORM0, we address the following two tasks:• learning an algorithm A θ: (φ, G) → f parameterized by θ that can find the function f i for (φ i, G i) ∈ D; • given a new task set D, adapt the above learned algorithm A θ and execute it on new tasks in D.This setting poses two difficulties in learning. First, the ground truth target function f is not readily available, making it difficult to formulate as a supervised learning problem. Second, the constraint φ is typically verified using an SAT or SMT solver, and this solver in turns expects the generated f to be complete. This means the weak supervision signal will only be given after the entire program is generated. Thus, it is natural to formulate A θ as a reinforcement learning algorithm. Since each instance (φ i, G i) ∈ D is an independent task with different syntactic and semantic constraints, the key to success is the design of such meta-learner, which we elaborate in Sec 3. This section presents our meta-solver model for solving the two problems formulated in Sec 2. We first introduce formal notation in Sec 3.1. To enable the transfer of knowledge across tasks with different syntactic and semantic constraints, we propose a representation framework in Sec 3.2 to jointly encode the two kinds of constraints. The representation needs to be general enough to encode constraints with different specifications. Lastly, we introduce the Grammar Adaptive Policy Network in Sec 3.3 that executes a program generation policy while automatically adapting to different grammars encoded in each task specification. We formally define key concepts in the SyGuS problem formulation as follows.semantic spec φ: The spec itself is a program written using some grammar. In our case, the grammar used in spec φ is different from the grammar G that specifies the syntax of the output program. However, in many practical cases the tokens (i.e., the dictionary of terminal symbols) may be shared across the input spec and the output program. DISPLAYFORM0 Here V denotes the nonterminal tokens, while Σ represents the terminal tokens. s is a special token that denotes the start of the language, and the language is generated according to the production rules defined in R. For a given non-terminal, the associated production rules can be written as α → β 1 |β 2... |β nα, where n α is the branching factor for non-terminal α ∈ V, and β i = u 1 u 2... u |βi| ∈ (V Σ) *. Each production rule α → β i ∈ R represents a way of expanding the grammar tree, by attaching nodes u 1, u 2,..., u |βi| to node α. The expansion is repeated until all the leaf nodes are terminals. Output function f: The output is a program in the language generated by G. A valid output f must satisfy both the syntactic constraints specified by G and the semantic constraints specified by φ. Different from traditional neural program synthesis tasks, where the program grammar and vocabulary is fixed, each individual task in our setting has its own form of grammar and semantic specification. Thus in the program generation phase (which we will elucidate in Sec 3.3), one cannot assume a fixed CFG and use a tree decoder like in BID26 and BID11. To enable such generalization across different grammars, the information about the CFG for each task needs to be captured in the task representation. Since the semantic spec program φ and the CFG G have rich structural information, it is natural to use graphs for their representation. Representing the programs using graphs has been successfully used in many programming language domains. In our work, we further extend the approach by BID2 with respect to the following aspects:• Instead of only representing the semantic spec program φ as a graph, we propose to jointly represent it along with the grammar G.• To allow information exchange between the two graphs, we leverage the idea of Static Single Assignment (SSA) form in compiler design. That is, the same variable (token) that may be assigned (defined) at many different places should be viewed differently, but on the other hand, these variations correspond to the same original thing. Specifically, we introduce global nodes for shared tokens and global links connecting these globally shared nodes and local nodes that (re)define corresponding tokens. The overall representation framework is described in FIG0. To construct the graph, we first build the abstract syntax tree (AST) for the semantic spec program φ, according to its own grammar (typically different from the output grammar G). To represent the grammar G, we associate each symbol in V Σ with a node representation. Furthermore, for a non-terminal α and its corresponding production rules α → β 1 |β 2... |β nα, we create additional nodes α i for each substitute β i. The purpose is to enable grammar adaptive program generation, which we elaborate in Sec 3.3. As a simplification, we merge all nodes α i representing β i that is a single terminal token into one node. Finally, the global nodes for shared tokens in Σ are created to link together the shared variable and operator nodes. This enables information exchange between the syntactic and semantics specifications. To encode the joint graph G(φ, G), we use graph neural networks to get the vector representation for each node in the graph. Specifically, for each node v ∈ G, we use the following parameterization for one step of message passing style update: DISPLAYFORM0 Lastly, {h DISPLAYFORM1 are the set of node embeddings. Here N (v) is the set of neighbor nodes of v, and e u,v denotes the type of edge that links the node u and v. We parameterize F in a way similar to GGNN BID28, i.e., F (h t, e) = σ(W e t h t) where we use different matrices W ∈ R d×d for different edge types and different propagation steps t. We sum over all the node embeddings to get the global graph embedding h(G).In addition to the node embeddings and global graph embedding, we also obtain the embedding matrix for each non-terminal node. Specifically, given node α, we will have the embedding matrix H α ∈ R nα×d, where the ith row of DISPLAYFORM2 α is the embedding of node α i that corresponds to substitution β i. This enables the grammar adaptive tree expansion in Sec 3.3. To enable the meta-solver to generalize across different tasks, both the task representation and program generation policy should be shared. We perform task conditional program generation for this purpose. Overall the generation is implemented using tree recursive generation, in the depth-first search (DFS) order. However, to handle different grammars specified in each task, we propose to use the grammar adaptive policy network. The key idea is to make the policy parameterized by decision embedding, rather than a fixed set of parameters. This mechanism is inspired by the pointer network BID44 and graph algorithm learning BID14.Specifically, suppose we are at the decision step t and try to expand the non-terminal node α t. For different tasks, the non-terminals may not be the same; furthermore, the number of ways to expand a certain non-terminal can also be different. As a , we cannot simply have a parameterized layer W h αt to calculate the logits of multinomial distribution. Rather, we use the embedding matrix H αt ∈ R nα t ×d to perform decision for this time step. This embedding matrix is obtained as described in Sec 3.2. DISPLAYFORM0 • Global graph embedding:• Embedding of production rules DISPLAYFORM0...... • Global graph embedding:• Embedding of production rules d1 -> d2 AND d2 Sampled production rule: DISPLAYFORM0 Figure 2: Generating solution using the grammar adaptive policy network. This figure shows one step of policy roll-out, which demonstrates how the same policy network handles different tasks with different grammar G 1 and G 2.Now we are able to build our policy network in an auto-regressive way. Specifically, the policy π(f |φ, G) can be parameterized as: DISPLAYFORM1 Here the probability of each action (in other words, each tree expansion decision) is defined as π(a t |h(G), DISPLAYFORM2, where s t ∈ R d is the context vector that captures the state of h(G) and T (t−1). In our implementation, s t is tracked by a LSTM decoder whose hidden state is updated by the embedding of the chosen action h αt. The initial state s 0 is obtained by passing graph embedding h(G) through a dense layer with matching size. In this section, we present a reinforcement learning framework for the meta-solver. Formally, let θ denote the parameters of graph embedding and adaptive policy network. For a given pair of instances (φ, G), we learn a policy π θ (f |φ, G) parameterized by θ that generates f such that φ ≡ f.Reward design: The RL episode starts by accepting the representation of tuple φ, G as initial observation. During the episode, the model executes a sequence of actions to expand non-terminals in f, and finishes the episode when f is complete. Upon finishing, the SAT solver is invoked and will return a binary flag indicating whether f satisfies φ or not. An obvious reward design would be directly using the binary value as the episode return. However, this leads to a high variance in returns as well as a highly non-smooth loss surface. Here, we propose to smooth the reward as follows: for each specification φ we maintain a test case buffer B φ that stores all input examples observed so far. Each time the SAT solver is invoked for φ, if f passes then a full reward of 1 is given, otherwise the solver will generate a counter-example b besides the binary flag. We then sample interpolated examples around b which we denote the set asB b. Then the reward is given as the fractions of examples in B φ andB b where f has the equivalent output as φ r = DISPLAYFORM0 At the end of the episode, the buffer is updated as B φ ← B φ ∪B b for next time usage. In the extreme case where all inputs can be enumerated, e.g. binary or discrete values, it reduces to computing the fraction of passed examples over the entire test case set. This is implemented in our experiment on the cryptographic circuit synthesis task. In the meta-learning setting, the framework learns to represent a set of different programs and navigate the generation process under different constraints. We utilize the Advantage Actor-Critic (A2C) for model training. Given a training set D, a minibatch of instances are sampled from D for each epoch. For each instance φ i, G i, the model performs a complete rollout using policy π θ (f |φ i, G i).The actor-critic method computes the gradients w.r.t to θ of each instance as DISPLAYFORM0 where γ denotes the discounting factor and V (s t ; ω) is a state value estimator parameterized by ω. In our implementation, this is modeled as a standard MLP with scalar output. It is learned to fit the expected return, i.e., min ω E |f | t=1 γ t r − V (s t ; ω). Gradients obtained from each instance are averaged over the minibatch before applying to the parameter. Figure 3: An example of a circuit synthesis task from the 2017 SyGuS competition. Given the original program specification which is represented as an abstract syntax tree (left), the solver is tasked to synthesize a new circuit f (right). The synthesis process is specified by the syntactic constraint G (top), and the semantic constraint (bottom) specifies that f must have functionality equivalent to the original program. We evaluate the our framework 1 on cryptographic circuit synthesis tasks BID17 which constitute a challenging benchmark suite from the general track of the. The dataset contains 214 tasks, each of which is a pair of logical specification, describing the correct functionality, and a context free grammar, describing the timing constraints for input signals. The goal is to find an equivalent logical expression which is required to follow the given context free grammar in order to avoid potential timing channel vulnerabilities. Figure 3 shows an illustrative example. Each synthesis task has a different logical specification as well as timing constraints, and both the logical specification and context free grammar varies from task to task, posing a significant challenge in representation learning. As a , this suite of tasks serves as an ideal testbed for our learning framework and its capability to generalize to unseen specifications and grammars. The experiments are conducted in two learning settings. First, we test our framework as an outof-box solver, which means the training set D and testing set D are the same and contain only one instance. In other words, the framework is tasked to solve only one instance at a time. This test-on-train setting serves to investigate the capacity of our framework in representation and policy learning, as the model can arbitrarily "exploit" the problem without worrying about overfitting. This setting also enables us to compare our framework to classical solvers developed in the formal methods community. As those solvers do not utilize learning-based strategies, it is sensible to also limit our framework not to carry over prior knowledge from a separate training set. Second, we evaluate the model as a meta-solver which is trained over a training set D, and finetuned on each of the new tasks in a separate set D. In this setting, we aim to demonstrate that our Table 1: Number of instances solved using: 1) EUSolver, 2) CVC4, 3) ESymbolic, and 4) Out-ofBox Solver. For each solver, the maximum time in solving an instance and the average and median time over all solved instances are also shown below. framework is capable of learning a transferable representation and policy in order to efficiently adapt to unseen tasks. In the out-of-box solver setting, we compare our framework against solvers built based on two classical approaches: a SAT/SMT constraint solving based approach and a search based approach. For the former, we choose CVC4 BID36, which is the state-of-the-art SMT constraint solver; for the latter, we choose EUSolver BID5, which is the winner of the SyGuS 2017 Competition BID4. Furthermore, we build a search based solver as baseline, ESymbolic, which systematically expands non-terminals in a predefined order (e.g. depth-first-search) and effectively prunes away partially generated candidates by reducing it to 2QBF BID6 satisfiability check. ESymbolic can be viewed as a generalization of EUSolver by replacing the carefully designed domain-specific heuristics (e.g. indistinguishability and unification) with 2QBF.In order to make the comparison fair, we run all solvers on the same platform with a single core CPU available 2, even though our framework could take advantage of hardware accelerations, for instance, via GPUs and TPUs. We measure the performance of each solver by counting the number of instances it can solve given a 6 hours limit spent on each task. It is worth noting that comparing running time only gives a limited view of the solvers' performance. Although the hardware is the same, the software implementation can make many differences. For instance, CVC4 is carefully redesigned and re-implemented in C++ as the successor of CVC3 , which has been actively improved for more than a decade. To our best knowledge, the design and implementation of EUSolver is directly guided by and heavily tuned according to SyGuS benchmarks. In contrast, our framework is a proof-of-concept prototype implemented in Python and has not yet been tuned for running time performance. In Table 1, we summarize the total number of instances solved by each solver as well as the maximum, average and median running time spent on solved instances. In terms of the absolute number of solved instances, our framework is not yet as good as EUSolver, which is equipped with specialized heuristics. However, EUSolver fails to solve 4 instances that are only solved by our framework. All instances solved by CVC4 and ESymbolic are a strict subset of instances solved by EUSolver. Thus, besides being a new promising approach, our framework already plays a supplementary role for improving the current state-of-the-art. Compared with the state-of-the-art CVC4 solver, our framework has smaller maximum time but higher average and median time usage. This suggests that our framework excels at solving difficult instances with better efficiency. This observation is further confirmed in FIG3, where we plot the time usage along with the number of instances solved. This suggests that canonical solvers such as CVC4 are efficient in solving simple instances, but have inferior scalability compared to our dynamically adapted approach when the problem becomes more difficult, where we can see a steeper increase in time usages by CVC4 in solving 110 and more instances. Though EUSolver has superior scalability, it is achieved by a number of heuristics that are manually designed and iteratively improved by experts with the same benchmark on hand. In contrast, our framework learns a policy to solve hard instances from scratch on the fly without requiring training data at all. We next evaluate whether our framework is capable of learning transferable knowledge across different synthesis tasks. We randomly split the 214 circuits synthesis tasks into two sets: 150 tasks for training and the rest 64 tasks for testing. The meta-solver is then trained on the training set for 35000 epochs using methods introduced in Sec 4.1. For each epoch, a batch of 10 tasks are sampled. The gradients of each task are averaged and applied to the model parameters using Adam optimizer. In testing phase, the trained meta-solver is finetuned on each task in the testing set until either a correct program is synthesized or timeout occurs. This process is similar to the setting in Sec 5.1 but with smaller learning rate and exploration. We compare the trained meta-solver with the out-of-box solver in solving tasks in the test set. Out of 64 testing tasks, the out-of-box solver and meta-solver can solve 36 and 37 tasks, respectively. Besides the additional task solved, the performance is also greatly improved by meta-solver, which is shown in FIG4. Table 5 (a) shows the accumulated number of candidates generated to successfully solve various ratios of testing tasks. We see that the number of explored candidates by meta-solver is significantly reduced: for 40% of testing tasks (i.e., 66% of solved tasks), meta-learning enable 4x reduction on average. The accumulated reduction for all solved tasks (60% of testing tasks) is not that significant. This is because meta-learning improve dramatically for most (relatively) easy tasks but helps slightly for a few hard tasks, which actually dominate the number of generated candidates. FIG4 (b) shows the speedup distribution over the 36 commonly solved tasks. Meta-solver achieves at least 2x speedup for most benchmarks, orders of magnitude improvement for 10 out of 36 unseen tasks, and solves one task that is not solvable without meta-learning. We survey work on symbolic program synthesis, neural program synthesis, and neural induction. Symbolic program synthesis. Automatically synthesizing a program from its specification was first posed by BID30. It received renewed attention with advances in SAT and SMT solvers BID41 and found application in problems in various domains as surveyed by BID22. In this context, SyGuS BID3 was proposed as a common format to express these problems. Several implementations of SyGuS solvers exist, including by constraint solving BID36, divide-and-conquer BID5, and stochastic MCMC search BID37, in addition to various domain-specific algorithms. A number of probabilistic techniques have been proposed to accelerate these solvers by modeling syntactic aspects of programs. These include PHOG BID27, log-bilinear tree-traversal models BID29, and graph-based statistical models.Neural program synthesis. Several recent works have used neural networks to accelerate the discovery of desired programs. These include DeepCoder BID7, Bayou BID31, RobustFill BID16, Differentiable FORTH BID10, neurosymbolic program synthesis BID33 BID11, neural-guided deductive search BID43, learning context-free parsers BID13, and learning program invariants BID39. The syntactic specification in these approaches is fixed by defining a domain-specific language upfront. Also, with the exception of BID39, the semantic specification takes the form of input-output examples. Broadly, these works have difficulty with symbolic constraints, and are primarily concerned with avoiding overfitting, coping with few examples, and tolerating noisy examples. Our work relaxes both these kinds of specifications to target the general SyGuS formulation. Recently BID18 propose gradually bootstrapping domain-specific languages for neurally-guided Bayesian program learning, while our work concerns learning programs that use similar grammars, which may or may not be incremental. Neural program induction. Another body of work includes techniques in which the neural network is itself the computational substrate. These include neural Turing machines BID20 ) that can learn simple copying/sorting programs, the neural RAM model BID25 to learn pointer manipulation and dereferencing, the neural GPU model BID24 to learn complex operations like binary multiplication, and BID12's work to incorporate recursion. These approaches have fundamental problems regarding verifying and interpreting the output of neural networks. In contrast, we propose tightly integrating a neural learner with a symbolic verifier so that we obtain the scalability and flexibility of neural learning and the correctness guarantees of symbolic verifiers. We proposed a framework to learn a transferable representation and strategy in solving a general formulation of program synthesis, i.e. syntax-guided synthesis (SyGuS). Compared to previous work on neural synthesis, our framework is capable of handling tasks where 1) the grammar and semantic specification varies from task to task, and 2) the supervision is weak. Specifically, we introduced a graph neural network that can learn a joint representation over different pairs of syntactic and semantic specifications; we implemented a grammar adaptive network that enables program generation to be conditioned on the specific task; and finally, we proposed a meta-learning method based on the Advantage Actor-Critic (A2C) framework. We compared our framework empirically against one baseline following a similar search fashion and two classical synthesis engines. Under the outof-box solver setting with limited computational resources and without any prior knowledge from training, our framework is able to solve 141 of 214 tasks, significantly outperforming the baseline ESymbolic by 110. In terms of the absolute number of solved tasks, the performance is comparable to two state-of-the-art solvers, CVC4 and EUSolver, which solve 129 and 153 respectively. However, the two state-of-the-art solvers failed on 4 tasks solved by our framework. When trained as a meta-solver, our framework is capable of accelerating the solving process by 2× to 100×.

Represent the short summary for finding the corresponding scientific article (long)

We propose a meta-learning framework that learns a transferable policy from only weak supervision to solve synthesis tasks with different logical specifications and grammars.

represent the tldr for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence. Recent approach for this task either apply a fixed policy on transformer, or a learnable monotonic attention on a weaker recurrent neural network based structure. In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which introduced the monotonic attention mechanism to multihead attention. We also introduced two novel interpretable approaches for latency control that are specifically designed for multiple attentions. We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach. Code will be released upon publication. Simultaneous machine translation adds the capability of a live interpreter to machine translation: a simultaneous machine translation model starts generating a translation before it has finished reading the entire source sentence. Such models are useful in any situation where translation needs to be done in real time. For example, simultaneous models can translate live video captions or facilitate conversations between people speaking different languages. In a usual neural machine translation model, the encoder first reads the entire sentence, and then the decoder writes the target sentence. On the other hand, a simultaneous neural machine translation model alternates between reading the input and writing the output using either a fixed or learned policy. Monotonic attention mechanisms fall into the learned policy category. Recent work exploring monotonic attention variants for simultaneous translation include: hard monotonic attention , monotonic chunkwise attention (MoChA) and monotonic infinite lookback attention (MILk) . MILk in particular has shown better quality / latency trade-offs than fixed policy approaches, such as wait-k or wait-if-* policies. MILk also outperforms hard monotonic attention and MoChA; while the other two monotonic attention mechanisms only consider a fixed reading window, MILk computes a softmax attention over all previous encoder states, which may be the key to its improved latencyquality tradeoffs. These monotonic attention approaches also provide a closed form expression for the expected alignment between source and target tokens. However, monotonic attention-based models, including the state-of-the-art MILk, were built on top of RNN-based models. RNN-based models have been outperformed by the recent state-of-the-art Transformer model , which features multiple encoder-decoder attention layers and multihead attention at each layer. We thus propose monotonic multihead attention (MMA), which combines the strengths of multilayer multihead attention and monotonic attention. We propose two variants, Hard MMA (MMA-H) and Infinite Lookback MMA (MMA-IL). MMA-H is designed with streaming systems in mind where the attention span must be limited. MMA-IL emphasizes the quality of the translation system. We also propose two novel latency regularization methods. The first encourages the model to be faster by directly minimizing the average latency. The second encourages the attention heads to maintain similar positions, preventing the latency from being dominated by a single or a few heads. The main contributions of this paper are: A novel monotonic attention mechanism, monotonic multihead attention, which enables the Transformer model to perform online decoding. This model leverages the power of the Transformer and the efficiency of monotonic attention. Better latencyquality tradeoffs compared to the MILk model, the previous state-of-the-art, on two standard translation benchmarks, IWSLT15 English-Vietnamese (En-Vi) and WMT15 German-English (De-En). Analyses on how our model is able to control the attention span and on the relationship between the speed of a head and the layer it belongs to. We motivate the design of our model with an ablation study on the number of decoder layers and the number of decoder heads. In this section, we review the monotonic attention-based approaches in RNN-based encoder-decoder models. We then introduce the two types of Monotonic Multihead Attention (MMA) for Transformer models: MMA-H and MMA-IL. Finally, we introduce strategies to control latency and coverage. The hard monotonic attention mechanism was first introduced in order to achieve online linear time decoding for RNN-based encoder-decoder models. We denote the input sequence as x = {x 1, ..., x T}, and the corresponding encoder states as m = {m 1, ..., m T}, with T being the length of the source sequence. The model generates a target sequence y = {y 1, ..., y U} with U being the length of the target sequence. At the i-th decoding step, the decoder only attends to one encoder state m ti with t i = j. When generating a new target token y i, the decoder chooses whether to move one step forward or to stay at the current position based on a Bernoulli selection probability p i,j, so that t i ≥ t i−1. Denoting the decoder state at the i-th position, starting from j = t i−1, t i−1 + 1, t i−1 + 2,..., this process can be calculated as follows: When z i,j = 1, we set t i = j and start generating a target token y i; otherwise, we set t i = j + 1 and repeat the process. During training, an expected alignment α is introduced to replace the softmax attention. It can be calculated in a recurrent manner, shown in Equation 4: also introduce a closed-form parallel solution for the recurrence relation in Equation 5: where cumprod(x) = [1, x 1, x 1 x 2, ..., In practice, the denominator in Equation 5 is clamped into a range of [, 1] to avoid numerical instabilities introduced by cumprod. Although this monotonic attention mechanism achieves online linear time decoding, the decoder can only attend to one encoder state. This limitation can diminish translation quality as there may be insufficient information for reordering. Moreover, the model lacks a mechanism to adjust latency based on different requirements at decoding time. To address these issues, introduce Monotonic Chunkwise Attention (MoChA), which allows the decoder to apply softmax attention to a fixed-length subsequence of encoder states. introduce Monotonic Infinite Lookback Attention (MILk) which allows the decoder to access encoder states from the beginning of the source sequence. The expected attention for the MILk model is defined in Equation 6. 2.2 MONOTONIC MULTIHEAD ATTENTION Previous monotonic attention approaches are based on RNN encoder-decoder models with a single attention and haven't explored the power of the Transformer model. 2 The Transformer architecture has recently become the state-of-the-art for machine translation . An important feature of the Transformer is the use of a separate multihead attention module at each layer. Thus, we propose a new approach, Monotonic Multihead Attention (MMA), which combines the expressive power of multihead attention and the low latency of monotonic attention. Multihead attention allows each decoder layer to have multiple heads, where each head can compute a different attention distribution. Given queries Q, keys K and values V, multihead attention The attention function is the scaled dot-product attention, defined in Equation 8: There are three applications of multihead attention in the Transformer model: 1. The Encoder contains self-attention layers where all of the queries, keys and values come from previous layers. 2. The Decoder contains self-attention layers that allow each position in the decoder to attend to all positions in the decoder up to and including that position. 3. The Encoder-Decoder attention contains multihead attention layers where queries come from the previous decoder layer and the keys and values come from the output of the encoder. Every decoder layer has a separate encoder-decoder attention. For MMA, we assign each head to operate as a separate monotonic attention in encoder-decoder attention. For a transformer with L decoder layers and H attention heads per layer, we define the selection process of the h-th head encoder-decoder attention in the l-th decoder layer as where W l,h is the input projection matrix, d k is the dimension of the attention head. We make the selection process independent for each head in each layer. We then investigate two types of MMA, MMA-H(ard) and MMA-IL(infinite lookback). For MMA-H, we use Equation 4 in order to calculate the expected alignment for each layer each head, given p l,h i,j. For MMA-IL, we calculate the softmax energy for each head as follows: and then use Equation 6 to calculate the expected attention. Each attention head in MMA-H hardattends to one encoder state. On the other hand, each attention head in MMA-IL can attend to all previous encoder states. Thus, MMA-IL allows the model to leverage more information for translation, but MMA-H may be better suited for streaming systems with stricter efficiency requirements. Finally, our models use unidirectional encoders: the encoder self-attention can only attend to previous states, which is also required for simultaneous translation. At inference time, our decoding strategy is shown in Algorithm 1. For each l, h, at decoding step i, we apply the sampling processes discussed in subsection 2.1 individually and set the encoder step at t l,h i. Then a hard alignment or partial softmax attention from encoder states, shown in Equation 13, will be retrieved to feed into the decoder to generate the i-th token. The model will write a new target token only after all the attentions have decided to write. In other words, the heads that have decided to write must wait until the others have finished reading. Figure 1 illustrates a comparison between our model and the monotonic model with one attention head. Compared with the monotonic model, the MMA model is able to set attention to different positions so that it can still attend to previous states while reading each new token. Each head can adjust its speed on-the-fly. Some heads read new inputs, while the others can stay in the past to retain the source history information. Even with the hard alignment variant (MMA-H), the model is still able to preserve the history information by setting heads to past states. In contrast, the hard monotonic model, which only has one head, loses the previous information at the attention layer. Effective simultaneous machine translation must balance quality and latency. At a high level, latency measures how many source tokens the model has to read until a translation is generated. The model we have introduced in subsection 2.2 is not able to control latency on its own. While MMA allows simultaneous translation by having a read or write schedule for each head, the overall latency is determined by the fastest head, i.e. the head that reads the most. It is possible that a head always reads new input without producing output, which would in the maximum possible latency. Note that the attention behaviors in MMA-H and MMA-IL can be different. In MMA-IL, a head reaching the end of the sentence will provide the model with maximum information about the source sentence. On the other hand, in the case of MMA-H, reaching the end of sentence for a head only gives a hard alignment to the end-of-sentence token, which provides very little information to the decoder. Furthermore, it is possible that an MMA-H attention head stays at the beginning of sentence Algorithm 1 MMA monotonic decoding. Because each head is independent, we compute line 3 to 16 in parallel Input: x = source tokens, h = encoder states, i = 1, j = 1, t l,h 0 = 1, y 0 = StartOfSequence. 1: while y i−1 = EndOfSequence do 2: Break 12: Read token x j Calculate state h j and append to h 16: 19: 20: without moving forward. Such a head would not cause latency issues but would degrade the model quality since the decoder would not have any information about the input. In addition, this behavior is not suited for streaming systems. To address these issues, we introduce two latency control methods. The first one is weighted average latency, shown in Equation 14: where g l,h i = |x| j=1 jα i,j. Then we calculate the latency loss with a differentiable latency metric C. , we use the Differentiable Average Lagging. It is important to notice that, unlike the original latency augmented training in , Equation 15 is not the expected latency metric given C, but weighted average C on all the attentions. The real expected latency isĝ = max l,h g l,h instead ofḡ, but using this directly would only affect the speed of the fastest head. Equation 15, however, can control every head in a way that the faster heads will be automatically assigned to larger weights and slower heads will also be moderately regularized. heads from getting faster. However, for MMA-H models, we found that the latency of are mainly due to outliers that skip almost every token. The weighted average latency loss is not sufficient to control the outliers. We therefore introduce the head divergence loss, the average variance of expected delays at each step, defined in Equation 16:: where λ avg, λ var are hyperparameters that control both losses. Intuitively, while λ avg controls the overall speed, λ var controls the divergence of the heads. Combining these two losses, we are able to dynamically control the range of attention heads so that we can control the latency and the reading buffer. For MMA-IL model, we only use L avg; for MMA-H we only use L var. We evaluate our model using quality and latency. For translation quality, we use tokenized BLEU 3 for IWSLT15 En-Vi and detokenized BLEU with SacreBLEU for WMT15 De-En. For latency, we use three different recent metrics, Average Proportion (AP) , Average Lagging (AL) and Differentiable Average Lagging (DAL) 4. We remind the reader of the metric definitions in Appendix A.2. Table 3: Effect of using a unidirectional encoder and greedy decoding to BLEU score. We evaluate our method on two standard machine translation datasets, IWSLT14 En-Vi and WMT15 De-En. Statistics of the datasets can be found in Table 1. For each dataset, we apply tokenization with the Moses tokenizer and preserve casing. IWSLT15 English-Vietnamese TED talks from IWSLT 2015 Evaluation Campaign . We follow the same settings from and. We replace words with frequency less than 5 by <unk>. We use tst2012 as a validation set tst2013 as a test set. WMT15 German-English We follow the setting from. We apply byte pair encoding (BPE) jointly on the source and target to construct a shared vocabulary with 32K symbols. We use newstest2013 as validation set and newstest2015 as test set. We evaluate MMA-H and MMA-IL models on both datasets. The MILK model we evaluate on IWSLT15 En-Vi is based on rather than RNMT+ . In general, our offline models use unidirectional encoders, i.e. the encoder self-attention can only attend to previous states, and greedy decoding. We report offline model performance in Table 2 and the effect of using unidirectional encoders and greedy decoding in Table 3. For MMA models, we replace the encoder-decoder layers with MMA and keep other hyperparameter settings the same as the 3 We acquire the data from https://nlp.stanford.edu/projects/nmt/, which is tokenized. We do not have the tokenizer which processed this data, thus we report tokenized BLEU for IWSLT15 4 Latency metrics are computed on BPE tokens for WMT15 De-En -consistent with -and on word tokens for IWSLT15 En-Vi. 5 report a BLEU score of 23.0 but they didn't mention what type of BLEU score they used. This score is from our implementation on the data aquired from https://nlp.stanford.edu/projects/nmt/ offline model. Detailed hyperparameter settings can be found in subsection A.1. We use the Fairseq library 6 for our implementation. Code will be released upon publication. In this section, we present the main of our model in terms of latency-quality tradeoffs, ablation studies and analyses. In the first study, we analyze the effect of the variance loss on the attention span. Then, we study the effect of the number of decoder layers and decoder heads on quality and latency. We also provide a case study for the behavior of attention heads in an example. Finally, we study the relationship between the rank of an attention head and the layer it belongs to. We plot the quality-latency curves for MMA-H and MMA-IL in Figure 2. The BLEU and latency scores on the test sets are generated by setting a latency range and selecting the checkpoint with best BLEU score on the validation set. We use differentiable average lagging when setting the latency range. We find that for a given latency, our models obtain a better translation quality. While MMA-IL tends to have a decrease in quality as the latency decreases, MMA-H has a small gain in quality as latency decreases. The reason is that a larger latency does not necessarily mean an increase in source information available to the model. In fact, the large latency is from the outlier attention heads, which skip the entire source sentence and point to the end of the source sentence. The outliers not only increase the latency but they also do not provide useful information. We introduce the attention variance loss to eliminate the outliers, as such a loss makes the attention heads focus on the current context for translating the new target token. It is interesting to observe that even MMA-H has a better latency-quality tradeoff than MILk 7 even though each head only attends to only one state. Although MMA-H is not yet able to handle an arbitrarily long input (without resorting to segmenting the input), since both the encoder and decoder self-attention have an infinite lookback, that model represents a good step in that direction. In subsection 2.3, we introduced the attention variance loss to MMA-H in order to prevent outlier attention heads from increasing the latency or increasing the attention span. We have already evaluated the effectiveness of this method on latency in subsection 4.1. We also want to measure the difference between the fastest and slowest heads at each decoding step. We define the average attention span in Equation 18:S It estimates the reading buffer we need for streaming translation. We show the relation between the average attention span (averaged over the IWSLT and WMT test sets) versus L var in Figure 3. As expected, the average attention span is reduced as we increase L var. One motivation to introduce MMA is to adapt the Transformer, which is the current state-of-the-art model for machine translation, to online decoding. Important features of the Transformer architecture include having a separate attention layer for each decoder layer block and multihead attention. In this section, we test the effect of these two components on the offline, MMA-H, and MMA-IL models from a quality and latency perspective. We report quality as measured by detokenized BLEU and latency as measured by DAL on the WMT13 validation set in Figure 4. We set λ avg = 0.2 for MMA-IL and λ var = 0.2 for MMA-H. The offline model benefits from having more than one decoder layer. In the case of 1 decoder layer, increasing the number of attention heads is beneficial but in the case of 3 and 6 decoder layers, we do not see much benefit from using more than 2 heads. The best performance is obtained for 3 layers and 2 heads (6 effective heads). The MMA-IL model behaves similarly to the offline model, and the best performance is observed with 6 layers and 4 heads (24 effective heads). For MMA-H, with 1 layer, performance improves with more heads. With 3 layers, the single-head setting is the most effective (3 effective heads). Finally, with 6 layers, the best performance is reached with 16 heads (96 effective heads). The general trend we observe is that performance improves as we increase the number of effective heads, either from multiple layers or multihead attention, up to a certain point, then either plateaus or degrades. This motivates the introduction of the MMA model. We also note that latency increases with the number of effective attention heads. This is due to having fixed loss weights: when more heads are involved, we should increase λ var or λ avg to better control latency. We characterize attention behaviors by providing a running example of MMA-H and MMA-IL, shown in Figure 5. Each curve represents the path that an attention head goes through at inference time. For MMA-H, shown in Figure 5a, we found that when the source and target tokens have the same order, the attention heads behave linearly and the distance between fastest head and slowest head is small. For example, this can be observed from partial sentence pair "I also didn't know that" and target tokens "Tôi cũng không biết rằng", which have the same order. However, when the source tokens and target tokens have different orders, such as "the second step" and "bước (step) thứ hai (second)", the model will generate "bước (step)" first and some heads will stay in the past to retain the information for later reordered translation "thứ hai (second)". We can also see that the attention heads have a near-diagonal trajectory, which is appropriate for streaming inputs. The behaviors of the heads in MMA-IL models are shown in Figure 5b. Notice that we remove the partial softmax alignment in this figure. Because we don't expect streaming capability for MMA-IL, some heads stop at early position of the source sentence to retain the history information. Moreover, because MMA-IL has more information when generating a new target token, it tends to produce better translation quality. In this example, the MMA-IL model has better translation on "isolate the victim" than MMA-H ("là cô lập nạn nhân" vs "là tách biệt nạn nhân") In Figure 6, we calculate the average and standard deviation of rank of each head when generating every target token. For MMA-IL, we find that heads in lower layers tend to have higher rank and are thus slower. However, in MMA-H, the difference of the average rank are smaller. Furthermore, the standard deviation is very large which means that the order of the heads in MMA-H changes frequently over the inference process. Recent work on simultaneous machine translation falls into three categories. In the first one, models use a rule-based policy for reading input and writing output. propose a WaitIf-* policy to enable an offline model to decode simultaneously. propose a wait-k policy where the model first reads k tokens, then alternates between read and write actions. propose an incremental decoding method, also based on a rule-based schedule. In the second category, a flexible policy is learnt from data. introduce a Markov chain to phrase-based machine translation models for simultaneous machine translation, in which they apply reinforcement learning to learn the read-write policy based on states. introduce an agent which learns to make decisions on when to translate from the interaction with a pre-trained offline neural machine translation model. used continuous rewards policy gradient for online alignments for speech recognition. proposed a hard alignment with variational inference for online decoding. propose a new operation "predict" which predicts future source tokens. Zheng et al. (2019b) introduce a restricted dynamic oracle and restricted imitation learning for simultaneous translation. Zheng et al. (2019a) train the agent with an action sequence from labels that are generated based on the rank of the gold target word given partial input. Models from the last category leverage monotonic attention and replace the softmax attention with an expected attention calculated from a stepwise Bernoulli selection probability. first introduce the concept of monotonic attention for online linear time decoding, where the attention only attends to one encoder state at a time. extended that work to let the model attend to a chunk of encoder state. also make use of the monotonic attention but introduce an infinite lookback to improve the translation quality. In this paper, we propose two variants of the monotonic multihead attention model for simultaneous machine translation. By introducing two new targeted loss terms which allow us to control both latency and attention span, we are able to leverage the power of the Transformer architecture to achieve better quality-latency trade-offs than the previous state-of-the-art model. We also present detailed ablation studies demonstrating the efficacy and rationale of our approach. By introducing these stronger simultaneous sequence-to-sequence models, we hope to facilitate important applications, such as high-quality real-time interpretation between human speakers. Average Proportion 1 |x||y| We provide the detailed in Figure 2 as Table 6 and Table 7. We explore a simple method that can adjust system's latency at inference time without training new models. In Algorithm 1 line 8, 0.5 was used as an threshold. One can set different threshold p during the inference time to control the latency. We run the pilot experiments on IWSLT15 En-Vi dataset and the are shown as We explore applying a simple average instead of a weighted average loss to MMA-H. The are shown in Figure 7 and Table 9. We find that even with very large weights, we are unable to reduce the overall latency. In addition, we find that the weighted average loss severely affects the translation quality negatively. On the other hand, the divergence loss we propose in Equation 16 can efficiently reduce the latency while retaining relatively good translation quality for MMA-H models.

Make the transformer streamable with monotonic attention.

Represent the short summary for finding the corresponding scientific article (long)

The paper introduces a method of training generative recurrent networks that helps to plan ahead. We run a second RNN in a reverse direction and make a soft constraint between cotemporal forward and backward states.