Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Estimating the location where an image was taken based solely on the contents of the image is a challenging task, even for humans, as properly labeling an image in such a fashion relies heavily on contextual information, and is not as simple as identifying a single object in the image. Thus any methods which attempt to do so must somehow account for these complexities, and no single model to date is completely capable of addressing all challenges. This work contributes to the state of research in image geolocation inferencing by introducing a novel global meshing strategy, outlining a variety of training procedures to overcome the considerable data limitations when training these models, and demonstrating how incorporating additional information can be used to improve the overall performance of a geolocation inference model. In this work, it is shown that Delaunay triangles are an effective type of mesh for geolocation in relatively low volume scenarios when compared to from state of the art models which use quad trees and an order of magnitude more training data. In addition, the time of posting, learned user albuming, and other meta data are easily incorporated to improve geolocation by up to 11% for country-level (750 km) locality accuracy to 3% for city-level (25 km) localities. Advancements in deep learning over the past several years have pushed the bounds of what is possible with machine learning beyond typical image classification or object localization. Whereas prior work in computer vision and machine learning has focused primarily on determining the contents of the image-"is there a dog in this photo?"-more recent methods and more complicated models have enabled deeper examinations of the contextual information behind an image. This deeper exploration allows a researcher to ask more challenging questions of its models. One such question is, "where in the world was this picture taken?" However, estimating the geographic origin of a ground-level image is a challenging task due to a number of different factors. Firstly, the volume of available images with geographic information associated with them is not evenly distributed across the globe. This uneven distribution increases the complexity of the model chosen and complicates the design of the model itself. Furthermore, there are additional challenges associated with geo-tagged image data, such as the potential for conflicting data (eg. incorrect geolabels and replica landmarks -St Peters Basilica in Nikko, Japan) and the ambiguity of geographic terms (eg. imagery of similar and ambiguous geological features, such as beaches, screenshots of websites, or generally, plates of food).The work presented herein focuses on the task of content-based image geolocation: the process of identify the geographic origin of an image taken at ground level. Given the significant growth in the production of data, and trends in social data to move away from pure text-based platforms to more images, videos, and mixed-media, tackling the question of inferring geographic context behind an image becomes a significant and relevant problem. Whereas social textual data may be enriched with geolocation information, images may or may not contain such information; EXIF data is often stripped from images or may be present but incomplete. Within the context of mixed-media data, using the geolocation information associated with a linked object may be unreliable or not even applicable. For example, a user on Twitter may have geolocation information enabled for their account, but may post a link to an image taken from a location different from the one where they made their post. A wide variety of approaches have been considered for geolocation from image content, please consult Brejcha &Čadík for a review. The approach taken in this paper builds on recent work in global geolocation from ground-based imagery BID19 for global scale geoinferencing. BID19 utilize a multi-class approach with a one-hot encoding. A common approach, as well, is to perform instance-level scene retrieval in order to perform geolocation of imagery BID6 BID18. These works query on previously geotagged imagery and assign a geolabel based on the similarity of the image query to the database. Further, BID18 builds on work by BID6; BID19 by utilizing the feature maps for the mesh-based classifier for the features of their nearest-neighbors scene retrieval approach. Prior work exists for data sampling strategies for large-scale classification problems for social media applications. considers weighted sampling of minority classes. In this work, the class selection is biased during training of the deep learning models. Biasing class selection with random noising (often called image regularization) is a well known way to allow a model to see more examples of rare classes, but also there are additional concerns related specifically to social media applications. For example, researchers in consider the case of sampling such that an individual user is only seen once per a training epoch. In this work, sampling is performed without respect to user, so that images are selected in epochs completely at random, but the broader influence of latent variables, other than user, and communities is of concern in social media geolocation. The first type of model considered in this work is purely geolocation for image content (M1). The use of time information forms a second set of models within this work (M2). User-album inputs form a third set of models within this work (M3). Our work contributes meaningful consideration of the use of an alternative mesh for geolocation in M1 and demonstrates the meaningful use of time and user information to improve geolocation (M2 and M3). Collected data holdings are derived from YFCC100M BID17, where training and validation was performed on a randomly selected 14.9M (12.2M training/2.7M validation) of the 48.4M geolabeled imagery. PlaNet, in comparison, used 125M (91M training/34M validation) images BID19 for model developemt. In using the data, it is assumed that the ground truth GPS location is exactly accurate, which is a reasonable approximation based on the of BID5.Every YFCC100M image has associated meta data. Importantly, these data contain user-id and posted-time. User-id is a unique token that groups images by account. Posted time is the time that the user uploaded data to the website that ultimately created the combined storage of all images. The most likely true GPS location for an image from varies by the time the image is uploaded to a website, as shown in Figure 1. The approach described in this work for the global-scale geolocation model is similar to the classification approach taken by PlaNet BID19. Spatially the globe is subdivied into a grid, forming a mesh, decribed in Section 3.1.1, of classification regions. An image classifier is to trained Figure 1: True Longitude versus Time of Posting. In these boxplots, we limit users to one image selected at random to decouple any user effects from time of day. What this image implies is that the prior distribution for image longitude rdiffers by the time of day that an image is posted. As an exaple, consider longitudes near 0. It would take less evidence to predict a longitude close to zero for an image posted around 01:00 UTC tahn to make a similar preidction at 21:00 UTC because of the observed priors. At 21:00 UTC, the boxplot indicates images are typically as far from zero longitude as will been seen in the data.to recognize the imagery whose ground truth GPS is contained in a cell and hence, during inference, the mesh ascribes a probability distribution of geo-labels for the input imagery. The classification structure is generated using a Delaunay triangle-based meshing architecture. This differs from the PlaNet approach, which utilizes a quad-tree mesh. Similarly to PlaNet, the mesh cells are generated such that they conserve surface area, but the approach is not as simple as with a quad-tree. The triangular mesh was deployed under the hypothesis that the Delaunay triangles would be more adaptive to the geometric features of the Earth. Since triangular meshes are unstructures, they can more easily capture water/land interfaces without the additional refinement needed by quadtree meshes. However, the triangular mesh loses refinement level information which comes with a structured quad-tree approach, which would allow for granularity to be controlled more simplistically, based which cells contain other cells. In order to control mesh refinment, cells are adapatively refined, or divided, when the cell contains more than some number of examples (refinement limit) or the mesh cell is dropped from being classified by the model (the contained imagery is also dropped from training) if the cell contains less than a number of samples (minimum examples). The options used for generating the three meshes in this paper are shown in Table 1. Parameters for the initialization of the mesh were not explored; each mesh was initialized with a 31 x 31 structured grid with equal surface area in each triangle. The mesh for the a) coarse mesh and b) fine mesh are shown in FIG2.1.1. Table 1: The geolocation classification mesh can be tuned by modifying the refinement level to adjust the maximum and minimum number of cells in each mesh cell. This table shows the meshing parameters used for the three meshes studied in this work. These meshes were selected to cover a reasonable range of mesh structures, with fine P meant to replicate the mesh parameters utilized by PlaNet. However, it should be noted that this isn't direcly comparable since the fine P mesh is generated with both a different methodology and a different dataset. Coarse 8000 1000 538 Fine 5000 500 6565 Fine P 10000 50 4771Figure 2: Coarse mesh and fine mesh classification structure for classification models developed in this paper. The red triangles indicate regions where the number of images per cell meets the criteria for training. The coarse mesh was generated with an early dataset of 2M YFCC100M images, but final training was performed by populating those cells with all training data without additional refinement. The fine and fine P meshes were generated with all available imagery (14M). The fine mesh does a better job at representing geographic regions, for example, the water/land interface of Spain and Portugal becomes evident. However, this is masked with the coarser mesh, which lacks fidelity to natural geographic boundaries. The Inception v4 convolutional neural network architecture proposed in BID16 ) is deployed to develop the mesh-based classification geolocation model (Model M1) presented in this work, which differs from the Inception v3 architecture used in PlaNet BID19. A softmax classification is utilized, similar to the PlaNet approach. One further difference between the work presented here and that of PlaNet is that cells are labeled based the cell centroid, in latitudelongitude. Here an alternate approach is used where the "center-of-mass" of the training data in a containing cell is computed, whereby the geolocation for each cell was specified as the lat/lon centroid of the image population. Significant improvements are expected for coarse meshes, as it can be noticed in FIG2.1.1 for example, that the cell centroid on the coast of Spain is out in the ocean. Therefore, any beach image, as an example, will have an intrinsically higher error that would be otherwise be captured by a finer mesh. This is expecially true for high density population regions. Models are evaluated by calculating the distance between the predicted cell and the ground-truth GPS coordinate using a great-circle distance: DISPLAYFORM0 where x i and y i is the latitude and longitude, respectively, in radians, and R is the radius of the Earth (assumed to be 6372.795 km). An error threshold here is defined as the number of images which are classified within a distance of d. Error thresholds of 1 km, 25 km, 200 km, 750 km, 2500 km are utilized to represent street, city, region, country, and continent localities to remain consistent with previous work BID6 BID19 BID18. As an example, if 1 out of 10 images are geolabeled with a distance of less than 200 km, then the 200 km threshold would be 10%. Every YFCC100M image has associated meta data; importantly these data consist of user id and posted time. Posting time is utilized in model M2, FIG2. Let Z ik be the one-hot encoding of the i th image for the k th class, so that it takes on value 1 if image i belongs to the k th geolocation class. Output N Softmax 1 is only strong evidence for P (Z ik = 1|t i) = P (Z ik = 1), and it could still be the case that DISPLAYFORM0 DISPLAYFORM1. Which is to say, conditioned on the content of an image there could be no dependence on time, but it seems prudent with the evidence in this figure to proceed under the assumption that there is time dependence. The operational research hypothesis for this model (M2) is that there remains a time-dependence after conditioning on image content. To incorporate time, related variables are appended to the output of the geolocation model (M1) to form a new input for M2. Every image has a vector of fit probabilitiesp i from the softmax layer of M1.p i is filtered so that only the top 10 maximum entries remain, and the other entries are set to 0 (p Model M3, FIG2, simultaneously geolocates many images from a single user with an LSTM model. The idea here is to borrow information from other images a user has posted to help aid geolocation. The Bidirectional LSTMs capitalizes on correlations in a single user's images. LSTMs were also considered by BID19, but in PlaNet the albums were created and organized by humans. When a human organizes images they may do so by topic or location. In M3, all images by a single user are organized sequentially in time with no particular further organization. The related research question is: does the success observed by BID19 extend to this less informative organization of images? All images from a user are grouped into albums of size 24. If there are not enough images to fill an album then albums are 0 padded and masking is utilized. During training, a user was limited to a single random album per epoch. Album averaging was considered by (PlaNet). In album averaging, the images in an album are assigned a mesh cell based on the highest average probability across all images. This method increases accuracy by borrowing information across related images. As a control, a similar idea is applied to user images, in which, the location of an image is determined as the maximum average probability across all images associated with the posting user. This assumes that all images from a user are from the same mesh grid cells. In addition with user-averaging, there is no optimization that controls class frequencies to be unbiased. Finally, LSTM on a time-ordered sequence of images was considered (without respect to user). However, we were unable to improve performance significantly past that gained by just adding time to the model, so albums without user are not further considered in this paper. the output of M1 is filtered to output only the top 10 mesh cell probabilities (making it mostly sparse) re-normalized to sum to 1. Training of M2 and M3 was only done on the validation data of M1 (using a new random split of test-validation). Time inputs are concatenated to filtered and normalized outputs at and a new training step is implied. A small abuse of notation is present: M3 (Time-Albums) is properly described as M2 concatenated with M3 layers. Meshing parameters are investigated to understand the sensitivity to mesh adaptation. The for each mesh is shown in TAB1. There is an apparent trade-off between fine-grain and coarsegrain geolocation. The coarse mesh demonstrates improved large granularity geolocation and the fine mesh performs better at finer-granularities, as would be expected. This observation was also noticed in BID18. In addition, the impact of classifying on the centroid of the training data is compared to utilizing the cell centroid for class labels. A dramatic improvement is noticed for the coarse mesh, with only a modest improvement for the fine mesh. BID20 model was used in conjunction with indoor-outdoor label deliminations to filter geolocation inference only on outdoor imagery. Note that the geolocation model was not re-trained on the "outdoor" imagery, this is only conducted as a filtering operation during inference. Results are shown in TAB2. In general, the improvement is quite good, about a 4-8% improvement in accuracy for region/country localities, with a more modest boost in smaller-scale regions. The Im2GPS testing data is utilized to test the model on the 237 images provided by the work of BID6. Results are tabluated in TAB3 for all of the meshes. The imagery centroid classification labels are generated with the YFCC100M training data, yet the performance is still greatly improved when applied to the Im2GPS testing set, demostrating generality with the approach. The performance of the M1 classification model is comparable to the performance of BID19 ) with a factor of ten less training data and far fewer classification regions (6k compared to 21k); the coarse mesh M1 model exceeds the performance of PlaNet for large regions (750 and 2500 km). Use of time improves geolocation in two ways. There is a slight gain of accuracy for using time as indicated in TAB2. 24.20% of images are geolocated to within 200 km as opposed to 23.99% without using time with the coarse mesh, and 12.28% versus 10.49% are within 25 km using the fine mesh. This small persistent advantage can be seen across all error calculations and is statistically significant. There is a measurable difference between the error of the coarse mesh using time (M2) and not using time (M1). There exist a matched pair of errors for each image: e 1 i and e 2 i, where i is a validation image index. The first superscript being the M1 error and the second superscript being M2 error for 1 i and likewise for µ 2. This hypothesis is tested with a Wilcoxon-Signed-Test for paired observations, which makes a minimal number of distributional assumptions. Specifically, normality of the errors is not assumed. The differenceμ 2 −μ 1 = 381 is highly significant (p-value < 10 −16) in favor of using time-inputs, so even though the effect of M2 is small, it is not explained by chance alone. It is the case that the distribution of errors is mean shifted, but it is not uniformly shifted to lower error, nor is it it the case that images are universally predicted closer to the truth. The median of coarse mesh e 1 is 1627 km while the median of e 2 is 1262 km TAB5.Time input models appear to have lower-bias class probabilities. Cross-entropy was optimized in training for both the classification model (M1) and time-inputs models (M2). In each training method the goal is to minimize these class biases. KL-Divergence(p,q) is calculated for each model, where p is the observed class proportions for the true labels in validation, and q is the observed class proportions for the model predicted labels in validation (in both cases, 1 is added to the counts prior to calculating proportions as an adjustment for 0 count issues). The KL-divergence of the model output class frequencies compared to the true class frequencies in validation are in TAB5."User-Averaging" is incorporated into because it is a simple method that appears to be more accurate then predicting individual images with M1 or M2; however, it biases cell count frequency TAB5. In general when using the average probability vector to predict a user's image, there is no guarantee that the class frequencies are distributed similarly to the truth; thus, improved accuracy can come with higher bias which is what is observed. Albums are a much better approach to borrow information across users because built into the training method is a bias reducing cross-entropy optimization, and indeed LSTMs on user albums had the lowest class bias of any model considered. Table 6: Percentage accuracy to specific resolution: Researched models compared at various spatial resolutions. Coarse and fine mesh have 538 and 6565 triangles across the globe (cells), respectively. " Time inputs" indicates time meta information has been concatenated with M1 output. Albums are created using user-id and contain 24 images. Bold is the best observed accuracy in column. Coarse Mesh and Fine Mesh Best Possible are not actual models, but rather the best possible accuracy if every image was given exactly the right class. In the fine mesh the best possible accuracy is incidental, but in the coarse mesh it is a severe limitation for street level accuracy. Conditioning on latent variables can only improve geolocation models. Universally using time of day in the models was observed to increase accuracy and lower bias. Time of day is a weak addition to Inception-like , but it is useful to be as accurate as possible, and it makes statistically significant improvements to geolocation. Both meshes were improved by using time information (M2). This is a that is not surprising, and as a research approach, can be applied to any number of meta data variables that might accompany images. Accounting for indoor/outdoor scenes in images explained variation in validation accuracy. Outdoor only are better than for all images. We suggest as future work that the probability that an image is outdoor could be concatenated to the input of M2. The accompanying research hypothesis is that in the case that an image is indoors, perhaps the model will learn to weight time or other meta data more heavily, or otherwise be able to use that information optimally. Increasing the granularity of a grid reduces accuracy at the country and regional level, while improving accuracy at street and city level accuracy. To be clear though, street level geoinferencing is not practical with a course mesh. This is shown by the best possible accuracy in Table 6, and so it is expected that a fine mesh would do better. On the other hand, there is no reason to assume that a fine mesh has to be better for geolocation at larger resolutions than 25 km, nor is there any explicit way to prove a fine mesh should do no worse than a course mesh. What we observe is that a course mesh is a superior grid for 200 km resolutions. Furthermore, we show that for both the coarse mesh and the fine mesh, using a Delunary triangle-based mesh provides the ability to train accurate models with far fewer training examples than what was previously published. Images were divided at random into training and validation sets of 12.2M and 2.7M images and associated metadata, respectively. Validation data used for M1 was further sub-divided at random into training and validations sets for training time-based models (M2 and M3), so that no data used to train M1 was used to also train M2 and M3. Historically, softmax classification is shown to perform quite poorly when the number of output classes is large BID0 BID12 BID15. During initial experiments with large meshes (> 4,000 mesh regions), a training procedure was developed to circumvent these challenges. Empirically, this procedure worked for training the large models presented in this paper; however, it is not demostrated to be ideal or that all steps are necessary for model training. This approach started by training the model, pre-trained on ImageNet , with Adagrad BID3. Second, the number of training examples were increased each epoch by 6%, with the initial number of examples equal to the number of cells times 200. Third, the classes were biased by oversampling minority classes, such that all classes were initially equally represented. A consequence of this approach, however, is that the minority classes are seen repeatedly and therefore the majority classes have significantly more learned diversity. Fourth, the class bias was reduced after each model had completed a training cycle -previous weights loaded and the model re-trained with a reduced bias. The final model was trained with SGD, using a linearly decreasing learning rate reduced at 4% with each epoch, without class biasing and with the full dataset per epoch. The initial value of the learning rate varied for each model (between 0.004 and 0.02). The values of those hyperparameters were empirically determined. The layers of M2 are described in TAB0. M2 is trained using He intializations (BID9), initial iterations of Adaboost BID4, followed by ADAM at learning rates of 0.005 and 0.001 BID10. Early stopping is used to detect a sustained decrease in validation accuracy BID2 ). The generality of the M1 classification model is demonstrated by performing a query-by-example on the 2K random Im2GPS. An example of an image of the Church of the Savior on Spilled Blood is shown in FIG3. By manual inspection (querying on the bounding box by GPS location), this church was not present in the training data nor in the 2K random dataset BID6. Each image is given categorical indicator variable z ik = 1 if the i th image is in the kth class, 0 otherwise. There exist a latent class distribution p i = P (Z i = 1) which is assumed constant between training, testing, and application. An estimate of this unknown distribution iŝ p z = 1 N j∈Training I(z ij = 1). The second to last layer in all trained networks is assumed to be a fit logit for class i: L i = (L 1i, ..., L Ki) for the i th image where K is the number of classes in the mesh grid. The last layer output from networks is a softmaxp models are compared we prefer the most accurate, but may also tilt toward unbiased models in classification distribution. If training has been done well, it should be the case that the KL-divergence between q and p is low: kp k logp k q k. As a matter of completeness we also consider the entropy of p and q.

Represent the short summary for finding the corresponding scientific article (long)

A global geolocation inferencing strategy with novel meshing strategy and demonstrating incorporating additional information can be used to improve the overall performance of a geolocation inference model.

Represent the TLDR for finding the corresponding scientific article.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the scientific article to find a short tldr.

Question: Hierarchical Bayesian methods have the potential to unify many related tasks (e.g. k-shot classification, conditional, and unconditional generation) by framing each as inference within a single generative model. We show that existing approaches for learning such models can fail on expressive generative networks such as PixelCNNs, by describing the global distribution with little reliance on latent variables. To address this, we develop a modification of the Variational Autoencoder in which encoded observations are decoded to new elements from the same class; the , which we call a Variational Homoencoder (VHE), may be understood as training a hierarchical latent variable model which better utilises latent variables in these cases. Using this framework enables us to train a hierarchical PixelCNN for the Omniglot dataset, outperforming all existing models on test set likelihood. With a single model we achieve both strong one-shot generation and near human-level classification, competitive with state-of-the-art discriminative classifiers. The VHE objective extends naturally to richer dataset structures such as factorial or hierarchical categories, as we illustrate by training models to separate character content from simple variations in drawing style, and to generalise the style of an alphabet to new characters. Learning from few examples is possible only with strong inductive biases. In machine learning such biases can come from hand design, as in the parametrisation of a model, or can be the of a meta-learning algorithm. Furthermore they may be task-specific, as in discriminative modelling, or may describe the world causally so as to be naturally reused across many tasks. Recent work has approached one-and few-shot learning from all of these perspectives. Siamese Networks BID15, Matching Networks, Prototypical Networks BID23 and MANNs BID22 ) are all models discriminatively trained for few-shot classification. Such models can achieve state-of-the-art performance at the task they were trained for, but provide no principled means for transferring knowledge to other tasks. Other work such as has developed conditional generative models, which take one or a few observations from a class as input, and return a distribution over new elements p(x|D). These models may be used as classifiers despite not being explicitly trained for this purpose, by comparing conditional likelihoods. They may also be used to generate full sets incrementally as p(X) = i p(x i |x 1, . . ., x i−1), as discussed in Generative Matching Networks BID0. However, such models are a more natural fit to sequences than sets as they typically lack exchangeability, and furthermore they do not expose any latent representation of shared structure within a set. Finally are hierarchical approaches that model shared structure through latent variables, as p(X) = A VAE treats all datapoints as independent, so only a single random element need be encoded and decoded each step. A Neural Statistician instead feeds a full set of elements X through both encoder and decoder networks, in order to share a latent variable c. In a VHE, we bound the full likelihood p(X) using only random subsamples D and x for encoding/decoding. Optionally, p(x|c) may be defined through a local latent variable z. In this work we propose the Variational Homoencoder (VHE), aiming to combine several advantages of the models described above:1. Like conditional generative approaches, we train on a few-shot generation objective which matches how our model may be used at test time. However, by introducing an encoding cost, we simultaneously optimise a likelihood lower bound for a hierarchical generative model, in which structure shared across elements is made explicit by shared latent variables. 2. Previous work BID6 has learned hierarchical Bayesian models by applying Variational Autoencoders to sets, such as classes of images. However, their approach requires feeding a full set through the model per gradient step FIG1 ), rendering it intractable to train on very large sets. In practice, they avoid computational limits by sampling smaller subset as training data. In a VHE, we instead optimise a likelihood bound for the complete dataset, while constructing this bound by subsampling. This approach can not only improve generalisation, but also departs from previous work by extending to models with richer latent structure, for which the joint likelihood cannot be factorised. 3. As with a VAE, the VHE objective includes both an encoding-and reconstruction-cost. However, by sharing latent variables across a large set of elements, the encoding cost per element is reduced significantly. This facilitates use of powerful autoregressive decoders, which otherwise often suffer from ignoring latent variables BID3. We demonstrate the significance of this by applying a VHE to the Omniglot dataset. Using a Pixel-CNN decoder , our generative model is arguably the first with a general purpose architecture to both attain near human-level one-shot classification performance and produce high quality samples in one-shot generation. When dealing with latent variable models of the form p(x) = z p(z)p(x|z)dz, the integration is necessary for both learning and inference but is often intractable to compute in closed form. Variational Autoencoders provide a method for learning such models by utilising neural-network based approximate posterior inference. Specifically, a VAE comprises a generative network p(z)p(x|z) parametrised by θ, alongside a separate inference network q(z; x) parameterised by φ. These are trained jointly to maximise a single objective: DISPLAYFORM0 DISPLAYFORM1 As can be seen from Equation 1, this objective L X is a lower bound on the total log likelihood of the dataset x∈X log p(x), while q(z; x) is trained to approximate the true posterior p(z|x) as accurately as possible. If it could match this distribution exactly then the bound would be tight so that the VAE objective equals the true log likelihood of the data. In practice, the ing model is typically a compromise between two goals: pulling p towards a distribution that assigns high likelihood to the data, but also towards one which allows accurate inference by q. Equation 2 provides a formulation for the same objective which can be optimised stochastically, using Monte-Carlo integration to approximate the expectation. The Neural Statistician BID6 ) is a Variational Autoencoder in which each item to be encoded is itself a set, such as the set X (i) of all images with a particular class label i: DISPLAYFORM0 The generative model for sets, p(X), is described by introduction of a corresponding latent variable c. Given c, individual x ∈ X are conditionally independent: DISPLAYFORM1 This functional form is justified by de Finetti's theorem under the assumption that elements within in each set X are exchangeable. The likelihood is again intractable to compute, but it can be bounded below via: DISPLAYFORM2 Unfortunately, calculating the variational lower bound for each set X requires evaluating both q(c; X) and p(X|c), meaning that the entire set must be passed through both networks for each gradient update. This can easily become intractable for classes with hundreds of examples. Indeed, previous work BID6 ensures that sets used for training are always of small size by instead maximising a likelihood lower-bound for randomly sampled subsets. In this work we instead replace the variational lower-bound in Equation 5 with a new objective, itself constructed via sub-sampled datasets of reduced size. We use a constrained variational distribution q(c; D), D ⊆ X for posterior inference and an unbiased stochastic approximation log p(x|c), x ∈ X for the likelihood. In the following section we show that the ing objective can be interpreted as a lower-bound on the log-likelihood of the data. This bound will typically be loose due to stochasticity in sampling D, and we view this as a regularization strategy: we aim to learn latent representations that are quickly inferable from a small number of instances, and the VHE objective is tailored for this purpose. We would like to learn a generative model for sets X of the form DISPLAYFORM0 We will refer our full dataset as a union of disjoint sets X = X 1 X 2... X n, and use X (x) to refer to the set X i x. Using the standard consequent of Jensen's inequality, we can lower bound the log-likelihood of each set X using an arbitrary distribution q. In particular, we give q as a fixed function of arbitrary data. DISPLAYFORM1 DISPLAYFORM2 where DISPLAYFORM3 Reparametrization gradient estimate using c, z (θ, φ) ← (θ, φ) + λg Gradient step, e.g SGD until convergence of (θ, φ)Splitting up individual likelihoods, we may rewrite DISPLAYFORM4 Finally, we can replace the universal quantification with an expectation under any distribution of D (e.g. uniform sampling from X without replacement): DISPLAYFORM5 This formulation suggests a simple modification to the VAE training procedure, as shown in Algorithm 1. At each iteration we select an element x, use resampled elements D ⊂ X (x) to construct the approximate posterior q(c; D), and rescale the encoding cost appropriately. If the generative model p(x|c) also describes a separate latent variable z for each element, we may simply introduce a second inference network q(z; c, x) in order to replace the exact reconstruction error of Equation 9 by a conditional VAE bound: DISPLAYFORM6 The above derivation applies to a dataset partitioned into disjoint subsets X = X 1 X 2... X n, each with a corresponding latent variable c i. However, many datasets offer a richer organisational structure, such as the hierarchical grouping of characters into alphabets or the factorial categorisation of rendered faces by identity, pose and lighting BID16.Provided that such organisational structure is known in advance, we may generalise the training objective in FIG1 to include a separate latent variable c i for each group X i within the dataset, even when these groups overlap. To do this we first rewrite this bound in its most general form, where c collects all latent variables: DISPLAYFORM0 As shown in Figure 2, a separate D i ⊂ X i may be subsampled for inference of each latent variable DISPLAYFORM1 This leads to an analogous training objective (Equation 15), which Figure 2: Application of VHE framework to hierarchical (left) and factorial (right) models. Given an element x such that x ∈ X 1 and x ∈ X 2, an approximate posterior is constructed for the corresponding shared latent variables c 1, c 2 using subsampled sets DISPLAYFORM2 may be applied to data with factorial or hierarchical category structure. For the hierarchical case, this objective may be further modified to infer layers sequentially, as in Supplementary Material 6.2. DISPLAYFORM3 3.3 DISCUSSION As evident in Equation 9, the VHE objective provides a formal motivation for KL rescaling in the variational objective (a common technique to increase use of latent variables in VAEs) by sharing these variables across many elements. This is of particular importance when using autoregressive decoder models, for which a common failure mode is to learn a decoder p(x|z) with no dependence on the latent space, thus avoiding the encoding cost. In the context of VAEs, this particular issue has been discussed by BID3 who suggest crippling the decoder as a potential remedy. The same failure mode can occur when training a VAE for sets if the inference network q is not able to reduce its approximation error D KL [q(c; D) p(c|D)] to below the total correlation of D, either because |D| is too small, or the inference network q is too weak. Variational Homoencoders suggest a potential remedy to this, encouraging use of the latent space by reusing the same latent variables across a large set X. This allows a VHE to learn useful representations even with |D| = 1, while at the same time utilising a powerful decoder model to achieve highly accurate density estimation. In a VAE, use of a recognition network encourages learning of generative models whose structure permits accurate amortised inference. In a VHE, this recognition network takes only a small subsample as input, which additionally encourages that the true posterior p(c|X) can be well approximated from only a few examples of X. For a subsample D ⊂ X, q(c; D) is implicitly trained to minimise the KL divergence from this posterior in expectation over possible sets X consistent with D. For a data distribution p d we may equivalently describe the VHE objective (Equation 12) as DISPLAYFORM0 Note that the variational gap on the right side of this equation is itself bounded by: DISPLAYFORM1 The left inequality is tightest when p(c|X) matches p(c|D) well across all X consistent with D, and exact only when these are equal. We view this aspect of the VHE loss as regulariser for constrained posterior approximation, encouraging models for which the posterior p(c|X) can be well determined by sampled subsets D ⊂ X. This reflects how we expect the model to be used at test time, and in practice we have found this'loose' bound to perform well in our experiments. In principle, the bound may also be tightened by introducing an auxiliary inference network (see Supplementary Material 6.1) which we leave as a direction for future research. With a Neural Statistician model, under-utilisation of latent variables is expected to pose the greatest difficulty either when |D| is too small, or the inference network q is insufficiently expressive. We demonstrate on simple 1D distributions that a Variational Homoencoder can bring improvements under these circumstances. For this we created five datasets as follows, each containing 100 classes from a particular parametric family, and with 100 elements sampled from each class.1. Gaussian: Each class is Gaussian with µ drawn from a Gaussian hyperprior (fixed σ 2).2. Mixture of Gaussians: Each class is an even mixture of two Gaussian distributions with location drawn from a Gaussian hyperprior (fixed σ 2 and separation). Our show that, when |D| is small, the Neural Statistician often places little to no information in q(c; D) (FIG2, top row). Our careful training suggests that this is not an optimisation difficulty, but is core to the objective as in BID3. In these cases a VHE better utilises the latent space, leading to improvements in both few-shot generation (by conditional NLL) and classification. Importantly, this is achieved while retaining good likelihood of test-set classes, typically matching or improving upon that achieved by a Neural Statistician (including a standard VAE, corresponding to |D| = 1). Please see Supplement 6.3 for further comparison to alternative objectives. FORMULA0 ). Secondly, since the VHE introduces both data-resampling and KL-rescaling as modifications to this baseline, we separate the contributions of each using two intermediate objectives: DISPLAYFORM0 Rescale only: DISPLAYFORM1 All models were trained on a random sample of 1200 Omniglot classes using images scaled to 28x28 pixels, dynamically binarised, and augmented by 8 rotations/reflections to produce new classes. We additionally used 20 small random affine transformations to create new instances within each class. Models were optimised using Adam , and we used training error to select the best parameters from 5 independent training runs. This was necessary to ensure a fair comparison with the Neural Statistician objective, which otherwise converges to a local optimum with q(c; D) = p(c). We additionally implemented the'sample dropout' trick of , but found that this did not have any effect on model performance. Discriminative models, log q(y|x, X, Y)Siamese Networks BID15 88.1% 97.0% Matching Networks BID25 93.8% 98.7% Convnet with memory module BID13 95.0% 98.6% mAP-DLM BID24 95.4% 98.6% Model-Agnostic Meta-learning BID7 95.8% 98.9% Prototypical Networks BID23 96.0% 98.9%*Uses train/test split from TAB0 collects classification of models trained using each of the four alternative training objectives, for both architectures. When using a standard deconvolutional architecture, we find little difference in classification performance between all four training objectives, with the Neural Statistician and VHE models achieving equally high accuracy. For the hierarchical PixelCNN architecture, however, significant differences arise between training objectives. In this case, a Neural Statistician learns an strong global distribution over images but makes only minimal use of latent variables c. This means that, despite the use of a higher capacity model, classification accuracy is much poorer (66%) than that achieved using a deconvolutional architecture. For the same reason, conditional samples display an improved sharpness but are no longer identifiable to the cue images on which they were conditioned (FIG6). Our careful training suggests that this is not an optimisation difficulty but is core to the objective, as discussed in BID3.By contrast, a VHE is able to gain a large benefit from the hierarchical PixelCNN architecture, with a 3-fold reduction in classification error (5-shot accuracy 98.8%) and conditional samples which are simultaneously sharp and identifiable (FIG6). This improvement is in part achieved by increased utilisation of the latent space, due to rescaling of the KL divergence term in the objective. However, our show that this common technique is insufficient when used alone, leading to overfitting to cue images with an equally severe impairment of classification performance (accuracy 62.8%). Rather, we find that KL-rescaling and data resampling must be used together in order to for the benefit of the powerful PixelCNN architecture to be realised. TAB3 lists the classification accuracy achieved by VHEs with both |D| = 1 and |D| = 5, as compared to existing deep learning approaches. We find that both networks are not only state-of-theart amongst deep generative models, but are also competitive against the best discriminative models trained directly for few-shot classification. Unlike these discriminative models, a VHE is also able to generate new images of a character in one-shot, producing samples which are simultaneously realistic and faithful to the class of the cue image (FIG5). As our goal is to model shared structure across images, we evaluate generative performance using joint log likelihood of the entire Omniglot test set (rather than separately across images). From this perspective, a single element VAE will perform poorly as it treats all datapoints as independent, optimising a sum over log likelihoods for each element. By sharing latent variables across all elements of the same class, a VHE can improve upon this considerably. Previous work which evaluates likelihood typically uses the train/test split of BID2. However, our most appropriate comparison is with Generative Matching Networks BID0 ) as they also model dependencies within a class; thus, we trained models under the same conditions as them, using the harder test split from Lake et al. FORMULA0 with no data augmentation. We evaluate the joint log likelihood of full character classes from the test set, normalised by the number of elements, using importance weighting with k=500 samples from q(c; X). As can be seen in TAB4, our hierarchical PixelCNN architecture is able to achieve state-of-the-art log likelihood only when trained using the full Variational Homoencoder objective. 1 n log i p(x i) DRAW BID8 < 96.5 nats Conv DRAW BID9 < 91.0 nats VLAE BID3 89.83 nats 1 n log i p(x i |x 1:i−1) Variational Memory Addressing BID1 > 73.9 nats Generative Matching Networks BID0 62.42 nats 1Shared-latent models DISPLAYFORM0 Homoencoder 61.22 nats To demonstrate how the VHE framework may apply to models with richer category structure, we built both a hierarchical and a factorial VHE (Figure 2) using simple modifications to the above architectures. For the hierarchical VHE, we extended the deconvolutional model with an extra latent layer a using the same encoder and decoder architecture as c. This was used to encode alphabet level structure for the Omniglot dataset, learning a generative model for alphabets of the form DISPLAYFORM0 Again, we trained this model using a single objective, using separately resampled subsets D a and D c to infer each latent variable (Supplement 6.2). We then tested our model at both one-shot character generation and 5-shot alphabet generation, using samples from previously unseen alphabets. As shown in FIG7, our single trained model is able to learn structure at both layers of abstraction. For the factorial VHE, we extended the Omniglot dataset by assigning each image to one of 30 randomly generated styles (independent of its character class), modifying both the colour and pen stroke of each image. We then extended the PixelCNN model to include a 6-dimensional latent variable s to represent the style of an image, alongside the existing c to represent the character. We used a CNN for style encoder q(s|D s), and for each image location we condition the PixelCNN decoder using the outer product s ⊗ c ij.We then test this model on a style transfer task by feeding separate images into the character encoder q(c|D c) and style encoder q(s|D s), then rendering a new image from the inferred (c, s) pair. We find that synthesised samples are faithful to the respective character and style of both cue images FIG8, demonstrating ability of a factorial VHE to successfully disentangle these two image factors using separate latent variables. We introduced the Variational Homoencoder: a deep hierarchical generative model learned by a novel training procedure which resembles few-shot generation. This framework allows latent variables to be shared across a large number of elements in a dataset, encouraging them to be well utilised even alongside highly expressive decoder networks. We demonstrate this by training a hierarchical PixelCNN model on Omniglot dataset, and show that our novel training objective is responsible for the state-of-the-art it achieves. This model is arguably the first which uses a general purpose architecture to both attain near human-level one-shot classification performance and produce high quality samples in one-shot generation. The VHE framework extends naturally to models with richer latent structure, as we demonstrate with two examples: a hierarchical model which generalises the style of an alphabet to produce new characters, and a factorial model which separates the content and drawing style of coloured character images. In addition to these modelling extensions, our variational bound may also be tightened by learning a subsampling procedure q(D; X), or by introducing an auxiliary inference network r(D; c, X) as discussed in Supplementary Material 6.1. While such modifications were unnecessary for our experiments on Omniglot character classes, we expect that they may yield improvements on other datasets with greater intra-class diversity.6 SUPPLEMENTARY MATERIAL The likelihood lower bound in the VHE objective may also be tightened by introduction of an auxiliary network r(D; c, X), trained to infer which subset D ⊂ X was used in q. This meta-inference approach was introduced in BID21 to develop stochastic variational posteriors using MCMC inference, and has recently been applied to approximate inference evaluation . Applied to Equation 12, this yields a modified bound for the VHE objective DISPLAYFORM0 where q (D; X) describes the stochastic sampling procedure for sampling D ⊂ X, which indeed may itself be learned using policy gradients. We have conducted preliminary experiments using fixed q and a simple functional form r(D; c, X) = i r(d i ; c, X) ∝ i f ψ (c) · ξ di, learning parameters ψ and embeddings {ξ d : d ∈ X}; however, on the Omniglot dataset we found no additional benefit over the strictly loose bound (Equation 12). We attribute this to the already high similarity between elements of the same Omniglot character class, allowing the approximate posterior q(c; D) to be relatively robust to different choices of D. However, we expect that the gain from using such a tightened objective may be much greater for domains with lower intra-class similarity (e.g. natural images), and thus suggest the tightened bound of Equation 21 as a direction for future research. The resampling trick may be applied iteratively, to construct likelihood bounds over hierarchically organised data. Expanding on Equation 12, suppose that we have collection of datasets DISPLAYFORM0 For example, each X might be a different alphabet whose latent description a generates many character classes X i, and for each of these a corresponding latent c i is used to generate many images x ij. From this perspective, we would like to learn a generative model for alphabets X of the form DISPLAYFORM1 Reapplying the same trick as before yields a bound taken over all elements x: DISPLAYFORM2 This suggests an analogous hierachical resampling procedure: Summing over every element x, we can bound the log likelihood of the full hierarchy by resampling subsets D c, D a, etc. at each level to construct an approximate posterior. All networks are trained together by this single objective, sampling x, D a and D c for each gradient step. Note that this procedure need only require passing sampled elements, rather than full classes, into the upper-level encoder q a. Our architecture uses a 8x28x28 latent variable c, with a full architecture detailed below. For our classification experiments, we trained 5 models on each of the objectives (VHE, Rescale only, Resample only, NS). Occasionally we found instability in optimisation, causing sudden large increases in the training objective. When this happened, we halted and restarted training. All models were trained for 100 epochs on 1000 characters from the training set (the remaining 200 have been used as validation data for model selection). Finally, for each objective we selected the parameters achieving the best training error. Note that we did not optimise or select models based on classification performance, other than through our development of our model's architecture. However, we find that classification performance is well correlated the generative training objective, as can be seen in the full table of . We perform classification by calculating the expected conditional likelihood under the variational posterior: Eq(c;D) p(x|c). This is approximated using 20 samples for the outer expectation, and importance sampling with k = 10 for the inner integral p(x|c) = Eq(t|x) DISPLAYFORM3 To evaluate and compare log likelihood, we trained 5 more models with the same architecture, this time on the canonical 30-20 alphabet split of Lake et al. We did not augment our training data. Again, we split the set into training data (25 alphabets) and validation data but do not use the validation set in training or evaluation for our final . We estimate the total class log likelihood by importance weighting, using k=20 importance samples of the class latent c and k=10 importance samples of the transformation latent t for each instance. [d] denotes a dimension d tensor. {t} denotes a set with elements of type t. Posteriors q are Gaussian. A PixelCNN with autoregressive weights along only the spatial (not depth) dimensions of c. We use 2 layers of masked 64x3x3 convolutions, followed by a ReLU and two 8x1x1 convolutions corresponding to the mean and log variance of a Gaussian posterior for the following pixel. We extend the same architecture described in Appendix B of BID6, with only a simple modification: we introduce a new latent layer containing a 64-dimensional variable a, with a Gaussian prior. We give p(c|a) the same functional form as p(z|c), and give q(a|D a) the same functional form as q(c; D c) using the shared encoder. We created a VHE using the same deconvolutional architecture as applied to omniglot, and trained it on the Caltech-101 Silhouettes dataset. 10 object classes were held out as test data, which we use to generate both 1-shot and 5-shot conditional samples.

Technique for learning deep generative models with shared latent variables, applied to Omniglot with a PixelCNN decoder.

represent the tldr for finding the corresponding scientific article

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Common-sense or knowledge is required to understand natural language, but in most neural natural language understanding (NLU) systems, the requisite knowledge is indirectly acquired from static corpora. We develop a new reading architecture for the dynamic integration of explicit knowledge in NLU models. A new task-agnostic reading module provides refined word representations to a task-specific NLU architecture by processing knowledge in the form of free-text statements, together with the task-specific inputs. Strong performance on the tasks of document question answering (DQA) and recognizing textual entailment (RTE) demonstrate the effectiveness and flexibility of our approach. Analysis shows that our models learn to exploit knowledge selectively and in a semantically appropriate way. Understanding natural language depends crucially on common-sense or knowledge, for example, knowledge about what concepts are expressed by the words being read (lexical knowledge), and what relations hold between these concepts (relational knowledge). As a simple illustration, if an agent needs to understand that the statement "King Farouk signed his abdication" is entailed by "King Farouk was exiled to France in 1952, after signing his resignation", it must know (among other things) that abdication means resignation of a king. In most neural natural language understanding (NLU) systems, the requisite knowledge is implicitly encoded in the models' parameters. That is, what knowledge is present has been learned from task supervision and also by pre-training word embeddings (where distributional information reliably reflects certain kinds of useful knowledge, such as semantic relatedness). However, acquisition of knowledge from static training corpora is limiting for two reasons. First, we cannot expect that all knowledge that could be important for solving an NLU task can be extracted from a limited amount of training data. Second, as the world changes, the facts that may influence how a text is understood will likewise change. In short: building suitably large corpora to capture all relevant information, and keeping the corpus and derived models up to date with changes to the world would be impractical. In this paper, we develop a new architecture for dynamically incorporating external knowledge in NLU models. Rather than relying only on static knowledge implicitly present in the training data, supplementary knowledge is retrieved from a knowledge base to assist with understanding text inputs. Since NLU systems must necessarily read and understand text inputs, our approach incorporates knowledge by repurposing this reading machinery-that is, we read the text being understood together with supplementary natural language statements that assert facts (assertions) which are relevant to understanding the content (§2).Our knowledge-augmented NLU systems operate in a series of phases. First, given the text input that the system must understand, which we call the context, a set of relevant supporting assertions is retrieved. While learning to retrieve relevant information for solving NLU tasks is an important question BID21 , inter alia), in this work, we focus on learning how to incorporate retrieved information, and use simple heuristic retrieval methods to identify plausibly relevant from an external knowledge base. Once the supplementary texts have been retrieved, we use a word embedding refinement strategy that incrementally reads the context and retrieved assertions starting with context-independent word embeddings and building successively refined embeddings of the words that ultimately reflect both the relevant supporting assertions and the input context (§3). These contextually refined word embeddings, which serve as dynamic memory to store newly incorporated knowledge, are used in any task-specific reading architecture. The overall architecture is illustrated in FIG0. Although we are incorporating a new kind of information into the NLU pipeline, a strength of this approach is that the architecture of the reading module is independent of the final NLU task-the only requirement is that the final architecture use word embeddings. We carry out experiments on several different datasets on the tasks of document question answering (DQA) and recognizing textual entailment (RTE) evaluating the impact of our proposed solution with both basic task architectures and a sophisticated task architecture for RTE (§4). We find that our embedding refinement strategy is quite effective (§5). On four standard benchmarks, we show that refinement helps-even refining the embeddings just using the context (and no additional information) can improve performance significantly, and adding knowledge helps further. Our are very competitive, setting a new state-of-the-art on the recent TriviaQA benchmarks which is remarkable considering the simplicity of the chosen task-specific architecture. Finally, we provide a detailed analysis of how knowledge is being used by an RTE system (§6), including experiments showing that our system is capable of making appropriate counterfactual inferences when provided with "false knowledge". Knowledge resources make information that could potentially be useful for improving NLU available in a variety different formats, such as (subject, predicate, object)-triples, relational databases, and other structured formats. Rather than tailoring our solution to a particular structured representation, we assume that all supplementary information either already exists in natural language statements or can easily be recoded as natural language. In contrast to mapping from unstructured to structured representations, the inverse problem is not terribly difficult. For example, given a triple (monkey, isA, animal) we can construct the free-text assertion "a monkey is an animal" using a few simple rules. Finally, the free-text format means that knowledge that exists only in unstructured text form is usable by our system. A major question that remains to be answered is: given some text that is to be understood, what supplementary knowledge should be incorporated? The retrieval of contextually relevant information from knowledge sources is a complex research topic by itself, and it is likewise crucially dependent on the format of the underlying knowledge base. There are several statistical BID16 and more recently neural approaches BID19 and approaches based on reinforcement learning BID21. In this work we make use of a simple heuristic from which we almost exhaustively retrieve all potentially relevant assertions (see §4), and rely on our reading architecture to learn to extract only relevant information. In the next section, we turn to the question of how to leverage the retrieved supplementary knowledge (encoded as text) in a NLU system. In order to incorporate information from retrieved input texts we propose to compute contextually refined word representations prior to processing the NLU task at hand and pass them to the task in the form of word embeddings. Word embeddings thus serve as a form of memory that not only contains general-purpose knowledge (as in typical neural NLU systems) but also contextual information (including retrieved knowledge). Our incremental refinement process encodes input texts followed by updates on the word embedding matrix using the encoded input in multiple reading steps. Words are first represented non-contextually (i.e., standard word type embeddings), which can be conceived of as the columns in an embedding matrix E 0. At each progressive reading step ≥ 1, a new embedding matrix E is constructed by refining the embeddings from the previous step E −1 using (user-specified) contextual information X for reading step, which is a set of natural language sequences (i.e., texts). An illustration of our incremental refinement strategy can be found in FIG0. In the following, we define this procedure formally. We denote the hidden dimensionality of our model by n and a fully-connected layer by DISPLAYFORM0 The first representation level consists of non-contextual word representations, that is, word representations that do not depend on any input; these can be conceived of as an embedding matrix E 0 whose columns are indexed by words in Σ *. The non-contextual word representation e 0 w for a single word w is computed by using a gated combination of fixed, pre-trained word vectors e p w ∈ R n with learned character-based embeddings e char w ∈ R n. The formal definition of this combination is given in Eq. 1: DISPLAYFORM1 We compute e char w using a single-layer convolutional neural network using n convolutional filters of width 5 followed by a max-pooling operation over time. Combining pre-trained with character based word embeddings in such a way is common practice. Our approach follows BID28 BID31. In order to compute contextually refined word embeddings E given prior representations E −1 we assume a given set of texts X = {x 1, x 2, . . .} that are to be read at refinement iteration. Each text x i is a sequence of word tokens. We embed all tokens of every x i using the embedding matrix from the previous layer, E −1. To each word, we concatenate a one-hot vector of length L with position set to 1, indicating which layer is currently being processed.1 Stacking the vectors into a matrix, we obtain a X i ∈ R d×|x i |. This matrix is processed by a bidirectional recurrent neural network, a BiLSTM BID8 in this work. The ing output is further projected toX i by a fully-connected layer followed by a ReLU non-linearity (Eq. 2). DISPLAYFORM0 To finally update the previous embedding e −1 w of word w, we initially maxpool all representations of occurrences matching the lemma of w in every x ∈ X ing inê w (Eq. 3). Finally, we combine the context-independent representation e w withê w to form a context-sensitive representation e w via a gated addition which lets the model determine how much to revise the embedding with the newly read information (Eq. 5). DISPLAYFORM1 e w = u w e −1 w DISPLAYFORM2 Note that we soften the matching condition for w using lemmatization, lemma(w), during the pooling operation of Eq. 3 because contextual information about certain words is usually independent of the current word form w they appear in. As a consequence, this minor linguistic pre-processing step allows for additional interaction between tokens of the same lemma. The important difference between our contextual refinement step and conventional multi-layer (RNN) architectures is the pooling operation that is performed over occurrences of tokens that share the same lemma. This effectively connects different positions within and between different texts with each other thereby mitigating the problems arising from long-distance dependencies. More importantly, however, it allows models to make use of additional input such as relevant knowledge. We run experiments on four benchmarks for two popular tasks, namely recognizing textual entailment (RTE) and document question answering (DQA). In the following we describe different aspects of our experimental setup in more detail. Task-specific Models Our primary interest is to explore the value of our refinement strategy with relatively generic task architectures. Therefore, we chose basic single-layer bidirectional LSTMs (BiLSTMs) as encoders with a task-specific, feed-forward neural network on top for the final prediction. Such models are common baselines for NLU tasks and can be considered general reading architectures as opposed to the more highly tuned, task-specific NLU systems that are necessary to achieve state-of-the art . However, since such models frequently underperform more customized architectures, we also add our refinement module to a reimplementation of a state-of-the-art architecture for RTE called ESIM BID4 ).All models are trained end-to-end jointly with the refinement module. For the DQA baseline system we add a simple lemma-in-question feature (liq) as suggested in BID31 when encoding the context to compare against competitive baseline . We provide the exact model implementations for our BiLSTM baselines and general training details in Appendix A.Question Answering We apply our DQA models on 2 recent DQA benchmark datasets, SQuAD BID25 and TriviaQA BID11. The task is to predict an answer span within a provided document p given a question q. Both datasets are large-scale, containing on the order of 100k examples. Because TriviaQA is collected via distant supervision the test set is divided into a large but noisy distant supervision part and a much smaller (on the order of hundreds) human verified part. We report on both. See Appendix A.1 for implementation details. Recognizing Textual Entailment We test on the frequently used SNLI dataset BID2, a collection of 570k sentence pairs, and the more recent MultiNLI dataset (433k sentence pairs) BID32. Given two sentences, a premise p and a hypothesis q, the task is to determine whether p either entails, contradicts or is neutral to q. See Appendix A.2 for implementation details. Knowledge Source We make use of ConceptNet 2 BID29, a freely-available, multi-lingual semantic network that originated from the Open Mind Common Sense project and incorporates selected knowledge from various other knowledge sources, such as Wiktionary 3, Open Multilingual WordNet 4, OpenCyc and DBpedia 5. It presents information in the form of relational triples. Assertion Retrieval We would like to obtain information about the relations of words and phrases between q and p from ConceptNet in order to strengthen the connection between the two sequences. Because assertions a in ConceptNet come in form of (subject, predicate, object)-triples (s, r, o), we retrieve all assertions for which s appears in q and o appears in p, or vice versa. Because still too many such assertions might be retrieved for an instance, we rank all retrievals based on their respective subject and object. To this end we compute a ranking score which is the inverse product of appearances of the subject and the object in the KB, that is score(a) = DISPLAYFORM0 −1, where I denotes the indicator function. This is very related to the popular idf score (inverted document frequency) from information retrieval which ranks terms higher that appear less frequently across different documents. During training and evaluation we only retain the top-k assertions which we specify for the individual experiments separately. Note that (although very rarely) it might happen that no assertions are retrieved at all. Refinement Order When employing our embedding-refinement strategy, we first read the document (p) followed by the question (q) in case of DQA, and the premise (p) followed by the hypothesis (q) for RTE, that is, X 1 = {p} and X 2 = {q}. Additional knowledge in the form of a set of assertions A is integrated after reading the task-specific input for both DQA and RTE, that is, X 3 = A. In preliminary experiments we found that the final performance is not significantly sensitive to the order of presentation so we decided to fix our order as defined above. Table 1 presents our on two question answering benchmarks. We report on the SQuAD development set 7 and the two more challenging TriviaQA test sets which demonstrate that the introduction of our reading architecture helps consistently with additional gains from using knowledge. Our systems do even outperform current state-of-the-art models on TriviaQA which is surprising given the simplicity of our task-specific architecture and the complexity of the others. For instance, the system of BID9 uses a complex multi-hop attention mechanism to achieve their . Even our baseline BiLSTM + liq system reaches very competitive on TriviaQA which is in line with findings of BID31. To verify that it is not the additional computation that gives the performance boosts when using only our reading architecture (without knowledge), we also ran experiments with 2-layer BiLSTMs (+liq) for our baselines which exhibit similar computational complexity to BiLSTM + reading. We found that the second layer even hurts performance. This demonstrates that pooling over word/lemma occurrences in a given context between layers, which constitutes the main difference to conventional stacked RNNs, is a powerful, yet simple technique. In any case, the most important finding of these experiments is that knowledge actually helps considerably with up to 2.2/2.9% improvements on the F1/Exact measures. TAB3 shows the of our RTE experiments. In general, the introduction of our refinement strategy almost always helps, both with and without external knowledge. When providing additional knowledge from ConceptNet, our BiLSTM based models improve substantially, while the ESIM-based models improve only on the more difficult MultiNLI dataset. Compared to previously published state-of-the-art systems, our models acquit themselves very well on the MultiNLI benchmark, and competitively on the SNLI benchmark. In parallel to this work, BID6 developed a novel task-specific architecture for RTE that achieves slightly better performance on MultiNLI than our ESIM+reading+knowledge based models. 8 It is worth observing that with our knowledge-enhanced embedding architecture, our generic BiLSTM-based task model outperforms ESIM on MultiNLI, which is architecturally much more complex and designed specifically for the RTE task. Finally, we remark that despite careful tuning, our re-implementation of ESIM fails to 2 http://conceptnet.io/ 3 http://wiktionary.org/ 4 http://compling.hss.ntu.edu.sg/omw/ 5 http://dbpedia.org/ 6 We exclude ConceptNet 4 assertions created by only one contributor and from Verbosity to reduce noise. 7 Due to restrictions on code sharing, we are not able to use the public evaluation server to obtain test set scores for SQuAD. However, for the remaining tasks we report both development accuracy and held-out test set performance.8 Our reading+knowledge refinement architecture can be used of course with this new model. Table 3: Development set when reducing training data and embedding dimsensionality with PCA. In parenthesis we report the relative differences to the respective directly above. match the 88% reported in BID4; however, with MultiNLI, we find that our implementation of ESIM performs considerably better (by approximately 5%). The instability of the suggests, as well as the failure of a custom RTE-architecture to consistently perform well suggests that current SotA RTE models may be overfit to the SNLI dataset. We find that there is only little impact when using external knowledge on the RTE task when using a more sophisticated task model such as ESIM. We hypothesize that the attention mechanisms within ESIM jointly with powerful, pre-trained word representations allow for the recovery of some important lexical relations when trained on a large dataset. It follows that by reducing the number of training data and impoverishing pre-trained word representations the impact of using external knowledge should become larger. To test this hypothesis, we gradually impoverish pre-trained word embeddings by reducing their dimensionality with PCA while reducing the number of training instances at the same time. 9 Our joint data and dimensionality reduction are presented in Table 3. They show that there is indeed a slightly larger benefit when employing knowledge in the more impoverished settings with largest improvements over using only the novel reading architecture when using around 10k examples and reduced dimensionality to 10. However, we observe that the biggest overall impact over the baseline ESIM model stems from our contextual refinement strategy (reading) which is especially pronounced for the 1k and 3k experiments. This highlights once more the usefulness of our refinement strategy even without the use of additional knowledge. Table 4: Three examples for the antonym ↔ synonym swapping experiment on MultiNLI. p-premise, h-hypothesis, a-assertion,ā-swapped assertion. Figure 2: Performance differences when ignoring certain types of knowledge, i.e., relation predicates during evaluation. Normalized performance differences are measured on the subset of examples for which an assertion of the respective relation predicate occurs. Is additional knowledge used? To verify whether and how our models make use of additional knowledge, we conducted several experiments. First, we evaluated models trained with knowledge on our tasks while not providing any knowledge at test time. This ablation drops performance by 3.7-3.9% accuracy on MultiNLI, and by 4% F1 on SQuAD. This indicates the model is refining the representations using the provided assertions in a useful way. Are models sensitive to semantics of the provided knowledge? The previous does not show that the models utilize the provided assertions in any consistent way (it may just reflect a mismatch of training and testing conditions). Therefore, to test our models sensitivity towards the semantics of the assertions, we run an experiment in which we swap the synonym with the antonym predicate in the provided assertions during test time. Because of our heuristic retrieval mechanism, not all such "counterfactuals" will affect the truth of the inference, but we still expect to see a more significant impact. The performance drop on MultiNLI examples for which either a synonym or an antonym-assertion is retrieved is about 10% for both the BiLSTM and the ESIM model. This very large drop clearly shows that our models are sensitive to the semantics of the provided knowledge. Examples of prediction changes are presented in Table 4. They demonstrate that the system has learned to trust the presented assertions to the point that it will make appropriate counterfactual inferences-that is, the change in knowledge has caused the change in prediction. What knowledge is used? After establishing that our models are somehow sensitive to semantics we wanted to find out which type of knowledge is important for which task. For this analysis we exclude assertions including the most prominent predicates in our knowledge base individually when evaluating our models. The are presented in Figure 2. They demonstrate that the biggest performance drop in total (blue bars) stems from related to assertions. This very prominent predicate appears much more frequently than other assertions and helps connecting related parts of the 2 input sequences with each other. We believe that related to assertions offer benefits mainly from a modeling perspective by strongly connecting the input sequences with each other and thus bridging long-range dependencies similar to attention. Looking at the relative drops obtained by normalizing the performance differences on the actually affected examples (green) we find that our models depend highly on the presence of antonym and synonym assertions for all tasks as well as partially on is a and derived from assertions. This is an interesting finding which shows that the sensitivity of our models is selective wrt. the type of knowledge and task. The fact that the largest relative impact stems from antonyms is very interesting because it is known that such information is hard to capture with distributional semantics contained in pre-trained word embeddings. The role of knowledge in natural language understanding has long been remarked on, especially in the context of classical models of AI BID27 BID18; however, it has only recently begun to play a role in neural network models of NLU BID0 BID34 BID15 BID5. However, previous efforts have focused on specific tasks or certain kinds of knowledge, whereas we take a step towards a more general-purpose solution for the integration of heterogeneous knowledge for NLU systems by providing a simple, general-purpose reading architecture that can read knowledge encoded in simple natural language statements, e.g., "abdication is a type of resignation". BID1 use textual word definitions as a source of information about the embeddings of OOV words. In the area of visual question answering BID33 utilize external knowledge in form of DBpedia comments (short abstracts/definitions) to improve the answering ability of a model. BID17 explicitly incorporate knowledge graphs into an image classification model. BID34 created a recall mechanism into a standard LSTM cell that retrieves pieces of external knowledge encoded by a single representation for a conversation model. Concurrently, BID5 exploit linguistic knowledge using MAGE-GRUs, an adapation of GRUs to handle graphs, however, external knowledge has to be present in form of triples. The main difference to our approach is that we incorporate external knowledge in free text form on the word level prior to processing the task at hand which constitutes a more flexible setup. BID0 exploit knowledge base facts about mentioned entities for neural language models. BID1 and BID15 create word embeddings on-the-fly by reading word definitions prior to processing the task at hand. BID24 seamlessly incorporate information about word senses into their representations before solving the downstream NLU task, which is similar. We go one step further by seamlessly integrating all kinds of fine-grained assertions about concepts that might be relevant for the task at hand. Another important aspect of our approach is the notion of dynamically updating wordrepresentations. Tracking and updating concepts, entities or sentences with dynamic memories is a very active research direction BID14 BID7 BID10 BID13. However, those works typically focus on particular tasks whereas our approach is taskagnostic and most importantly allows for the integration of external knowledge. Other related work includes storing temporary information in weight matrices instead of explicit neural activations (such as word representations) as a biologically more plausible alternative. We have presented a novel task-agnostic reading architecture that allows for the dynamic integration of knowledge into neural NLU models. Our solution, which is based on the incremental refinement of word representations by reading supplementary inputs, is flexible and be used with virtually any existing NLU architecture that rely on word embeddings as input. Our show that embedding refinement using both the system's text inputs, as well as supplementary texts encoding knowledge can yield large improvements. In particular, we have shown that relatively simple task architectures (e.g., based on simple BiLSTM readers) can become competitive with state-of-the-art, task-specific architectures when augmented with our reading architecture. In the following we explain the detailed implementation of our two task-specific, baseline models. We assume to have computed the contextually (un-)refined word representations depending on the setup and embedded our input sequences q = (q 1, ..., q L Q) and p = (p 1, ..., p L P) to Q ∈ R n×L Q and P ∈ R n×L P, respectively. The word representation update gate in Eq. 4 is initialized with a bias of 1 to refine representations only slightly in the beginning of training. In the following as before, we denote the hidden dimensionality of our model by n and a fully-connected layer by DISPLAYFORM0 A.1 QUESTION ANSWERING Encoding In the DQA task q refers to the question and p to the supporting text. At first we process both sequences by identical BiLSTMs in parallel (Eq. 6) followed by separate linear projections (Eq. 7).Q = BiLSTM(Q)P = BiLSTM(P)Q ∈ R 2n×L Q,P ∈ R 2n×L P DISPLAYFORM1 U Q, U P ∈ R n×2n are initialized by [I; I] where I ∈ R n×n is the identity matrix. Prediction Our prediction-or answer layer is the same as in BID31. We first compute a weighted, n-dimensional representationq of the processed questionQ (Eq. 8). DISPLAYFORM2 The probability distribution p s /p e for the start/end location of the answer is computed by a 2-layer MLP with a ReLU activated, hidden layer s j as follows: DISPLAYFORM3 The model is trained to minimize the cross-entropy loss of the predicted start and end positions, respectively. During evaluation we extract the span (i, k) with the best span-score p s (i) · p e (k) of maximum token length k − i = 16. Encoding Analogous to DQA we encode our input sequences by BiLSTMs, however, for RTE we use conditional encoding BID26 instead. Therefore, we initially process the embedded hypothesis Q by a BiLSTM and use the respective end states of the forward and backward LSTM as initial states for the forward and backward LSTM that processes the embedded premise P.Prediction We concatenate the outputs of the forward and backward LSTMs processing the premise p, i.e., p f w t;p bw t ∈ R 2n and run each of the ing L P outputs through a fullyconnected layer with ReLU activation (h t) followed by a max-pooling operation over time ing in a hidden state h ∈ R n. Finally, h is used to predict the RTE label as follows: DISPLAYFORM0 The probability of choosing category c ∈ {entailment, contradiction, neutral} is defined in Eq. 10. Finally, the model is trained to minimize the cross-entropy loss of the predicted category probability distribution p. As pre-processing steps we lowercase all inputs and tokenize it. Additionally, we make use of lemmatization as described §3.2 in which is necessary for matching. As pre-trained word representations we use 300-dimensional word-embeddings from Glove BID23. We employed ADAM BID12 for optimization with an initial learning-rate of 10 −3 which was halved whenever the F1 measure (DQA) or the accuracy (RTE) dropped on the development set between 1000/2000 minibatches for DQA and RTE respectively. We used mini-batches of size 16 for DQA and 64 for RTE. Additionally, for regularization we make use of dropout with a rate of 0.2 on the computed non-contextual word representations e w defined in §3.1 with the same dropout mask for all words in a batch. All our models were trained with 3 different random seeds and the top performance is reported.

Represent the TLDR for finding the corresponding scientific article

In this paper we present a task-agnostic reading architecture for the dynamic integration of explicit background knowledge in neural NLU models.

represent the tldr for finding the corresponding scientific article.