Question: represent the article for finding a single-sentence tldr

Question: Comparing the inferences of diverse candidate models is an essential part of model checking and escaping local optima. To enable efficient comparison, we introduce an amortized variational inference framework that can perform fast and reliable posterior estimation across models of the same architecture. Our Any Parameter Encoder (APE) extends the encoder neural network common in amortized inference to take both a data feature vector and a model parameter vector as input. APE thus reduces posterior inference across unseen data and models to a single forward pass. In experiments comparing candidate topic models for synthetic data and product reviews, our Any Parameter Encoder yields comparable posteriors to more expensive methods in far less time, especially when the encoder architecture is designed in model-aware fashion. We consider the problem of approximate Bayesian inference for latent variable models, such as topic models , embedding models , and dynamical systems models . An important step in using such probabilistic models to extract insight from large datasets is model checking and comparison. While many types of comparison are possible , we focus on a problem that we call within-model comparison. Given several candidate parameter vectors θ 1, θ 2,..., all from the same space Θ ⊆ R D, our goal is to efficiently determine which parameter θ m is best at explaining a given dataset of N examples {x n} N n=1. Multiple ways exist to rank candidate parameters, including performance on heldout data or human-in-the-loop inspection. A principled choice is to select the parameter that maximizes the data's marginal likelihood: N n=1 log p(x n |θ m). For our latent variable models of interest, computing this likelihood requires marginalizing over a hidden variable h n: p(x n |θ m) = p(x n |h n, θ m)p(h n |θ m)dh n. This integral is challenging even for a single example n and model m. One promising solution is variational inference (VI). Using VI, we can estimate an approximate posterior q(h n |x n, θ m) over hidden variables. Approximate posteriors q can be used to compute lower bounds on marginal likelihood, and can also be helpful for human inspection of model insights and uncertainties. However, it is expensive to estimate a separate q at each example n and model m. In this paper, we develop new VI tools 1 that enable rapid-yet-effective within-model comparisons for large datasets. The need for within-model comparison (and our methods) is present in many practical modeling tasks. Here we discuss two possible scenarios, with some details specialized to our intended topic modeling applications . First, in human-in-the-loop scenarios, a domain expert may inspect some estimated parameter θ and then suggest an alternative parameter θ that improves interpretability. In topic modeling, this may mean removing "intruder words" to make topics more coherent . Second, in automated parameter learning scenarios, many algorithms propose data-driven transformations of the current solution θ into a new candidate θ, in order to escape the local optima common in non-convex optimization objectives for latent variable models , Examples include split-merge proposal moves or evolutionary algorithms . Across both these scenarios, new candidates θ arise repeatedly over time, and estimating approximate posteriors for each is essential to assess fitness yet expensive to perform for large datasets. Our contribution is the Any Parameter Encoder (APE), which amortizes posterior inference across models θ m and data x n. We are inspired by efforts to scale a single model to large datasets by using an encoder neural network (NN) to amortize posterior inference across data examples . Our key idea is that to additionally generalize across models, we feed model parameter vector θ m and data feature vector x n as input to the encoder. APE is applicable to any model with continuous hidden variables for which amortized inference is possible via the reparameterization trick. We consider a general family of probabilistic models that use parameter vector θ m to generate a dataset of N continuous hidden variables h n and observations x n via a factorized distribution: N n=1 p(h n |θ m)p(x n |h n, θ m). Our goal is fast-yet-accurate estimation of each example's local posterior p(h n |x n, θ m) for a range of model parameters θ 1, θ 2,... ∈ Θ. Topic Models. As a sample application, we focus on the Logistic Normal topic model from. Given known vocabulary size V, we observe N documents represented by count vectors x n (vector of size V counting the types of all T n words in document n). We model each x n as a mixture of K possible topics. Let hidden variable h nk be the probability that a word in document n is produced by topic k. Thus, h n ∈ ∆ K is a non-negative vector of size K that sums to one. We model h n with a Logistic Normal prior, with mean and covariance set to be similar to a sparse Dirichlet(0.01) prior for interpretability. Given h n, we model the observed word-count vector for document n with a Multinomial likelihood:. This is a document-specific mixture of topics, where each topic k is defined by a word probability vector θ k ∈ ∆ V. Our parameter of interest is the topic-word probability vector θ = {θ k} K k=1. VI Approximations for the Single Example Posterior. While the true posterior p(h n |x n, θ m) is usually intractable, we use variational inference (VI) to approximate it. We choose a simpler density q(h n |λ n) and optimize parameter λ n to minimize KL divergence from the true posterior. Inference reduces to the well-known evidence lower bound (ELBO) optimization problem given example x n and model θ m: Inference: Given several parameters of interest, we can perform model comparison by solving the above optimization problem separately for each θ m. However, this is expensive. Solving Eq. for a model θ m requires dozens of iterative updates of gradient ascent for each example. VI Amortized across Data Examples. and have sped up inference by setting per-example variational parameters λ n to the output of an encoder neural network (NN) instead of an iterative optimization procedure. The "Standard" encoder, with weights parameters φ, takes input data x n and produces variational parameters λ NN φ (x n). Inference for example n reduces to one fast forward pass: λ n ← λ NN φ (x n). While encoders often produce λ n with worse ELBO scores than optimal solutions to Eq. , they are preferred for their speed. However, for our model comparison goals the standard encoder is expensive, because for each parameter θ m of interest we must train separate specialized NN weights φ m. Contribution: VI Amortized over Model Parameters. Our goal is to enable rapid estimation of posteriors p(h n |x n, θ m) for many possible parameters θ 1, θ 2,... ∈ Θ (not all known in advance). We thus consider a family of approximating densities q that explicitly conditions on both a given data vector x n and the query parameter vector θ m. Again, we use a neural network to transform these inputs into the variational parameters, λ n ← λ NN φ (x n, θ m). We call this the Any Parameter Encoder. Unlike the earlier Standard Encoder, which trains φ for one specific θ, our approach can directly generalize to many θ. Encoder Architecture Design for Topic Models. Given the difficulty of posterior inference even for a single parameter θ, developing an effective Any Parameter Encoder requires careful selection of a NN architecture that can transform its two inputs, data x n and model θ, to produce accurate approximate posteriors. Following previous work , we use multi-layer perceptrons. We further suggest that an architecture designed to capture structure in the generative model should improve further. Our baseline "naive" architecture defines the input of the neural net as simply the concatenation of vector x n and vector θ. While simple, we suggest this will be difficult to train effectively given the size of the input ((K + 1)V for the topic model) and lack of inductive bias to prioritize the model's needed interactions between entries of x n and θ. As an improvement, we consider a model-aware encoder architecture. Our design is motivated by a view of posterior inference as roughly moment-matching when data is plentiful. For our topic model, each document's Multinomial likelihood has a mean vector equal to k h nk θ k = θh n, writing θ as a V × K matrix. This mean vector should be (roughly) equal to the observed word-frequency vector 1 Tn x n. If µ n is the mean of q(h n) and used as a plug-in estimate for h n, then we want to satisfy 1 Tn x n ≈ θµ n. Solving for µ n via least squares, we get µ n ≈ 1 Tn (θ T θ) −1 θ T x n, which we might simplify to a non-linear function of θ T x n. Thus, we suggest using the following model-aware encoder architecture: This model-aware architecture has encoder input dimension K, which is much smaller than (K + 1)V for the naive approach (and thus hopefully easier to train). Furthermore, this should provide desirable inductive bias to produce useful mean and covariance estimates. We emphasize that this design is specialized to the topic model, and further work is needed to develop model-aware architecture design strategies for general latent variable models. Training the Encoder. Training our encoder parameters φ requires an available set of M parameter vectors {θ m} M m=1 of interest. We choose these to be representative of the subset of Θ we wish to generalize well to. We then maximize ELBO across all M models: We use stochastic gradient ascent to solve for φ, using the reparameterization trick to estimate gradients for a minibatch of examples and models at each step. We can interpret this objective as an expectation over samples θ m from a target distribution over parameters. We compare our proposed Any Parameter Encoder (APE) to several other inference methods on two topic modeling tasks. For all VI methods, we choose q to be a Logistic Normal parameterized by a mean and a diagonal covariance. The appendix has complete details. APE. We consider both naive and model-aware encoder architectures described above. Both use MLPs with 2 layers with 100 units per hidden layer, selected via grid search. Baselines. We consider three baselines implemented in Pyro and PyTorch . First, Variational Inference (VI) uses gradient ascent to optimize Eq.. Second, we use Standard encoder VAEs for topic models . This encoder is specialized to a single parameter θ, with architecture size selected via grid search (similar to APE). Finally, we run Pyro's off-the-shelf implementation of Hamiltonian Monte Carlo with the No U-Turn Sampler (NUTS) , though we expect specialized implementations to be more performant. Synthetic Data Experiments. We consider a V = 100 vocabulary dataset inspired by the "toy bars" of. Using K = 20 true topics θ *, we sample Error from VI ELBO Product Reviews Figure 1: Left: ELBO vs. elapsed time for VI on a test set of 300 document-θ m combinations on synthetic data. We show a randomly-initialized run (black) and a warm-started run (green) initialized via our Any-Parameter Encoder (red "X"). The randomly-initialized VI would require over 400 milliseconds (vertical line) to reach the quality our APE achieved in <20 ms. Right: Kernel Density Estimation of absolute difference between encoder ELBO and VI ELBO across different topics. APE (red) are closer to VI (i.e. less error). a 500-document dataset. We consider M = 50, 000 possible model parameters {θ m} M m=1, sampled from a symmetric, sparse Dirichlet prior over the vocabulary. Typical θ m look unlike the true topics θ *, as shown in the supplement, so inference must handle diversity well. We train our APE on 25 million possible document-θ pairs for two epochs, then evaluate on unseen document-θ pairs drawn from the same generative process. Product Reviews. We model 6,343 text documents of consumer product reviews . We use the V = 3000 most frequent vocabulary terms and K = 30 topics. We generate training topics in the same way as in the synthetic data experiments, and we evaluate on test topics found via Gibbs sampling with several separately initialized runs. Results: Encoder Design. Results comparing naive and model-aware encoder architectures are in Table 1. Our proposed model-aware input layer yields better heldout likelihoods than the naive alternative, which we suggest is due to its more effective inductive bias. Results: Quality-vs-Time Tradeoff. Comparing across Table 1 and Fig. 1, we see that while the Standard Encoder understandably fails to generalize across models, our Any Parameter Encoder achieves quality close to non-amortized VI and NUTS with a speed up factor of over 100-1000x. APE can also provide a useful warm start initialization to VI. Results: Agreement in model comparison. Motivated by the need to rapidly assess proposal moves that escape local optima, we gather 10 different models and measure whether each encoder's ranking of a pair θ, θ on the test set agrees with VI's ranking. Table 1 shows that APE agrees with VI in 75% of 45 cases in the real data scenario, while Standard Encoder agrees just 29% of the time. This suggests APE may be trustworthy for accept/reject decisions, though further work is needed to improve this number further. Across two datasets and many model parameters, our Any Parameter Encoder produces posterior approximations that are nearly as good as expensive VI, but over 100x faster. Future opportunities include simultaneous training of parameters and encoders, and handling Bayesian nonparametric models where θ changes size during training (n)) For encoder methods, the parameters {µ n, log σ 2 n} are the output of a shared encoder NN. For VI, these are free parameters of the optimization problem. Variational Inference (VI). We perform using gradient ascent to maximize the objective in Eq., learning a per-example mean and variance variational parameter. We run gradient updates until our moving average loss (window of 10 steps) has improved by less than 0.001% of its previous value. For our VI runs from random initializations, we use the Adam optimizer with an initial learning rate of.01, decaying the rate by 50% every 5000 steps. For our warm-started runs, we use an initial learning rate of 0.0005. In practice, we ran VI multiple times with different learning rate parameters and took the best one. Table 1 only reports the time to run the best setting, not the total time which includes various restarts. Standard encoder. We use a standard encoder that closely matches the VAE for topic models in. The only architectural difference is the addition of a temperature parameter on the µ n vector before applying the softmax to ensure the means lie on the simplex. We found that the additional parameter sped up training by allowing the peakiness of the posterior to be directly tuned by a single parameter. We use a feedforward encoder with two hidden layers, each 100 units. We chose the architecture via hyperparameter sweeps. The total number of trainable parameters in the model is 24,721 on the synthetic data and 316,781 on the real data; this is compared to 16,721 and 19,781 parameters for model-aware APE. NUTS. For the Hamiltonian Monte Carlo (HMC) with the No U-Turn Sampler (NUTS) , we use a step size of 1 adapted during the warmup phase using Dual Averaging scheme. Upon inspection, we find that the method's slightly lower posterior predictive log likelihood relative to VI is due to its wider posteriors. We also find that the Pyro implementation is (understandably) quite slow and consequently warm-start the NUTS sampler using VI to encourage rapid mixing. We are aware that there exist faster, more specialized implementations, but we decided to keep our tooling consistent for scientific purposes. We generate a set of different models {θ 0, θ 1, ...θ M} from a symmetric Dirichlet prior with α = 0.1. We train our Any-Parameter Encoder in random batches of document-topic combinations. With 500 documents and 50,000 topics (i.e. D = 500, M = 50, 000), we have 25 million combinations in total. The topics used to generate the synthetic data represent "toy bars", inspired by . See Figure 2 for a visualization. We use this same toy bars-biased prior to generate all our topics in the holdout set, though the order of the topics is random. See For training both APE and the Standard encoder on the synthetic data, we use Adam with an exponential decay learning schedule, a starting learning rate of 0.01, and a decay rate of.8 every 50,000 steps. We find that this schedule tends to be fairly robust; these hyperparameters were used for both APE and the Standard encoder on both the synthetic and real data. We chose our initial learning rate via a learning rate finder posed in , and we train for 2 epochs with a batch size of 100. We train our standard VAE encoder on a single model with parameters θ drawn randomly from a symmetric Dirichlet prior with α = 0.1. To train the standard encoder, we pass in our model of interest to the decoder, holding its weights fixed as we perform stochastic backpropagation to update the encoder weights. The same thing happens for APE, though the same topics are additionally included as part of the input into the encoder. We develop VAEs where the encoder takes a model parameter vector as additional input, so we can do rapid inference for many models

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We develop VAEs where the encoder takes a model parameter vector as input, so we can do rapid inference for many models

We use continuous time dynamics to define a generative model with exact likelihoods and efficient sampling that is parameterized by unrestricted neural networks.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The ability to transfer knowledge to novel environments and tasks is a sensible desiderata for general learning agents. Despite the apparent promises, transfer in RL is still an open and little exploited research area. In this paper, we take a brand-new perspective about transfer: we suggest that the ability to assign credit unveils structural invariants in the tasks that can be transferred to make RL more sample efficient. Our main contribution is Secret, a novel approach to transfer learning for RL that uses a backward-view credit assignment mechanism based on a self-attentive architecture. Two aspects are key to its generality: it learns to assign credit as a separate offline supervised process and exclusively modifies the reward function. Consequently, it can be supplemented by transfer methods that do not modify the reward function and it can be plugged on top of any RL algorithm. To some, intelligence is measured as the capability of transferring knowledge to unprecedented situations. While the notion of intellect itself is hard to define, the ability to reuse learned information is a desirable trait for learning agents. The coffee test , presented as a way to assess general intelligence, suggests the task of making coffee in a completely unfamiliar kitchen. It requires a combination of advanced features (planning, control and exploration) that would make the task very difficult if not out of scope for the current state-of-the-art Reinforcement Learning (RL) agents to learn. On the other hand, it is solved trivially by humans, who exploit the universally invariant structure of coffee-making: one needs to fetch a mug, find coffee, power the coffee machine, add water and launch the brewing process by pushing the adequate buttons. Thus, to solve the coffee test, transfer learning appears necessary. Were we to possess a random kitchen simulator and a lot of compute, current transfer methods would still fall short of consistently reusing structural information about the task, hence also falling short of efficient adaptation. Credit assignment, which in RL refers to measuring the individual contribution of actions to future rewards, is by definition about understanding the structure of the task. By structure, we mean the relations between elements of the states, actions and environment rewards. In this work, we investigate what credit assignment can bring to transfer. Encouraged by recent successes in transfer based on supervised methods, we propose to learn to assign credit through a separate supervised problem and transfer credit assignment capabilities to new environments. By doing so, we aim at recycling structural information about the underlying task. To this end, we introduce SECRET (SElf-attentional CREdit assignment for Transfer), a transferable credit assignment mechanism consisting of a self-attentive sequence-to-sequence model whose role is to reconstruct the sequence of rewards from a trajectory of agent-environment interactions. It assigns credit for future reward proportionally to the magnitude of attention paid to past state-action pairs. SECRET can be used to incorporate structural knowledge in the reward function without modifying optimal behavior, as we show in various generalization and transfer scenarios that preserve the structure of the task. Existing backward-view credit assignment methods require to add auxiliary terms to the loss function used to train agents, which can have detrimental effects to the learning process (de), and rely on an external memory, which hinder the generality of their approach. SECRET does neither. Also, as we show in Sec. 3.1, the architecture we consider for SECRET has interesting properties for credit assignment. We elaborate about our novelty with respect to prior work in Sec. 4. We insist on the fact that the focus of our work is on transfer and that it is not our point to compete on credit assignment capabilities. We would like to emphasize several aspects about the generality of SECRET: 1) our method does not require any modification to the RL algorithm used to solve the tasks considered, 2) it does not require any modification to the agent architecture either and 3) it does not alter the set of optimal policies we wish to attain. Moreover, our method for credit assignment is offline, and as a , it can use interaction data collected by any mean (expert demonstrations, replay memories , backup agent trajectories... ). We believe that this feature is of importance for real-world use cases where a high number of online interactions is unrealistic but datasets of interactions exist as a byproduct of experimentation. Background We place ourselves in the classical Markov Decision Process (MDP) formalism . An MDP is a tuple (S, A, γ, R, P) where S is a state space, A is an action space, γ is a discount factor (γ ∈), R: S × A × S → R is a bounded reward function that maps state-action pairs to the expected reward for taking such an action in such a state. Note that we choose a form that includes the ing state in the definition of the reward function over the typical R: S × A → R. This is for consistency with objects defined later on. Finally, P: S × A → ∆ S is a Markovian transition kernel that maps state-action pairs to a probability distribution over ing states, ∆ S denoting the simplex over S. An RL agent interacts with an MDP at a given timestep t by choosing an action a t ∈ A and receiving a ing state s t+1 ∼ P (·|s t, a t) and a reward r t = R(s t, a t, s t+1) from the environment. A trajectory τ = (s i, a i, r i) i=1,...,T is a set of state-action pairs and ing rewards accumulated in an episode. A subtrajectory is a portion of trajectory that starts at the beginning of the episode. The performance of an agent is evaluated by its expected discounted cumulative reward E ∞ t=0 γ t r t. In a partially observable MDP (POMDP), the agent receives at each timestep t an observation o t ∼ O(·|s t) that contains partial information about the underlying state of the environment. 2 SECRET: SELF-ATTENTIONAL CREDIT ASSIGNMENT FOR TRANSFER SECRET uses previously collected trajectories from environments in a source distribution. A selfattentive sequence model is trained to predict the final reward in subtrajectories from the sequence of observation-action pairs. The distribution of attention weights from correctly predicted nonzero rewards is viewed as credit assignment. In target environments, the model gets applied to a small set of trajectories. We use the credit assigned to build a denser and more informative reward function that reflects the structure of the (PO)MDP. The case where the target distribution is identical to the source distribution (in which we use held-out environments to assess transfer) will be referred to as generalization or in-domain transfer, as opposed to out-of-domain transfer where the source and the target distributions differ. Credit assignment as offline reward prediction We learn to assign credit through an offline reward prediction task, based on saved trajectories of agent-environment interactions. We create a sequence-to-sequence (seq2seq) model ) that takes as input the sequence of observation-action pairs and has to reconstruct the corresponding sequence of environment rewards. Being offline, the reward prediction task is learned separately from the RL task, and the reward prediction model does not share representations with the agent. This way, the representations learned for credit assignment do not affect or get mixed with the representations learned for control. Operating offline brings several advantages: one can directly interact with the replay memory of agents and even use expert demonstrations or arbitrary saved transitions as a source of supervision, which could be useful in settings where on-policy interactions are costly, such as robotics. We equip our seq2seq model with an attention mechanism and view the attention weights of the reward reconstruction task as our primary source of assigned credit. The motivation to do so is that the seq2seq model looks into the past to find predictive signal in order to reconstruct the reward, so observation-action pairs it attends to should be those which reduce its uncertainty about the future, in other words those that explain future reward and should be credited. On the use of observations In MDPs, environment states follow the Markov property: they summarize the history of previous interactions and are sufficient to predict the future. As such, predictive models are highly biased towards focusing on the current sequence element, which hinders credit assignment. Under that consideration, when dealing with MDPs, we turn states into observations by applying transformations that hide a certain amount of information from states and break the Markov assumption. For instance, in gridworlds with visual states, we crop the image and get a player centered image with a given window size. Doing so encourages the model to look into the past to find predictive signal, and allow us to track the relative importance given to each element to reconstruct the credit assigned. In POMDPs, this might be unnecessary depending on the amount of information shared between observations and true states. Self-attention for credit assignment Unlike other seq2seq architectures, self-attentive models like Transformers have direct computational paths between pairs of sequence elements, due to their representations that depend on projections of all sequence elements. This feature is key to long-term credit assignment. As an illustration, consider an RL task where the terminal reward depends only on the first observation, which is drawn randomly. Predicting the reward correctly requires to remember the first observation, which would be very challenging for a recurrent architecture whose memory goes through O(n) transformations, n being the size of the sequence. On the other hand, a self-attentive model directly accesses the value of the initial observation, which makes credit assignment easier. Reward prediction architecture We use a Transformer decoder with a single self-attention layer and a single attention head. The model input is a sequence of observationaction couples (o t, a t) t=0,...,T. Each observation goes through a series of convolutional layers (for visual inputs) followed by a series of feed-forward layers. Each action representation, a one-hot vector in the discrete action case, is concatenated to the learned observation embedding. Those representations of dimensionality d i are combined with positional encoding (PE), fed to a self-attention layer and then to a position-wise feedforward layer that outputs logits for reward prediction classes. PE encodes the relative positions of sequence elements, see Appendix A for details. Self-attention is an attention mechanism with parameterization (W k, W q, W v), each matrix belonging to R di×d k, that puts sequence elements in relation by computing non-linear similarity scores for all pairs of elements in the sequence. To do so, each sequence element is mapped to a query vector that is matched against keys and values obtained from the previous elements. To be consistent with the goal of assigning credit, the model should not be able to peek into the future. Thus, we restrict the computational window of each sequence element to the information stored in representations of the previous elements in the sequence and its own by applying a causal mask M c to the of the pairwise similarity computations, assigning a value of 0 to masked elements after the softmax. Let X = (x t) t=0,...,T ∈ R T ×di denote the input sequence in a matrix form, x t being the of internal computations of the model on its t th input. In the same fashion, we note Z = (z t) t=0,...,T ∈ R T ×d k the sequence ing from the application of self-attention. We then have where Q = XW q ∈ R T ×d k stores queries, K = XW k ∈ R T ×d k keys, and V = XW v ∈ R T ×d k values as linear projections of the input; d k stands for the dimension of the key vectors, M c ∈ {0, 1} T ×T is a binary matrix that acts as a causal mask (a lower triangular matrix), is the Hadamard product and C is a large constant (10 9 in practice). Notably, the ing observation-action representation can be viewed as a linear combination of the values of previous elements: The vector α t contains the normalized attention weights for the prediction at timestep t and sums to 1. Since observations contain only a portion of their initial information, the fact that the model succeeds in the prediction task indicates that it reconstructed the missing information from its past. Therefore, attention weights themselves can be viewed as a form of credit assignment, and will be used as such in what follows. While performing regression on the rewards could also be an option, our experiments found that regression tends to converge to poor local optima. Consequently, we predict the sign of the experi-enced rewards: q(r) = sign(r) with sign = 0. We chose the sign as the classification target for its invariance to the scale of the rewards. We use a weighted sequential cross-entropy as the loss function over the class-wise model predictions f θ,c, writing τ (o, a) the subtrajectory of τ ending with the observation-action couple (o, a) to translate the effect of the binary mask: We have found class weighting to be very important to keep the variance of prediction metrics across a variety of datasets of sampled trajectories low. Generating trajectories To train SECRET, we generate a dataset of trajectories which contains a certain proportion of successful trajectories. If source environments are simple enough so that the task has sufficient chance to be solved by acting randomly, we use a random policy to generate trajectories. For more complex distributions of environments, we use a RL agent (either trained or in the learning phase) to generate trajectories. We think purely exploratory methods like could have advantages over using an RL agent and leave the study of their use for future work. In this subsection, we explain how we use credit assignment to make learning more sample-efficient. Reward shaping In RL, agents often deal with sparse rewards that make the learning process slow. Reward shaping is a technique that aims at densifying the reward so as to improve sample efficiency. It defines a class of reward functions that can be added to the original environment rewards without modifying the set of optimal policies. For a given MDP M = (S, A, γ, R, P), we define a new MDP M = (S, A, γ, R, P) where R = R + F is the shaped reward and F the shaping. The reward shaping theorem states that if there exists a function φ such that F: (s, a, s) → γφ(s) − φ(s), then M and M admit the same set of optimal policies. φ is called a potential function. With domain knowledge, one can use reward shaping to design more informative reward functions without encouraging unwanted behavior. Nevertheless, shaping rewards requires good priors for the task and the potential function must often be engineered manually. Since SECRET weighs the contribution of observation-action pairs to future reward, we use it to derive a shaped reward that corresponds to the sum of future reward reachable from the underlying state, weighted by the attention calculated by the model. We explain the process in the following. Computing the potential function We define the redistributed return R ← τ of a trajectory τ as: where α i←j is the attention weight on (o i, a i) when predicting the reward r j and s i are environment states. Indeed, SECRET uses observations but we keep the states they are constructed from to compute the potential. In POMDPs, we recover an approximate state from the observation, either manually or through inference. In this work, we use a state constructed manually, see Appendix A for details. To compute the potential function, we generate a set D of trajectories like described in Sec. 2.1. Since we operate on trajectories, the same state-action pair can appear twice in a sequence and benefit from a different amount of attention, which is why we must include the first summation. In the reward shaping formalism, the potential function φ depends only on the state. To stay within its bounds, we define φ as the forwarded redistributed return. It is computed as the following estimate: Note that in practice we only redistribute individual rewards that were successfully predicted. Also, some states are generally missing from the data distribution induced by the set of trajectories used. For those states, we set to potential to 0, which in a −φ(s) additional reward when transitioning to those from the state s. As a , it gives agents incentive to stay on the support of the data distribution unless they encounter high-reward states. Because it relies on reward shaping, SECRET conserves optimal policies. We empirically find that agents learn faster with the ing augmented reward function. A way to look at it is that we densify the learning signal and bias the agent towards behaviors that encourage future rewards. We start by conveying intuition as to why SECRET should transfer to new environments. In fields other than RL, seq2seq models similar to that of SECRET have shown outstanding transfer capabilities (; ;), even in low-resource settings . In transfer scenarios that preserve the structure of the MDP, the optimal finegrained control sequence can vary drastically from one environment to another. This is why credit assignment is an interesting alternative to the transfer of weights: given an underlying environment state and a specific action, their contribution to future rewards is not fundamentally altered. Such scenarios include specific changes in the state (or observation) distribution and changes to the reward function that preserve the optimal policies. These also include changes in the dynamics of the environment, and though it affects credit assignment, we show later on that SECRET adapts surprisingly well to such scenarios. Another point that motivates the use of our method for transfer is the fact that we keep the representations learned for credit assignment separate from the control representations learned by agents. Indeed, de showed that RL representations were not optimal for transfer. Transfer setting We argue that transfer should be considered effective when agents learn to solve target tasks efficiently because efficiency gains in the target domain compound while the cost of training in the source is fixed. Hence, we use the Total Target Time Scenario metric to assess transfer. Nevertheless, collecting trajectories in the source domain can be costly. We report the number of trajectories used to train SECRET in each scenario. As before, SECRET is trained on episodes of interaction sampled from the source distribution. In each target environment, we sample multiple trajectories (see the following section for details about the policies used to generate the trajectories). We then compute the attentional potential function by calculating an estimate of the expected redistributed reward, as described in Sec. 2.2. In this section, we aim to answer the following questions: can SECRET improve the sample efficiency of learning for RL agents? Does it generalize and/or transfer? How does it compare to transfer baselines? Is the credit assigned by SECRET interpretable? The of complementary experiments are presented in Appendix B. The Triggers environment We introduce Triggers, an interpretable and customizable environment that we use to assess the quality of the credit inferred with our method. In Triggers, the agent is located in a two-dimensional bounded grid. Its actions consist solely of moving of one cell in one of the cardinal directions. Any action that would lead the agent outside the boundaries of the environment (as indicated by the walls in the figure) is ignored but still counted as an action taken by the agent. The goal of the agent (represented as a yellow square) is to activate all the switches (red squares) and then collect all the prizes (pink squares). Prizes are the only source of reward and give a −1 penalty unless all switches are activated, in which case they give a +1 bonus. Both prizes and switches disappear once collected. The main feature of Triggers is that every positive reward is conditional to the presence of a known subset of states in the agent history, and thus credit assignment can be assessed in a rigorous way. In that setup, agents benefit from understanding the link between keys and doors. We hypothesized that our credit assignment mechanism might identify this relation and reward the agent for picking up keys. To assert this, we modified the setting so that picking up keys does not provide rewards. Additionally, the visual input is richer than the one from Triggers environments and the average number of steps per episode is extended. Finally, agents move and rotate across the room. Since picking up a key does not require to actually see the key, it can be hard to know whether a key was taken and predict further door opening rewards. Trajectories are generated with a trained agent. Agents used We use tabular Q-learning for in-domain and out-ofdomain experiments in Triggers except for the transfer to environments with modified dynamics where we use Deep Q-Networks (DQN) . We use Proximal Policy Optimization (PPO) agents for in-domain experiments in DMLab. We provide an analysis of the credit inferred by SECRET. The analysis is qualitative and quantitative, since we rely on both visual assessment and binary detection metrics. The process of evaluating credit assignment in Triggers goes as follows: we first generate trajectories and train the model. We then compare the credit assigned by SECRET on trajectories sampled from held-out environments to a ground truth credit assignment. We build that ground truth by exploiting the exact knowledge of where triggers are. It is a vector that is 0 almost everywhere and 1 on state-action couples that precede the activation of a Triggers. By doing so, we explicitly target the state-action couples whose ing state is causally linked to the reward experienced later. We find the redistribution to be near optimal in simple instances of Triggers (see Fig. 3 -left): attention concentrates quasi exclusively on state-action pairs that enable the collection of future reward. This is confirmed by precision-recall analysis: over the distribution of scenarios considered, a simple binarization heuristic over attention values yields an average precision of 0.96 for an average recall of 0.94. More information on the heuristic is in Appendix A. In keys_doors_puzzle, we adopt the same set of experiments. Since the agent can move backward and spin, in some scenarios it takes a key that is not in is line of sight. In addition, the granularity of the state space is such that off-by-one prediction errors are common but do not hinder the credit mechanism: attributing credit to the state-action couple preceding the collection of a key or the previous one leads to imperceptible changes in the ing shaped rewards. Fig. 3 -right shows similar as for Triggers. Appendix A also provides a heatmap for this task that shows that attention concentrates around the keys. We then study how we can leverage the inferred credit and transfer representations that are helpful in new scenarios. We show that agents train faster when using shaped rewards from SECRET. As before, the reward model is trained on episodes of interaction in environments sampled from the source distribution. In transfer environments, we sample multiple trajectories, each using the same maze configuration. We then compute the attentional potential function by calculating an estimate of the expected redistributed reward, as described in Sec. 2.2. To evaluate its effect, we compare agents trained from environment rewards to agents that use the ing shaped reward. In-domain transfer For in-domain transfer, we transfer the representations for credit assignment to held-out instances of the same distribution over MDPs. For the Triggers environment, the RL agents are tabular Q-learners. For the DMLab environment, we use PPO agents and modify the original task: we do not reward the agent for collecting keys but only to open doors so that the attention can focus on the key positions. Note that this makes the task harder. As we display in Fig. 4 agents learn visibly faster to solve tasks when benefitting from SECRET in both environments. Out-of-domain transfer For out-of-domain transfer we use the Triggers environment and consider two scenarios that are hard for standard agents: transfer to bigger environments (see Fig. 5) and transfer to environments with inverted dynamics (see Fig. 6). In the bigger setting, direct weight transfer cannot be used since the visual input has bigger spatial dimensions. On the other hand, SECRET can be used since the transformation we apply to turn states into observations conserves the visual input dimensions. In the inverted dynamics setting, the effect of the agent's actions are inverted, which makes the task hard for transfer methods. In that setting, we compare the transferability figure). The effect of the shaping is exclusively beneficial, while transferring weights from the source task can be detrimental to the learning process. of our mechanism to that of the representations learned by an agent equipped with deep function approximation. To this end we use DQN agents and either train them from scratch in the target environments or start from the set of weights learned in the source environments (WT in Fig. 6). In both settings, shaping the rewards assists the agent in learning to solve the task. We display some in Fig. 5 and Fig. 6. When transferring to bigger environments, the agent benefits very early on from the shaped reward, while also reaching better asymptotical performance. Transfer in RL While a lot of approaches exist in the transfer literature, to the best of our knowledge none explicitly transfer credit assignment capabilities. Previous works aimed at making the training of an agent in the same task more sample-efficient by using a pretrained model as a teacher (a;). We learn to assign credit as a parallel task that does not modify the representations of the RL agent. Others learn auxiliary reward functions in the hope that they will enable transfer by imposing consistency in the reward (; ;). Although we also learn additional reward signal, it is based on a redistribution of rewards from the environment, which ensures consistency with the original reward function. Transfer is also viewed as learning tasks in a sequential way (b;) and this suggests to introduce inductive bias to the neural architectures of agents to reduce catastrophic forgetting. Our method does not require to alter the agent's architecture. Other explicitly address the problem of transfer through the lens of multitask learning while we stick to learning from an initial distribution of environments. Meta-learning approaches aim to train agents on a distribution of tasks or environments so that their learned skills and representations work across the underlying continuum, and allow for fast adaptation of the agents (; ; ; ; ;). In contrast to meta-learning methods, we do not modify the RL algorithm used to train the agent and SECRET is compatible with any core algorithm for RL. Credit assignment Previous works investigated the role of attention mechanisms for credit assignment. propose SAB, a sparse attention mechanism used to derive a modified backpropagation algorithm. We draw inspiration from SAB but operate in the RL context without sparsity assumptions about the attention weights. RUDDER proposes to equip RL agents with an online method for credit assignment based on return decomposition. Our method operates offline and decomposes individual rewards. RUDDER requires a specific exploration scheme, an additional episodic replay buffer, a compute-heavy contribution analysis method and the addition of several auxiliary losses to the objective the RL agent optimizes. In comparison, SECRET is a lightweight method that does not deal with exploration. Crucially, the focus of is on online credit assignment while ours is on transfer. provide an agent with an external memory and the unsupervised task of reconstructing its inputs (both states and rewards). The agent uses memory reads as a way to identify related elements in sequences, and uses those to transfer the value of states providing delayed rewards to the bootstrapping target of contributing elements. In contrast, SECRET makes use of a non-autoregressive architecture, does not reconstruct states, makes use of reward shaping instead of modifying the update function and most importantly does not rely on an external memory. Recall Traces ) use a generative model that goes backward from high-reward states and samples state-action pairs that could have led to that state. SECRET also works backward from high-reward states but creates links to previous states from existing trajectories instead of sampling them. In this work, we investigated the role credit assignment could play in transfer learning and came up with SECRET, a novel transfer learning method that takes advantage of the relational properties of self-attention and transfers credit assignment instead of policy weights. We showed that SECRET led to improved sample efficiency in generalization and transfer scenarios in non-trivial gridworlds and a more complex 3D navigational task. To the best of our knowledge, this is the first line of work in the exciting direction of credit assignment for transfer. We think it would be worth exploring how SECRET could be incorporated into online reinforcement learning methods and leave this for future work. In this section we provide additional details about our experimental setup and the hyperparameters we use. In Triggers, episodes stop after 50 timesteps in 8x8 grids and 100 timesteps in 12x12 grids. Those values were chosen large enough such that the policies used to train the reward predictor have a chance to gather informative rewards. In DMLab, episodes stop after 900 timesteps, as defined by the standard settings. We use the same set of hyperparameters in all our experiments with few variation. In Triggers experiments, we use 128 units per dense layer, 32 convolutional filters and a single convolutional layer to process partial states and actions. We apply dropout at several places in the model as a regularizer. We use a dropout rate of 0.1 after dense layers, a dropout rate of 0.2 in the self-attention mechanism, and a dropout rate of 0.2 in the normalization blocks of the Transformer architecture. Since Transformers create representations via pooling, they need additional information so as to take into account the relative positions of sequence elements. Hence, we use the same positional encoding scheme as in , that is we add sine and cosine signals of varying frequencies to the sequence of input embeddings X emb before the self-attention layer: In these equations, t refers to the timestep in the sequence, i to the dimension of the embedding considered, and ν has a constant value of 10000. In DMLab experiments, we use two convolutional layers, 16 filters for each, and otherwise identical hyperparameters. Typically, we set the class weights in the loss function to w = w(−1) = 0.499, w = 0.02. Fig. 7 gives an overview of the whole architecture. In Sec. 3.1, we compare the attention vectors we get as outputs from the reward prediction model to ideal credit assignment with binary metrics. The ground truth we use is a binary vector of the size of the attention vector. Its values are 0 everywhere and 1 for timesteps that correspond to the activation of a Triggers. To do so, we introduce a simple heuristic to binarize the attention scalars: we consider all values above a threshold α to correspond to events to be credited. Then, we can measure precision and recall as in a binary classification paradigm. The precision and recall reported are the average precision and recall over 4 scenarios: Triggers with a 8x8 grid, 1 trigger and 1 reward; Triggers with a 8x8 grid, 1 trigger and 2 rewards; Triggers with a 8x8 grid, 2 triggers and 2 rewards; and Triggers with a 8x8 grid, 3 triggers and 1 reward. In each scenario, we train the model over a set of 40000 trajectories, each of which is drawn from a randomly sampled maze. Then, we apply it on 5000 trajectories from held-out environments and collect the attention weights corresponding to predictions on timesteps where the agent experiences positive reward. We use a fixed α of 0.2. We collect 40000 trajectories sampled from random policies to train the prediction reward model in all distributions of environments. In experiments involving tabular Q-learning we use online Q-learning with a learning rate of 0.1 and a constant greediness factor also equal to 0.1. For the out-of-domain transfer experiment with modified dynamics, we use a smaller version of the DQN architecture in. The first convolutional layer has 8 filters, a 3x3 kernel size and a stride of 2. The second and the third convolutional layers have both 16 filters, a 3x3 kernel size and a stride of 1. Those are completed by a feed-forward layer with 64 units followed by another feedforward layer with as many units as the number of available actions. The greediness factor is decayed linearly from 1 to 0.01 over 250000 steps in the environment at train time and has a constant value of 0.001 at test time. We use RMSProp as an optimizer with a base learning rate of 0.00025. We update the target network every 2000 steps and initially fill the replay buffer with 5000 transitions sampled following a random policy. The replay buffer has a maximum size of 1000000. We provide additional details about this setup: we train SECRET using 10000 trajectories sampled from a distribution of mazes that are generated randomly. These trajectories are sampled using an agent trained over the same distribution. We do so to increase the proportion of trajectories where rewards are experienced. Indeed, we found that using random policies yielded very few of these. Once the model is trained, we use it to compute the attentional potential function over a fixed maze. 1000 trajectories are sampled on the fixed maze using the same policy as the one that generated the trajectories used to train the reward prediction model. Since consecutive frames can be very similar, we consider a positive reward prediction to be correct (and thus use the corresponding attention weights when estimating the potential) if it happens within 5 frames of a reward actually experienced in the environment. We then compare the performance of agents trained with the original reward function to those trained with the shaped reward. An important point is that we use the knowledge of the position and the keys the agent possesses to build the state used to compute the potential function. This information is not given to the agent. We create an approximate state as such: it is the concatenation of the discretized position and the identifier of the key possessed. The discretized position is the of the euclidean division by a cell size integer c s and is necessary since the DMLab position is continuous. c s is fixed and has the value of 50 in our experiments. We acknowledge that relying on a manually constructed state limits the generality of our approach, but we are confident that this limitation can be addressed in future work by using an estimate of the true state. All agents mentioned in that section are PPO learners with a learning rate of 0.00019, an entropy coefficient of 0.0011, 12 actors, a discount factor γ = 0.99. They use generalized advantage estimation In this section we provide the of additional experiments that are useful to understand several aspects of SECRET. We study the influence of the size of the window in the transformation used to turn states into observations in Triggers. The goal of this experiment is notably to study the effect of the degree of partial observability to the assigned credit, and also the effect of transferring a possibly badly assigned credit to an agent, through reward shaping. We consider the following window types: 3x3, 5x5 and 7x7 (odd numbers because the window is centered on the agent), as well as the full state. We place ourselves in the in-domain setting and evaluate SECRET on held-out environments from the same distribution as the source. Environments are 8x8 mazes with 3 triggers and 1 prize. We display reward prediction and credit assignment metrics for each window size in Table 1, and the average discounted return of tabular Q-learning agents for each window size in Fig 9a. Notice that the considered reward prediction accuracy (Table 1) is weighted, such that all classes have the same importance. In this setting, we can observe that bigger window sizes are detrimental to the quality of the credit assigned, even if it helps to predict reward (Table 1). This was to be expected: the more "observable" is the sequence of observations used to train the reward predictor, the easier is the reward prediction (in the full state case, it can be predicted solely from the current state-action couple), but also the less likely the assigned credit will be centered on the trigger. However, even with a low credit precision, the shaped reward still accelerates the learning process (Fig. 9a). We stated that class weighting was important to keep the variance of prediction metrics across a variety of datasets of sampled trajectories low, the classes (sign of the reward) being quite imbalanced. Here, we study the influence of the class weights considered in the loss of the reward prediction model in Triggers. We consider the following class weights for the class corresponding to zero rewards: w = 1, w = 0.1, w = 0.01 and w = 0.001. The other class weights are fixed and have the value w = w(−1) = 1. We place ourselves in the in-domain setting and evaluate SECRET on held-out environments from the same distribution as the source. Environments are 8x8 mazes with 3 triggers and 1 prize. We display reward prediction and credit assignment metrics for each window size in Table 2, and the average discounted return of tabular Q-learning agents for each window size in Fig 9b. In that setting, putting heavier misclassification penalties for under-represented classes (−1 and 1) in improved credit and RL performance. We study the influence of the amount of data used by SECRET in a Triggers task. The objective is to assess how data-demanding is the proposed approach, and the effect of a lack of data on the transfer. There are two types of data: the data from the source domain (used to train its reward prediction component), and the data from the target domain (used to build the potential function). We consider the following number of trajectories to train the self-attentive reward predictor: 500, 1000, 5000, 10000, 25000 and 50000. We place ourselves in the in-domain setting and evaluate SECRET on held-out environments from the same distribution as the source. Environments are 8x8 mazes with 3 triggers and 1 prize. We display reward prediction and credit assignment metrics for each window size in Table 3, and the average discounted return of tabular Q-learning agents for each number of trajectories in Fig. 9c. We observe that increasing the size of the dataset improves the , both for the efficiency of the reward predictor (Table 3) and for the transfer (Fig. 9c). Having too few data can slow down learning compared to a vanilla agent (Fig. 9c), but it does not prevent from learning to solve the task. Number of source episodes Reward prediction accuracy Credit precision Credit recall 500 0.53 ± 0.11 0.23 ± 0.37 0.09 ± 0.2 1000 0.65 ± 0.07 0.49 ± 0.33 0.52 ± 0.42 5000 0.79 ± 0.08 0.9 ± 0.30 0.9 ± 0.30 10000 0.84 ± 0.07 0.9 ± 0.30 0.9 ± 0.30 25000 0.88 ± 0.09 0.9 ± 0.30 0.9 ± 0.30 50000 0.83 ± 0.11 0.8 ± 0.4 0.8 ± 0.4 We consider the following number of trajectories sampled from the target environment to build the potential function: 100, 500, 1000, 2000, 5000, 10000. We place ourselves in the in-domain setting and evaluate SECRET on held-out environments from the same distribution as the source. Environments are 8x8 mazes with 3 triggers and 1 prize. We display the average discounted return of tabular Q-learning agents for each number of trajectories in the source domain in Fig.9d. Here again, we observe that the number of episodes in the target distribution is an important parameter for SECRET, but it has less impact than the number of episodes in the source domain. In the main paper, we have only shown the distribution of attention for an in-domain scenario. Here, following the same protocol as in Sec. 3.1, we measure the distribution of attention weights in the two out-of-domain scenarios considered in our experiments: bigger mazes and inverted dynamics. The are reported in Fig. 10. We observe a similar distribution of attention, peaked on the triggers.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Secret is a transfer method for RL based on the transfer of credit assignment.

Represent the TLDR for finding the corresponding scientific article

We introduce the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended to policy gradient methods.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Recent advances in Neural Variational Inference allowed for a renaissance in latent variable models in a variety of domains involving high-dimensional data. In this paper, we introduce two generic Variational Inference frameworks for generative models of Knowledge Graphs; Latent Fact Model and Latent Information Model. While traditional variational methods derive an analytical approximation for the intractable distribution over the latent variables, here we construct an inference network conditioned on the symbolic representation of entities and relation types in the Knowledge Graph, to provide the variational distributions. The new framework can create models able to discover underlying probabilistic semantics for the symbolic representation by utilising parameterisable distributions which permit training by back-propagation in the context of neural variational inference, ing in a highly-scalable method. Under a Bernoulli sampling framework, we provide an alternative justification for commonly used techniques in large-scale stochastic variational inference, which drastically reduces training time at a cost of an additional approximation to the variational lower bound. The generative frameworks are flexible enough to allow training under any prior distribution that permits a re-parametrisation trick, as well as under any scoring function that permits maximum likelihood estimation of the parameters. Experiment display the potential and efficiency of this framework by improving upon multiple benchmarks with Gaussian prior representations. Code publicly available on Github. In many fields, including physics and biology, being able to represent uncertainty is of crucial importance BID18. For instance, when link prediction in Knowledge Graphs is used for driving expensive pharmaceutical experiments , it would be beneficial to know what is the confidence of a model in its predictions. However, a significant shortcoming of current neural link prediction models BID13 BID38 -and for the vast majority of neural representation learning approaches -is their inability to express a notion of uncertainty. Furthermore, Knowledge Graphs can be very large and web-scale BID14 and often suffer from incompleteness and sparsity BID14. In a generative probabilistic model, we could leverage the variance in model parameters and predictions for finding which facts to sample during training, in an Active Learning setting BID22 BID17. BID16 use dropout for modelling uncertainty, however, this is only applied at test time. However, current neural link prediction models typically only return point estimates of parameters and predictions BID32, and are trained discriminatively rather than generatively: they aim at predicting one variable of interest conditioned on all the others, rather than accurately representing the relationships between different variables BID31, however, BID16 could still be applied to get uncertainty estimates for these models. The main argument of this article is that there is a lack of methods for quantifying predictive uncertainty in a knowledge graph embedding representation, which can only be utilised using probabilistic modelling, as well as a lack of expressiveness under fixed-point representations. This constitutes a significant contribution to the existing literature because we introduce a framework for creating a family of highly scalable probabilistic models for knowledge graph representation, in a field where there has been a lack of this. We do this in the context of recent advances in variational inference, allowing the use of any prior distribution that permits a re-parametrisation trick, as well as any scoring function which permits maximum likelihood estimation of the parameters. In this work, we focus on models for predicting missing links in large, multi-relational networks such as FREEBASE. In the literature, this problem is referred to as link prediction. We specifically focus on knowledge graphs, i.e., graph-structured knowledge bases where factual information is stored in the form of relationships between entities. Link prediction in knowledge graphs is also known as knowledge base population. We refer to BID32 for a recent survey on approaches to this problem. A knowledge graph G {(r, a 1, a 2)} ⊆ R × E × E can be formalised as a set of triples (facts) consisting of a relation type r ∈ R and two entities a 1, a 2 ∈ E, respectively referred to as the subject and the object of the triple. Each triple (r, a 1, a 2) encodes a relationship of type r between a 1 and a 2, represented by the fact r(a 1, a 2).Link prediction in knowledge graphs is often simplified to a learning to rank problem, where the objective is to find a score or ranking function φ Θ r: E × E → R for a relation r that can be used for ranking triples according to the likelihood that the corresponding facts hold true. Recently, a specific class of link predictors received a growing interest BID32. These predictors can be understood as multi-layer neural networks. Given a triple x = (s, r, o), the associated score φ Θ r (s, o) is given by a neural network architecture encompassing an encoding layer and a scoring layer. In the encoding layer, the subject and object entities s and o are mapped to low-dimensional vector representations (embeddings) h s h(s) ∈ R k and h o h(o) ∈ R k, produced by an encoder h Γ: E → R k with parameters Γ. Similarly, relations r are mapped to h r h(r) ∈ R k. This layer can be pre-trained BID41 or, more commonly, learnt from data by back-propagating the link prediction error to the encoding layer BID4 BID32 BID38.The scoring layer captures the interaction between the entity and relation representations h s, h o and h r are scored by a function φ Θ (h s, h o, h r), parametrised by Θ. Other work encodes the entity-pair in one vector BID34. Summarising, the high-level architecture is defined as: DISPLAYFORM0 Ideally, more likely triples should be associated with higher scores, while less likely triples should be associated with lower scores. While the literature has produced a multitude of encoding and scoring strategies, for brevity we overview only a small subset of these. However, we point out that our method makes no further assumptions about the network architecture other than the existence of an argument encoding layer. Given an entity e ∈ E, the entity encoder h Γ is usually implemented as a simple embedding layer h Γ (e) [Γ] e, where Γ is an embedding matrix BID32. For pre-trained embeddings, the embedding matrix is fixed. Note that other encoding mechanisms are conceivable, such as; recurrent, graph convolution BID26 b) or convolutional neural networks BID13. DistMult DISTMULT BID43 represents each relation r using a parameter vector Θ ∈ R k, and scores a link of type r between (h s, h o, h r) using the following scoring function: DISPLAYFORM0 where ·, ·, · denotes the tri-linear dot product. ComplEx COMPLEX BID38 ) is an extension of DISTMULT using complex-valued embeddings while retaining the mathematical definition of the dot product. In this model, the scoring function is defined as follows: DISPLAYFORM1 k are complex vectors, x denotes the complex conjugate of x, Re (x) ∈ R k denotes the real part of x and Im (x) ∈ C k denotes the imaginary part of x. Let D {(τ 1, y 1),..., (τ n, y n)} denote a set of labelled triples, where τ i s i, p i, o i, and y i ∈ {0, 1} denotes the corresponding label, denoting that the fact encoded by the triple is either true or false. We can assume D is generated by a corresponding generative model. In the following, we propose two alternative generative models. In Figure 1's graphical model, we assume that the Knowledge Graph was generated according to the following generative model. Let V E × R × E the space of possible triples. where τ s, p, o, and h τ [h s, h p, h o] denotes the sampled embedding representations of s, o ∈ E and p ∈ R.Note that, in this model, the embeddings are sampled for each triple. As a consequence, the set of latent variables in this model is H {h τ | τ ∈ E × R × E}.The joint probability of the variables p θ (H, D) is defined as follows: DISPLAYFORM0 The marginal distribution over D is then bounded as follows, with respect to our variational distribution q: DISPLAYFORM1 Proposition 1 As a consequence, the log-marginal likelihood of the data, under the Latent Fact Model, is bounded by: DISPLAYFORM2 Proof. We refer the reader to the Appendix 6 for a detailed proof LFM's ELBO.Assumptions: LFM model assumes each fact of is a randomly generated variable, as well as a mean field variational distribution and that each training example is independently distributed. Note that this is an enormous sum over |D| elements. However, this can be approximated via Importance Sampling, or Bernoulli Sampling BID6. DISPLAYFORM0 By using Bernoulli Sampling, ELBO can be approximated by: DISPLAYFORM1 where p θ (s τ = 1) = b τ can be defined as the probability that for the coefficient s τ each positive or negative fact τ is equal to one (i.e is included in the ELBO summation). The exact ELBO can be recovered from setting b τ = 1.0 for all τ. We can define a probability distribution of sampling from D + and D − -similarly to Bayesian Personalised Ranking BID33, we sample one negative triple for each positive one -we use a constant probability for each element depending on whether it is in the positive or negative set. Proposition 2 The Latent Fact models ELBO can be estimated similarly using a constant probability for positive or negative samples, we end up with the following estimate: DISPLAYFORM2 where DISPLAYFORM3 In Figure 2's graphical model, we assume that the Knowledge Graph was generated according to the following generative model. Let V E × R × E the space of possible triples. DISPLAYFORM0 The marginal distribution over D is then defined as follows: DISPLAYFORM1 Proposition 3 The log-marginal likelihood of the data, under the Latent Information Model, is the following: DISPLAYFORM2 Proof. We refer the reader to the Appendix 6 for a detailed proof LIM's ELBO.Assumptions: LIM assumes each variable of information is randomly generated, as well as a mean field variational distribution and that each training example is independently distributed. This leads to a factorisation of the ELBO that seperates the KL term from the observed triples, making the approximation to the ELBO through Bernoulli sampling simpler, as the KL term is no longer approximated and instead fully computed. Similarly to Section 3.1.1, by using Bernoulli Sampling the ELBO can be approximated by: DISPLAYFORM0 Which can be estimated similarly using a constant probability for positive or negative samples, we end up with the following estimate:Proposition 4 The Latent Information Models ELBO can be estimated similarly using a constant probability for positive or negative samples, we end up with the following estimate: DISPLAYFORM1 where DISPLAYFORM2 Variational Deep Learning has seen great success in areas such as parametric/non-parametric document modelling BID29 BID30 and image generation BID25. Stochastic variational inference has been used to learn probability distributions over model weights BID3, which the authors named "Bayes By Backprop". These models have proven powerful enough to train deep belief networks BID40, by improving upon the stochastic variational bayes estimator BID25, using general variance reduction techniques. Previous work has also researched word embeddings within a Bayesian framework BID40, as well as researched graph embeddings in a Bayesian framework. However, these methods are expensive to train due to the evaluation of complex tensor inversions. Recent work by BID0 BID8 show that it is possible to train word embeddings through a variational Bayes BID2 framework. KG2E proposed a probabilistic embedding method for modelling the uncertainties in KGs. However, this was not a generative model. BID42 argued theirs was the first generative model for knowledge graph embeddings. However, their work is empirically worse than a few of the generative models built under our proposed framework, and their method is restricted to a Gaussian distribution prior. In contrast, we can use any prior that permits a re-parameterisation trick -such as a Normal (a) or von-Mises distribution.Later, BID27 ) proposed a generative model for graph embeddings. However, their method lacks scalability as it requires the use of the full adjacency tensor of the graph as input. Moreover, our work differs in that we create a framework for many variational generative models over multi-relational data, rather than just a single generative model over uni-relational data BID27 BID20. In a different task of graph generation, similar models have been used on graph inputs, such as variational auto-encoders, to generate full graph structures, such as molecules BID35 BID28.Recent work by BID9 ) constructed a variational path ranking algorithm, a graph feature model. This work differs from ours for two reasons. Firstly, it does not produce a generative model for knowledge graph embeddings. Secondly, their work is a graph feature model, with the constraint of at most one relation per entity pair, whereas our model is a latent feature model with a theoretical unconstrained limit on the number of existing relationships between a given pair of entities. We run each experiment over 500 epochs and validate every 50 epochs. Each KB dataset is separated into 80 % training facts, 10% development facts, and 10% test facts. During the evaluation, for each fact, we include every possible corrupted version of the fact under the local closed world assumption, such that the corrupted facts do not exist in the KB. Subsequently, we make a ranking prediction of every fact and its corruptions, summarised by mean rank and filtered hits@m. During training Bernoulli sampling to estimate the ELBO was used, with linear warm-up BID7, compression cost BID3, ADAM Glorot's initialiser for mean vectors BID19 and variance values initialised uniformly to embedding size −1. We experimented both with a N and a N (0, embedding size −1) prior on the latent variables. Table 1 shows definite improvements on WN18 for Variational ComplEx compared with the initially published ComplEX. We believe this is due to the well-balanced model regularisation induced by the zero mean unit variance Gaussian prior. Table 1 also shows that the variational framework is outperformed by existing non-generative models, highlighting that the generative model may be better suited at identifying and predicting symmetric relationships. WordNet18 (b) (WN18) is a large lexical database of English. WN18RR is a subset with only asymmetric relations. FB15K is a large collaboratively made dataset which covers a vast range of relationships and entities, with FB15K-257 BID37, with 257 relations -a significantly reduced number from FB15K due to being a similarly refined asymmetric dataset. We now compare our model to the previous state-of-the-art multi-relational generative model TransG BID42, as well as to a previously published probabilistic embedding method KG2E (similarly represents each embedding with a multivariate Gaussian distribution) on the WN18 dataset. Table 2 makes clear the improvements in the performance of the previous state-of-the-art generative multi-relational knowledge graph model. LFM has marginally worse performance than the state-of-the-art model on raw Hits@10. We conjecture two reasons may cause this discrepancy. Firstly, the fact the authors of TransG use negative samples provided only (True negative examples), whereas we generated our negative samples using the local closed world assumption (LCWA). Secondly, we only use one negative sample per positive to estimate the Evidence Lower Bound using Bernoulli sampling, whereas it is likely they used significantly more negative samples. Scoring Table 1: Filtered and Mean Rank (MR) for the models tested on the WN18, WN18RR, and FB15K datasets. Hits@m metrics are filtered. Variational written with a "V". *Results reported from BID39 ) and **Results reported from BID13 for ComplEx model. "-" in a table cell equates to that statistic being un-reported in the models referenced paper. Dataset Scoring Function MR Raw Hits@ Filtered Hits @ Raw Filter 10 1 3 10 WN18 KG2E 362 345 0.805 --0.932 TransG (Generative) BID42 345 FORMULA7 We split the analysis into the predictions of subject ((?, r, o)) or object ((s, r, ?)) for each test fact. Note all are filtered predictions, i.e., ignoring the predictions made on negative examples generated under LCWA -using a randomly corrupted fact (subject or object corruption) as a negative example. TAB3 shows that the relation "_derivationally_related_form", comprising 34% of test subject predictions, was the most accurate relation to predict for Hits@1 when removing the subject from the tested fact. Contrarily, "_member_of_domain_region" with zero Hits@1 subject prediction, making up less than 1% of subject test predictions. However, "_member_meronym " was the least accurate and prominent (8% of the test subject predictions) for subject Hits@1. We learn from this that even for a near state-of-the-art model there is a great deal of improvement to be gained among asymmetric modelling. TAB3, as before the relation "_derivationally_related_form" was the most accurate relation to predict Hits@1. TAB3 as it highlights the Latent Information Model's inability to achieve a high Hits@1 performance predicting objects for the "_hypernym" relation, which is significantly hindering model performance as it is the most seen relation in the test set-its involvement in 40% of object test predictions. These hint at the possibility that the slightly stronger of WN18 are due to covariances in our variational framework able to capture information about symbol frequencies. We verify this by plotting the mean value of covariance matrices, as a function of the entity or predicate frequencies FIG2 ). The plots confirm our hypothesis: covariances for the variational Latent Information Model grows with the frequency, and hence the LIM would put a preference on predicting relationships between less frequent symbols in the knowledge graph. This also suggests that covariances from the generative framework can capture genuine information about the generality of symbolic representations. We project the high dimensional mean embedding vectors to two dimensions using Probabilistic Principal Component Analysis (PPCA) BID36 to project the variance embedding vectors down to two dimensions using Non-negative Matrix Factorisation (NNMF) BID15. Once we have the parameters for a bivariate normal distribution, we then sample from the bivariate normal distribution 1,000 times and then plot a bi-variate kernel density estimate of these samples. By visualising these two-dimensional samples, we can conceive the space in which the entity or relation occupies. We complete this process for the subject, object, relation, and a randomly sampled corrupted entity (under LCWA) to produce a visualisation of a fact, as shown in FIG3. Figure 4 displays a clustering of the subject, object, and predicate that create a positive (true) fact. We also observe a separation between the items which generate a fact and a randomly sampled (corrupted) entity which is likely to create a negative (false) fact. The first test fact "(USA, Commonbloc0, Netherlands)" shows clear irrationality similarity between all objects in the tested fact, i.e. the vectors are pointing towards a south-east direction. We can also see that the corrupted entity Jordan is quite a distance away from the items in the tested fact, which is good as Jordan does not share a common bloc either USA or Netherlands. We used scoring functions which measure the similarity between two vectors, however, for more sophisticated scoring functions which distance/ similarity is not important to the end we would unlikely see such interpretable images. This analysis of the learnt distributions is evidence to support the notion of learnt probabilistic semantics through using this framework. We have successfully created a framework allowing a model to learn embeddings of any prior distribution that permits a re-parametrisation trick via any score function that permits maximum likelihood estimation of the scoring parameters. The framework reduces the parameter by one hyperparameter -as we typically would need to tune a regularisation term for an l1/ l2 loss term, however as the Gaussian distribution is self-regularising this is deemed unnecessary for matching state-ofthe-art performance. We have shown, from preliminary experiments, that these display competitive with current models. Overall, we believe this work will enable knowledge graph researchers to work towards the goal of creating models better able to express their predictive uncertainty. The score we acquire at test time even through forward sampling does not seem to differ much compared with the mean embeddings, thus using the learnt uncertainty to impact the positively is a fruitful path. We would also like to see additional exploration into various encoding functions, as we used only the most basic for these experiments. We would also like to see more research into measuring how good the uncertainty estimate is. We would like to thank all members of the Machine Reading lab for useful discussions.

Working toward generative knowledge graph models to better estimate predictive uncertainty in knowledge inference.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.