Question: Knowledge Graph Embedding (KGE) is the task of jointly learning entity and relation embeddings for a given knowledge graph. Existing methods for learning KGEs can be seen as a two-stage process where (a) entities and relations in the knowledge graph are represented using some linear algebraic structures (embeddings), and (b) a scoring function is defined that evaluates the strength of a relation that holds between two entities using the corresponding relation and entity embeddings. Unfortunately, prior proposals for the scoring functions in the first step have been heuristically motivated, and it is unclear as to how the scoring functions in KGEs relate to the generation process of the underlying knowledge graph. To address this issue, we propose a generative account of the KGE learning task. Specifically, given a knowledge graph represented by a set of relational triples (h, R, t), where the semantic relation R holds between the two entities h (head) and t (tail), we extend the random walk model (a) of word embeddings to KGE. We derive a theoretical relationship between the joint probability p(h, R, t) and the embeddings of h, R and t. Moreover, we show that marginal loss minimisation, a popular objective used by much prior work in KGE, follows naturally from the log-likelihood ratio maximisation under the probabilities estimated from the KGEs according to our theoretical relationship. We propose a learning objective motivated by the theoretical analysis to learn KGEs from a given knowledge graph. The KGEs learnt by our proposed method obtain state-of-the-art performance on FB15K237 and WN18RR benchmark datasets, providing empirical evidence in support of the theory. Knowledge graphs such as Freebase BID2 organise information in the form of graphs, where entities are represented by vertices in the graph and the relation between two entities is represented by the edge that connects the corresponding two vertices. By embedding entities and relations that exist in a knowledge graph in some (possibly lower-dimensional and latent) space we can infer previously unseen relations between entities, thereby expanding a given knowledge graph BID11 BID20; BID9 BID16 BID17 BID4.Existing KGE methods can be seen as involving two main steps. First, given a knowledge graph represented by a set of relational triples (h, R, t), where a semantic relation R holds between a head entity h and a tail entity t, entities and relations are represented using some mathematical structures such as vectors, matrices or tensors. Second, a scoring function is proposed that evaluates the relational strength of a triple (h, R, t) and entity and relation embeddings that optimise the defined scoring function are learnt using some optimisation method. Table 1 shows some of the scoring functions proposed in prior work in KGE learning. Despite the wide applications of entity and relation embeddings created via KGE methods, the existing scoring functions are motivated heuristically to capture some geometric requirements of the embedding space. For example, TransE BID4 assumes that the entity and relation embeddings co-exist in the same (possibly lower dimensional) vector space and translating (shifting) the head entity embedding by the relation embedding must make it closer to the tail entity embedding, whereas ComplEx BID16 ) models the asymmetry in relations using Score function f (h, R, t) Relation parameters Unstructured BID4 h − t 1/2 none Structured embeddings BID4 R 1 h − R 2 t 1,2 R 1, R 2 ∈ R d×d TransE BID4 h + R − t 1/2 R ∈ R d DistMult BID20 h, R, t R ∈ R d RESCAL BID9 h Rt R d×d ComplEx BID16 h, R,t R ∈ C d Table 1: Score functions proposed in selected prior work on KGE. Entity embeddings h, t ∈ R d are vectors in all models, except in ComplEx where h, t ∈ C d. Here, x 1/2 denotes either 1 or 2 norm of the vector x. In ComplEx,x is the elementwise complex conjugate, and ·, ·, · denotes the component-wise multi-linear inner-product.the component-wise multi-linear inner-product among entity and relation embeddings. Relational triples extracted from a given knowledge graph are used as positive training instances, whereas pseudo-negative BID4 instances are automatically generated by randomly corrupting positive instances. Finally, KGE are learnt such that the prediction loss computed over the positive and negative instances is minimised. Despite the good empirical performances of the existing KGE methods, theoretical understanding of KGE methods is comparatively under developed. For example, it is not clear how the heuristically defined KGE objectives relate to the generative process of a knowledge graph. In this paper, we attempt to fill this void by providing a theoretical analysis of KGE. Specifically, in section 2, we propose a generative process where we explain the formation of a relation R between two entities h and t using the corresponding relation and entity embeddings. Following this generative story, we derive a relationship between the probability of R holding between h and t, p(h, t | R), and the embeddings of R, h and t. Interestingly, the derived relationship is not covered by any of the previously proposed heuristically-motivated scoring functions, providing the first-ever KGE method with a provable generative explanation. Next, in section 3, we show that the margin loss, which has been popularly used as a training objective in prior work on KGE, naturally arises as the log-likelihood ratio computed from p(h, t | R). Based on this , we derive a training objective that we subsequently optimise for learning KGEs that satisfy our theoretical relationship. Using standard benchmark datasets proposed in prior work on KGE learning, we evaluate the learnt KGEs on a link prediction task and a triple classification task. Experimental show that the learnt KGEs obtain state-of-the-art performance on FB15K237 and WN18RR benchmarks, thereby providing empirical evidence to support the theoretical analysis. Let us consider a knowledge graph D where the knowledge is represented by relational triples (h, R, t) ∈ D. Here, R is a relational predicate of two arguments, where h (head) and t (tail) entities respectively filling the first and second arguments. We assume relations to be asymmetric in general. In other words, if (h, R, t) ∈ D then it does not necessarily follow that (t, R, h) ∈ D. The goal of KGE is to learn embeddings (representations) for the relations and entities in the knowledge graph such that the entities that participate in similar relations are embedded closely to each other in the entity embedding space, while at the same time relations that hold between similar entities are embedded closely to each other in the relational embedding space. We call the learnt entity and relation embeddings collectively as KGEs. Following prior work on KGE BID4 BID16 BID20, we assume that entities and relations are embedded in the same vector space, allowing us to perform linear algebraic operations using the embeddings in the same vector space. Let us consider a random walk characterised by a time-dependent knowledge vector c k, where k is the current time step. The knowledge vector represents the knowledge we have about a particular group of entities and relations that express some facts about the world. For example, the knowledge that we have about people that are employed by companies can be expressed using entities of classes such as people and organisation, using relations such as CEO-of, employed-at, works-for, etc. We assume that entities h and t are represented by time-independent d-dimensional vectors, respectively h, t ∈ R d.We assume the task of generating a relational triple (h, R, t) in a given knowledge graph to be a two-step process as described next. First, given the current knowledge vector at time k, c = c k and the relation R, we assume that the probability of an entity h satisfying the first argument of R to be given by. DISPLAYFORM0 Here, R 1 ∈ R d×d is a relation-specific orthogonal matrix that evaluates the appropriateness of h for the first argument of R. For example, if R is the CEO-of relation, we would require a person as the first argument and a company as the second argument of R. However, note that the role of R 1 extends beyond simply checking the types of the entities that can fill the first argument of a relation. For our example above, not all people are CEOs and R 1 evaluates the likelihood of a person to be selected as the first argument of the CEO-of relation. Z c is a normalisation coefficient such that h∈V p(h | R, c) = 1, where the vocabulary V is the set of all entities in the knowledge graph. After generating h, the state of our random walker changes to c = c k+1, and we next generate the second argument of R with the probability given by. DISPLAYFORM0 Here, R 2 ∈ R d×d is a relation-specific orthogonal matrix that evaluates the appropriateness of t as the second argument of R. Z c is a normalisation coefficient such that t∈V p(t | R, c) = 1. Following our previous example of the CEO-of relation, R 2 evaluates the likelihood of an organisation to be a company with a CEO position. Importantly, R 1 and R 2 are representations of the relation R and independent of the entities. Therefore, we consider (R 1 and R 2) to collectively represent the embedding of R. Orthogonality of R 1, R 2 is a requirement for the mathematical proof and also act as a regularisation constraint to prevent overfitting by restricting the relational embedding space. We first perform our mathematical analysis for relational embeddings represented by orthogonal matrices and discuss later how this requirement can be relaxed. We assume a slow random walk where the knowledge vectors do not change significantly between consecutive time steps (c k ≈ c k+1). More specifically, we assume that c k − c k+1 ≤ 2 for some small 2 > 0. This is a realistic assumption for generating the two entity arguments in the same relational triple because, if the knowledge vectors were significantly different in the two generation steps, then it is likely that the corresponding relations are also different, which would not be coherent with the above-described generative process. Moreover, we assume that the knowledge vectors are distributed uniformly in the unit sphere and denote the distribution of knowledge vectors by C.To learn KGEs, we must estimate the probability that h and t satisfy the relation R, p(h, t | R), which can be obtained by taking the expectation of p(h, t | R, c, c) w.r.t. c, c ∼ C given by. DISPLAYFORM1 Here, partition functions are given by Z c = h∈V c∈C exp h R 1 c and Z c = t∈V c ∈C exp t R 2 c. follows from our two-step generative process where the generation of h and t in each step is independent given the relation and the corresponding knowledge vectors. Computing the expectation in is generally difficult because of the two partition functions Z c and Z c. However, Lemma 1 shows that the partition functions are narrowly distributed around a constant value for all c (or c) values with high probability. Lemma 1 (Concentration Lemma). If the entity embedding vectors satisfy the Bayesian prior v = sv, wherev is from the spherical Gaussian distribution, and s is a scalar random variable, which is always bounded by a constant κ, then the entire ensemble of entity embeddings satisfies that DISPLAYFORM2 for z = O(1/ √ n), and δ = exp(−Ω(log 2 n)), where n ≥ d is the number of words and Z c is the partition function for c given by c∈V exp h R 1 c.proof: To prove the concentration lemma, we show that the mean E h [Z c] of Z c is concentrated around a constant for all knowledge vectors c and its variance is bounded. Recall that DISPLAYFORM3 If P is an orthogonal matrix and x is a vector, then P x DISPLAYFORM4 2, because P P = I. Therefore, from FORMULA4 and the orthogonality of the relational embeddings, we see that R 1 c is a simple rotation of c and does not alter the length of c. We represent h = s hĥ, where s h = h andĥ is a unit vector (i.e. ĥ 2 = 1) distributed on the spherical Gaussian with zero mean and unit covariance matrix I d ∈ R d×d. Let s be a random variable that has the same distribution as s h. Moreover, let us assume that s is upper bounded by a constant κ such that s ≤ κ. From the assumption of the knowledge vector c, it is on the unit sphere as well, which is then rotated by R 1.We can write the partition function using the inner-product between two vectors h and R 1 c, Z c = h∈V exp h (R 1 c). BID0 showed that (Lemma 2.1 in their paper) the expectation of a partition function of this form can be approximated as follows: DISPLAYFORM5 where n = |V| is the number of entities in the vocabulary. follows from the expectation of a sum and the independence of h and R 1 from c. The inequality of FORMULA6 is obtained by applying the Taylor expansion of the exponential series and the final equality is due to the symmetry of the spherical Gaussian. From the law of total expectation, we can write DISPLAYFORM6 where, x = h R 1 c. Note that conditioned on s h, h is a Gaussian random variable with variance σ 2 = s 2 h. Therefore, conditioned on s h, x is a random variable with variance σ 2 = σ 2 h. Using this distribution, we can evaluate E x|s h exp h R 1 c as follows: DISPLAYFORM7 Therefore, it follows that DISPLAYFORM8 where s is the variance of the 2 norms of the entity embeddings. Because the set of entities is given and fixed, both n and σ are constants, proving that E[Z c] does not depend on c. Next, we calculate the variance V c [Z c] as follows: DISPLAYFORM9 Because 2h R 1 t is a Gaussian random variable with variance 4σ 2 = 4s 2 h from a similar calculation as in FORMULA0 we obtain, DISPLAYFORM10 for Λ = exp(8κ 2) a constant bounding s ≤ κ as stated. From above, we have bounded both the mean and variance of the partition function by constants that are independent of the knowledge vector. Note that neither exp h R 1 c nor exp t R 2 c are subGaussian nor sub-exponential. Therefore, standard concentration bounds derived for sub-Gaussian or sub-exponential random variables cannot be used in our analysis. However, the argument given in Appendix A.1 in BID1 for a partition function with bounded mean and variance can be directly applied to Z c in our case, which completes the proof of the concentration lemma. From the symmetry between h and t, Lemma 1 also applies for the partition function t∈V t R 2 c. Under the conditions required to satisfy Lemma 1, the following main theorem of this paper holds: Theorem 1. Suppose that the entity embeddings satisfy. Then, we have DISPLAYFORM11 DISPLAYFORM12 where DISPLAYFORM13 The complete proof of Theorem 1 is given in Appendix A. Below we briefly sketch the main steps. Proof sketch: Let F be the event that both c and c are within (1 ± z)Z. Then, from Lemma 1 and the union bound, event F happens with probability at least 1 − 2 exp(−Ω(log 2 n)). The R.H.S. of can be split into two parts T 1 and T 2 according to whether F happens or not. DISPLAYFORM14 T 1 can be approximated as given by. DISPLAYFORM15 On the other hand, T 2 can be shown to be a constant, independent of d, given by. DISPLAYFORM16 The vocabulary size n of real-world knowledge graphs is typically over 10 5, for which T 2 becomes negligibly small. Therefore, it suffices to consider only T 1. Because of the slowness of the random walk we have c ≈ c Using the law of total expectation we can write T 1 as follows: DISPLAYFORM17 where A(c):= E c |c exp t R 2 c. Doing some further evaluations we show that The relationship given by indicates that head and tail entity embeddings are first transformed respectively by R 1 and R 2, and the squared 2 norm of the sum of the transformed vectors is proportional to the probability p(h, t | R). DISPLAYFORM18 In this section, we derive a training objective from Theorem 1 that we can then optimise to learn KGE. The goal is to empirically validate the theoretical by evaluating the learnt KGEs. Knowledge graphs represent information about relations between two entities in the form of relational triples. The joint probability p(h, R, t) given by Theorem 1 is useful for determining whether a relation R exists between two given entities h and t. For example, if we know that with a high probability that R holds between h and t, then we can append (h, R, t) to the knowledge graph. The task of expanding knowledge graphs by predicting missing links between entities or relations is known as the link prediction problem BID16. In particular, if we can automatically append such previously unknown knowledge to the knowledge graph, we can expand the knowledge graph and address the knowledge acquisition bottleneck. To derive a criteria for determining whether a link must be predicted among entities and relations, let us consider a relational triple (h, R, t) ∈ D that exists in a given knowledge graph D. We call such relational triples as positive triples because from the assumption it is known that R holds between h and t. On the other hand, consider a negative relational triple (h, R, t) ∈ D formed by, for example, randomly perturbing a positive triple. A popular technique for generating such (pseudo) negative triples is to replace h or t with a randomly selected different instance of the same entity type. As an alternative for random perturbation, proposed a method for generating negative instances using adversarial learning. Here, we are not concerned about the actual method used for generating the negative triples but assume a set of negative triples,D, generated using some method, to be given. Given a positive triple (h, R, t) ∈ D and a negative triple (h, R, t) ∈D, we would like to learn KGEs such that a higher probability is assigned to (h, R, t) than that assigned to (h, R, t). We can formalise this requirement using the likelihood ratio given by. DISPLAYFORM0 Here, η > 1 is a threshold that determines how higher we would like to set the probabilities for the positive triples compares to that of the negative triples. By taking the logarithm of both sides in we obtain DISPLAYFORM1 If a positive triple (h, R, t) is correctly assigned a higher probability than a negative triple p(h, R, t), then the left hand side of will be negative, indicating that there is no loss incurred during this classification task. Therefore, we can re-write to obtain the marginal loss BID5, L(D,D), a popular choice as a learning objective in prior work in KGE, as shown in. DISPLAYFORM2 We can assume 2d log η to be the margin for the constraint violation. Theorem 1 requires R 1 and R 2 to be orthogonal. To reflect this requirement, we add two 2 regularisation terms R 1 R 1 − I 2 2 and R 2 R 2 − I 2 2 respectively with regularisation coefficients λ 1 and λ 2 to the objective function given by. In our experiments, we compute the gradients w.r.t. each of the parameters h, t, R 1 and R 2 and use stochastic gradient descent (SGD) for optimisation. This approach can be easily extended to learn from multiple negative triples as shown in Appendix B. At a high-level of abstraction, KGE methods can be seen as differing in their design choices for the following two main problems: (a) how to represent entities and relations, and (b) how to model the interaction between two entities and a relation that holds between them. Next, we briefly discuss prior proposals to those two problems (refer BID17 BID10 BID6 for an extended survey on KGE).A popular choice for representing entities is to use vectors, whereas relations have been represented by vectors, matrices or tensors. For example, TransE BID4, TransH BID18, TransD , TransG BID19, TransR , lppTransD BID21, DistMult BID20, HolE BID11 and ComplEx BID16 represent relations by vectors, whereas Structured Embeddings BID4 Given entity and relation embeddings, a scoring function is defined that evaluates the strength of a relation R between two entities h and t in a triple (h, R, t). The scoring functions that encode various intuitions have been proposed such as the 1 or 2 norms of the vector formed by a translation of the head entity embedding by the relation embedding over the target embedding, or by first performing a projection from the entity embedding space to the relation embedding space BID21 As an alternative to using vector norms as scoring functions, DistMult and ComplEx use the component-wise multi-linear dot product. Once a scoring function is defined, KGEs are learnt that assign better scores to relational triples in existing knowledge graphs (positive triples) over triples where the relation does not hold (negative triples) by minimising a loss function such as the logistic loss (RESCAL, DistMult, ComplEx) or marginal loss (TransE, TransH, TransD, TransD). Because knowledge graphs record only positive triples, a popular method to generate pseudo negative triples is to perturb a positive instance by replacing its head or tail entity by an entity selected uniformly at random from the vocabulary of the entities. However, uniformly sampled negative triples are likely to be obvious examples that do not provide much information to the learning process and can be detected by simply checking for the type of the entities in a triple. proposed an adversarial learning approach where a generator assigns a probability to each relation triple and negative instances are sampled according to this probability distribution to train a discriminator that discriminates between positive and negative instances. BID19 proposed TransG, a generative model based on the Chinese restaurant process, to model multiple relations that exist between a pair of entities. However, their relation embeddings are designed to satisfy vector translation similar to TransE.As an alternative to directly learning embeddings from a graph, several methods (; BID12 BID13 have considered the vertices visited during truncated random walks over the graph as pseudo sentences, and have applied popular word embedding learning algorithms such as skip-gram with negative sampling or continuous bag-of-words model to learn vertex embeddings. However, pseudo sentences generated this way are syntactically very different from sentences in natural languages. On the other hand, our work extends the random walk analysis by BID0 that derives a useful connection between the joint co-occurrence probability of two words and the 2 norm of the sum of the corresponding word embeddings. Specifically, they proposed a latent variable model where the words in a corpus are generated by a probabilistic model parametrised by a time-dependent discourse vector that performs a random walk. However, unlike in our work, they do not consider the relations between two co-occurring words in a corpus. BID3 extended the model proposed by BID0 to capture co-occurrences involving more than two words. They defined the co-occurrence of k unique words in a given context as a k-way co-occurrence, where BID0's could be seen as a special case corersponding to k = 2. Moreover, BID3 showed that it is possible to learn word embeddings that capture some types of semantic relations such as antonymy and collocation using 3-way co-occurrences more accurately than using 2-way co-occurrences. However, their model does not explicitly consider the relations between words/entities and uses only a corpus for learning the word embeddings. To empirically evaluate the theoretical stated in Theorem 1, we learn KGEs (denoted by RelWalk) by minimising the marginal loss objective derived in section 3. We use the FB15k237, FB13 (subsets of Freebase) and WN11, WN18RR (subsets of WordNet) datasets, which are standard benchmarks for KGE. We use the standard training, validation and test splits as detailed in TAB2. We generate negative triples by replacing a head or a tail entity in a positive triple by a randomly selected different entity and learn KGEs. We train the model until convergence or at most 1000 epochs over the training data where each epoch is divided into 100 mini-batches. The best model is selected by early stopping based on the performance of the learnt embeddings on the validation set (evaluated after each 20 epochs). The training details and hyperparameter settings are detailed in Appendix C. RelWalk is implemented in the open-source toolkit OpenKE . We conduct two evaluation tasks: link prediction (predict the missing head or tail entity in a given triple (h, R, ?) or (?, R, t)) BID4 and triple classification (predict whether a relation R holds between h and t in a given triple (h, R, t)) BID14. We evaluate the performance in the link prediction task using mean reciprocal rank (MRR), mean rank (MR (the average of the rank assigned to the original head or tail entity in a corrupted triple) and hits at ranks 1, 3 and 10 (H@1,3,10), whereas in the triple classification task we use accuracy (percentage of the correctly classified test triples). We only report scores under the filtered setting BID5, which removes all triples appeared in training, validating and testing sets from candidate triples before obtaining the rank of the ground truth triple. In link prediction, we consider all entities that appear in the corresponding argument in the entire knowledge graph as candidates. In TAB0 we compare the KGEs learnt by RelWalk against prior work using the published . For link prediction, RelWalk reports SoTA on both WN18RR and FB15K237 in all evaluation measures, except against ConvE in WN18RR measured by MRR. WN18RR excludes triples from WN18 that are simply inverted between train and test partitions BID15 ). RelWalk's consistently good performance on both versions of this dataset shows that it is considering the global structure in the knowledge graph when learning KGEs. For triple classification, RelWalk reports the best performance on FB13, whereas TransG reports the best performance on WN11. Considering that both TransG and RelWalk are generative models, it would be interesting to further investigate generative approaches for KGE in the future. Overall, the experimental support our theoretical claim and emphasise the importance of theoretically motivating the scoring function design process. We proposed RelWalk, a generative model of KGE and derived a theoretical relationship between the probability of a triple and entity, relation embeddings. We then proposed a learning objective based on the theoretical relationship we derived. Experimental on a link prediction and a triple classification tasks show that RelWalk obtains strong performances in multiple benchmark datasets. A PROOF OF THEOREM 1Let us consider the probabilistic event that DISPLAYFORM0 Then from the union bound we have, DISPLAYFORM1 Moreover, let F be the probabilistic event that both F c and F c being True. Then from DISPLAYFORM2 We can decompose the expectation in the R.H.S. in FORMULA2 into two terms T 1 and T 2 depending on whether respectively F is True or False as follows: DISPLAYFORM3 Here, 1 F and 1F are indicator functions given by: DISPLAYFORM4 and DISPLAYFORM5 Let us first show that T 2 is negligibly small. For two real integrable functions ψ 1 (x) and ψ 2 (x) in [a, b], the Cauchy-Schwarz's inequality states that DISPLAYFORM6 The second term of FORMULA2 is upper bounded by DISPLAYFORM7 The first term of can be bounded as follows: DISPLAYFORM8 where α > 1. Therefore, it is sufficient to bound E c exp(αh R 1 c) DISPLAYFORM9 Let us denote by z the random variable 2h R 1 c. Moreover, let r(z) = E c |z [1F], which is a function of z between. We wish to upper bound E c [exp(z)r(z)]. The worst-case r(z) can be quantified using a continuous version of Abel's inequality (proved as Lemma A.4 in BID1), we can upper bound E c [exp(z)r(z)] as follows: DISPLAYFORM10 where DISPLAYFORM11 is a function that takes the value 1 when z ≥ t and zero elsewhere. Then, we claim Pr c [z ≥ t] ≤ exp(−Ω(log 2 n)) implies that t ≥ Ω(log .9 n).If c was distributed as N (0, DISPLAYFORM12, this would be a simple tail bound. However, as c is distributed uniformly on the sphere, this requires special care, and the claim follows by applying the tail bound for the spherical distribution given by Lemma A.1 in BID0 instead. Finally, applying Corollary A.3 in BID0, we have: DISPLAYFORM13 From a similar argument as above we can obtain the same bound for c as well. Therefore, T 2 in can be upper bounded as follows: DISPLAYFORM14 Because n = |V|, the size of the entity vocabulary, is large (ca. n > 10 5) in most knowledge graphs, we can ignore the T 2 term in. Combining this with we obtain an upper bound for p(h, t | R) given by. DISPLAYFORM15 where |D| is the number of relational tuples (h, R, t) in the KB D and δ 0 = |D| exp(−Ω(log 1.8 n)) ≤ exp(−Ω(log 1.8 n)) by the fact that Z ≤ exp(2κ)n = O(n), where κ is the upper bound on h R 1 c and t R 2 c, which is regarded as a constant. On the other hand, we can lower bound p(h, t | R) as given by. DISPLAYFORM16 Taking the logarithm of both sides, from and, the multiplicative error translates to an additive error given by. DISPLAYFORM17 where A(c):= E c |c exp t R 2 c.We assumed that c and c are on the unit sphere and R 1 and R 2 to be orthogonal matrices. Therefore, R 1 c and R 2 c are also on the unit sphere. Moreover, if we let the upper bound of the 2 norm of the entity embeddings to be κ DISPLAYFORM18 Then we can lower bound A(c) as follows: DISPLAYFORM19 For some 2 > 0. The last inequality holds because DISPLAYFORM20 To obtain a lower bound on A(c) from the first-order Taylor approximation of exp(x) ≥ 1 + x we observe that DISPLAYFORM21 Therefore, from our model assumptions we have DISPLAYFORM22 Hence, DISPLAYFORM23 Therefore, from FORMULA3 and FORMULA6 we have DISPLAYFORM24 Plugging A(c) back in we obtain DISPLAYFORM25 = log E c exp h R 1 c exp t R 2 c ± δ 0 − 2 log Z + 2 log(1 ± z) + log(1 ± 2)= log E c exp h R 1 c + t R 2 c ± δ 0 − 2 log Z + 2 log(1 ± z) + log(1 ± 2)= log E c exp R 1 h + R 2 t c ± δ 0 − 2 log Z + 2 log(1 ± z) + log(1 ± 2)Note that c has a uniform distribution over the unit sphere. In this case, from Lemma A.5 in BID1 ), holds approximately. DISPLAYFORM26 where 3 =Õ(1/d). Plugging FORMULA1 in FORMULA0 we have that 8 n) ). Therefore, δ 0 can be ignored. Note that 3 =Õ(1/d) and z =Õ(1/ √ n) by assumption. Therefore, we obtain that DISPLAYFORM27 DISPLAYFORM28 DISPLAYFORM29 In this section, we show how the margin loss-based learning objective derived in section 3 can be extended to learn from more than one negative triples per each positive triple. This formulation leads to rank-based loss objective used in prior work on KGE. Considering that negative triples are generated via random perturbation, it is important to consider multiple negative triples during training to better estimate the classification boundary. Let us consider that we are given a positive triple, (h, R, t) and a set of K negative triples {(h k, R, t k)} K k=1. We would like our model to assign a probability, p(h, t | R), to the positive triple that is higher than that assigned to any of the negative triples. This requirement can be written as. DISPLAYFORM0 We could further require the ratio between the probability of the positive triple and maximum probability over all negative triples to be greater than a threshold η ≥ 1 to make the requirement of to be tighter. DISPLAYFORM1 By taking the logarithm of we obtain log p(h, t | R) − log max k=1,...,K DISPLAYFORM2 Therefore, we can define the margin loss for a misclassification as follows:L (h, R, t), {(h k, R, t k)} K k=1 = max 0, log max k=1,...,K p(h k, t k | R) + log(η) − log p(h, t | R) FORMULA2 However, from the monotonicity of the logarithm we have ∀x 1, x 2 > 0, if log(x 1) ≥ log(x 2) then x 1 ≥ x 2. Therefore, the logarithm of the maximum can be replaced by the maximum of the logarithms in as shown in.L (h, R, t), {(h k, R, t k)} K k=1 = max 0, max k=1,...,K log p(h k, t k | R) + log(η) − log p(h, t | R)By substituting for the probabilities in we obtain the rank-based loss given by.L (h, R, t), {(h k, R, t k)} In practice, we can use p(h k, t k | R) to select the negative triple with the highest probability for training with the positive triple. C TRAINING DETAILSThe statistics of the benchmark datasets are show in TAB2.We selected the initial learning rate (α) for SGD in {0.01, 0.001}, the regularisation coefficients (λ 1, λ 2) for the orthogonality constraints of relation matrices in {0, 1, 10, 100}. The number of randomly generated negative triples n neg for each positive example is varied in {1, 10, 20, 50, 100} and d ∈ {50, 100}. Optimal hyperparameter settings were: λ 1 = λ 2 = 10, n neg = 100 for all the datasets, α = 0.001 for FB15K, FB15K237 and FB13, α = 0.01 for WN18, WN18RR and WN11. For FB15K237 and WN18RR d = 100 was the best, whereas for all other datasets d = 50 performed best.

We present a theoretically proven generative model of knowledge graph embedding.

represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR.

Question: Currently the only techniques for sharing governance of a deep learning model are homomorphic encryption and secure multiparty computation. Unfortunately, neither of these techniques is applicable to the training of large neural networks due to their large computational and communication overheads. As a scalable technique for shared model governance, we propose splitting deep learning model between multiple parties. This paper empirically investigates the security guarantee of this technique, which is introduced as the problem of model completion: Given the entire training data set or an environment simulator, and a subset of the parameters of a trained deep learning model, how much training is required to recover the model’s original performance? We define a metric for evaluating the hardness of the model completion problem and study it empirically in both supervised learning on ImageNet and reinforcement learning on Atari and DeepMind Lab. Our experiments show that the model completion problem is harder in reinforcement learning than in supervised learning because of the unavailability of the trained agent’s trajectories, and its hardness depends not primarily on the number of parameters of the missing part, but more so on their type and location. Our suggest that model splitting might be a feasible technique for shared model governance in some settings where training is very expensive. With an increasing number of deep learning models being deployed in production, questions regarding data privacy and misuse are being raised BID4. The trend of training larger models on more data BID28, training models becomes increasingly expensive. Especially in a continual learning setting where models get trained over many months or years, they accrue a lot of value and are thus increasingly susceptible to theft. This prompts for technical solutions to monitor and enforce control over these models BID41. We are interested in the special case of shared model governance: Can two or more parties jointly train a model such that each party has to consent to every forward (inference) and backward pass (training) through the model?Two popular methods for sharing model governance are homomorphic encryption (HE; BID36 and secure multi-party computation (MPC; BID48 . The major downside of both techniques is the large overhead incurred by every multiplication, both computationally, >1000x for HE BID29 BID14, >24x for MPC BID25 BID7, in addition to space (>1000x in case of HE) and communication (>16 bytes per 16 bit floating point multiplication in case of MPC). Unfortunately, this makes HE and MPC inapplicable to the training of large neural networks. As scalable alternative for sharing model governance with minimal overhead, we propose the method of model splitting: distributing a deep learning model between multiple parties such that each party holds a disjoint subset of the model's parameters. Concretely, imagine the following scenario for sharing model governance between two parties, called Alice and Bob. Alice holds the model's first layer and Bob holds the model's remaining layers. In each training step Alice does a forward pass through the first layer, sends the ing activations to Bob, Bob completes the forward pass, computes the loss from the labels, and does a backward pass to the first layer, sends the ing gradients to Alice, and Alice finishes the backward pass. How much security would Alice and Bob enjoy in this setting? To answer this question, we have to consider the strongest realistic attack vector. In this work we assume that the adversary has access to everything but the missing parameters held by the other party. How easy would it be for this adversary to recover the missing part of the model? We introduce this as the problem of model completion:Given the entire training data set or an environment simulator, and a subset of the parameters of a trained model, how much training is required to recover the model's original performance?In this paper, we define the problem of model completion formally (Section 3.1), propose a metric to measure the hardness of model completion (Section 3.2), and provide empirical (Section 4 and Section 5) in both the supervised learning (SL) and in reinforcement learning (RL). For our SL experiments we use the AlexNet convolutional network BID26 and the ResNet50 residual network BID17 on ImageNet BID9 ); for RL we use A3C and Rainbow BID19 in the Atari domain BID1 and IMPALA BID11 on DeepMind Lab BID0. After training the model, we reinitialize one of the model's layers and measure how much training is required to complete it (see FIG1).Our key findings are: Residual networks are easier to complete than nonresidual networks (The closest well-studied phenomenon to model completion is unsupervised pretraining, first introduced by BID22 . In unsupervised pretraining a subset of the model, typically the lower layers, is trained in a first pass with an unsupervised reconstruction loss BID10 . The aim is to learn useful high-level representations that make a second pass with a supervised loss more computationally and sample efficient. This second pass could be thought as model completion. In this paper we study vertical model completion where all parameters in one layer have to be completed. Instead we could have studied horizontal model completion where some parameters have to be completed in every layer. Horizontal model completion should be easy as suggested by the effectiveness of dropout as a regularizer BID40, which trains a model to be resilient to horizontal parameter loss. Pruning neural networks BID27) is in a sense the reverse operation to model completion. BID6 prune individual connections and BID33 prune entire feature maps using different techniques; their findings, lower layers are more important, are compatible with ours. BID13 present empirical evidence for the lottery ticket hypothesis: only a small subnetwork matters (the 'lottery ticket') and the rest can be pruned away without loss of performance. The model completion problem for this lottery ticket (which is spread over all layers) would be trivial by definition. All of these works only consider removing parts of the model horizontally. The model completion problem can also be viewed as transfer learning from one task to the same task, while only sharing a subset of the parameters BID34 BID44. BID49 investigate which layers in a deep convolutional model contain general versus task-specific representations; some of their experiments follow the same setup as we do here and their are in line with ours, but they do not measure the hardness of model completion task. Finally, our work has some connections to distillation of deep models BID5 BID3. Distillation can be understood as a'reverse' of model completion, where we want to find a smaller model with the same performance instead of completing a smaller, partial model. The literature revolves around two techniques for sharing model governance: homomorphic encryption (HE; BID36 and secure multi-party computation (MPC; BID48 BID8 . Both HE and MPC have been successfully applied to machine learning on small datasets like MNIST BID14 BID32 BID7 BID46 and the Wisconsin Breast Cancer Data set BID16 .HE is an encryption scheme that allows computation on encrypted numbers without decrypting them. It thus enables a model to be trained by an untrusted third party in encrypted form. The encryption key to these parameters can be cryptographically shared between several other parties who effectively retain control over how the model is used. Using MPC numbers can be shared across several parties such that each share individually contains no information about these numbers. Nevertheless computational operations can be performed on the shared numbers if every party performs operations on their share. The of the computation can be reconstructed by pooling the shares of the . While both HE and MPC fulfill a similar purpose, they face different tradeoffs for the additional security benefits: HE incurs a large computational overhead BID29 while MPC incurs a much smaller computational overhead in exchange for a greater communication overhead BID25 . Moreover, HE provides cryptographic security (reducing attacks to break the cipher on well-studied hard problems such as the discrete logarithm) while MPC provides perfect information-theoretic guarantees as long as the parties involved (3 or more) do not collude. There are many applications where we would be happy to pay for the additional overhead because we cannot train the model any other way, for example in the health sector where privacy and security are critical. However, if we want to scale shared model governance to the training of large neural networks, both HE and MPC are ruled out because of their prohibitive overhead. In contrast to HE and MPC, sharing governance via model splitting incurs minimal computational and manageable communication overhead. However, instead of strong security guarantees provided by HE and MPC, the security guarantee is bounded from above by the hardness of the model completion problem we study in this paper. Let f θ be a model parameterized by the vector θ. We consider two settings: supervised learning and reinforcement learning. In our supervised learning experiments we evaluate the model f θ by its performance on the test loss L(θ).In reinforcement learning an agent interacts with an environment over a number of discrete time steps BID43: In time step t, the agent takes an action a t and receives an observation o t+1 and a reward r t+1 ∈ R from the environment. We consider the episodic setting in which there is a random final time step τ ≤ K for some constant K ∈ N, after which we restart with timestep t = 1. The agent's goal is to maximize the episodic return G:= τ t=1 r t. Its policy is a mapping from sequences of observations to a distribution over actions parameterized by the model f θ. To unify notation for SL and RL, we equate L(θ) = E at∼f θ (o1,...,ot−1) [−G] such that the loss function for RL is the negative expected episodic return. To quantify training costs we measure the computational cost during (re)training. To simplify, we assume that training proceeds over a number of discrete steps. A step can be computation of gradients and parameter update for one minibatch in the case of supervised learning or one environment step in the case of reinforcement learning. We assume that computational cost are constant for each step, which is approximately true in our experiments. This allows us to measure training cost through the number of training steps executed. Let T denote the training procedure for the model f θ and let θ 0, θ 1,... be the sequence of parameter vectors during training where θ i denotes the parameters in training step i. Furthermore, let *:= min{L(θ i) | i ≤ N } denote the best model performance during the training procedure T (not necessarily the performance of the final weights). We define the training cost as the random variable C T := arg min i∈N {L(θ i) ≤ }, the number of training steps until the loss falls below the given threshold ∈ R. After we have trained the model f θ for N steps and thus end up with a set of trained parameters θ N with loss L(θ N), we split the parameters θ N = [θ How hard is the model completion problem? To answer this question, we use the parameters DISPLAYFORM0 N are the previously trained parameters and θ 0 2 are freshly initialized parameters. We then execute a (second) retraining procedure T ∈ T from a fixed set of available retraining procedures T.1 The aim of this retraining procedure is to complete the model, and it may be different from the initial training procedure T. We assume that T ∈ T since retraining the entire model from scratch (reinitializing all parameters) is a valid way to complete the model. Let θ 0, θ 1,... be the sequence of parameter vectors obtained from running the retraining procedure T ∈ T. Analogously to before, we define C T := arg min i∈N {L(θ i) ≤ } as the retraining cost to get a model whose test loss is below the given threshold ∈ R. Note that by definition, for T = T we have that C T is equal to C T in expectation. In addition to recovering a model with the best original performance *, we also consider partial model completion by using some higher thresholds * DISPLAYFORM1. These higher thresholds * α correspond to the relative progress α from the test loss of the untrained model parameters L(θ 0) to the best test loss DISPLAYFORM2 We define the hardness of model completion as the expected cost to complete the model as a fraction of the original training cost for the fastest retraining procedure T ∈ T available: DISPLAYFORM3 where the expectation is taken over all random events in the training procedures T and T.It is important to emphasize that the hardness of model completion is a relative measure, depending on the original training cost C T (* α). This ensures that we can compare the hardness of model completion across different tasks and different domains. In particular, for different values of α we compare like with like: MC-hardness T (α) is measured relative to how long it took to get the loss below the threshold * α during training. Importantly, it is not relative to how long it took to train the model to its best performance *. This means that naively counter-intuitive such as MC-hardness T (0.8) being less than MC-hardness T (0.5) are possible. Since C T and C T are nonnegative, MC-hardness T (α) is nonnegative. Moreover, since T ∈ T by assumption, we could retrain all model parameters from scratch (formally setting T to T). Thus we have MC-hardness T (α) ≤ 1, and therefore MC-hardness is bounded between 0 and 1. Equation 1 denotes an infimum over available retraining procedures T. However, in practice there is a vast number of possible retraining procedures we could use and we cannot enumerate and run all of them. Instead, we take an empirical approach for estimating the hardness of model completion: we investigate the following set of retraining strategies T to complete the model. All the retraining strategies, if not noted otherwise, are built on top of the original training procedure T. Our best are only an upper bound on the hardness of model completion. It is likely that much faster retraining procedures exist. T 1 Optimizing θ 0 1 and θ 0 2 jointly. We repeat the original training procedure T on the preserved parameters θ 0 1 and reinitialized parameters θ 0 2. The objective function is optimized with respect to all the trainable variables in the model. We might vary in hyperparameters such as learning rates or loss weighting schemes compared to T, but keep hyperparameters that change the structure of the model (e.g. size and number of layers) fixed. T 2 Optimizing θ 0 2, but not θ 0 1. Similarly to T 1, in this retraining procedure we keep the previous model structure. However, we freeze the trained weights θ 0 1, and only train the reinitialized parameters θ 0 2.T 3 Overparametrizing the missing layers. This builds on retraining procedure T 1. Overparametrization is a common trick in computer vision, where a model is given a lot more parameters than required, allowing for faster learning. This idea is supported by the'lottery ticket hypothesis' BID13: a larger number of parameters increases the odds of a subpart of the network having random initialization that is more conducive to optimization. T 4 Reinitializing parameters θ 0 2 using a different initialization scheme. Previous research shows that parameter initialization schemes can have a big impact on convergence properties of deep neural networks BID15 BID42. In T 1 our parameters are initialized using a glorot uniform scheme. This retraining procedure is identical to T 1 except that we reinitialize θ 0 2 using one of the following weight initialization schemes: glorot normal BID15, msra BID18 or caffe BID24. Our main experimental establish upper bounds on the hardness of model completion in the context of several state of the art models for both supervised learning and reinforcement learning. In all the experiments, we train a model to a desired performance level (this does not have to be stateof-the-art performance), and then reinitialize a specific part of the network and start the retraining procedure. Each experiment is run with 3 seeds, except IMPALA (5 seeds) and A3C (10 seeds).Supervised learning. We train AlexNet BID26 and ResNet50 BID17 on the ImageNet dataset BID9 to minimize cross-entropy loss. The test loss is the top-1 error rate on the test set. AlexNet is an eight layer convolutional network consisting of five convolutional layers with max-pooling, followed by two fully connected layers and a softmax output layer. ResNet50 is a 50 layer convolutional residual network: The first convolutional layer with max-pooling is followed by four sections, each with a number of ResNet blocks (consisting of two convolutional layers with skip connections and batch normalization), followed by average pooling, a fully connected layer and a softmax output layer. We apply retraining procedures T 1 and T 2 and use a different learning rate schedule than in the original training procedure because it performs better during retraining. All other hyperparameters are kept the same. Reinforcement learning. We consider three different state of the art agents: A3C BID31, Rainbow BID19 and the IMPALA reinforcement learning agent BID11. A3C comes from a family of actor-critic methods which combine value learning and policy gradient approaches in order to reduce the variance of the gradients. Rainbow is an extension of the standard DQN agent, which combines double Q-learning (van), dueling networks BID47 ), distributional RL (and noisy nets BID12 . Moreover, it is equipped with a replay buffer that stores the previous million transitions of the form (o t, a t, r t+1, o t+1), which is then sampled using a prioritized weighting scheme based on temporal difference errors BID38. Finally, IMPALA is an extension of A3C, which uses the standard actor-critic architecture with off-policy corrections in order to scale effectively to a large scale distributed setup. We train IMPALA with population based training BID23.For A3C and Rainbow we use the Atari 2600 domain BID1 and for IMPALA DeepMind Lab BID0. In both cases, we treat the list of games/levels as a single learning problem by averaging across games in Atari and training the agent on all level in parallel in case of DeepMind Lab. In order to reduce the noise in the MC-hardness metric, caused by agents being unable to learn the task and behaving randomly, we filter out the levels in which the original trained agent performs poorly. We apply the retraining procedures T 1, T 2 on all the models, and on A3C we apply additionally T 3 and T 4. All the hyperparameters are kept the same during the training and retraining procedures. Further details of the training and retraining procedures for all models can be found in Appendix A, and the parameter counts of the layers are listed in Appendix B. Our experimental on the hardness of the model completion problem are reported in FIG4. These figures show on the x-axis different experiments with different layers being reinitialized (lower to higher layers from left to right). We plot MC-hardness T (α) as a bar plot with error bars showing the standard deviation over multiple experiment runs with different seeds; the colors indicate different values of α. The numbers are provided in Appendix C. In the following we discuss the .1. In the majority of cases, T 1 is the best of our retraining procedures. From the retraining procedures listed in Section 3.3 we use T 1 and T 2 in all experiments and find that T 1 performs substantially better in all settings except two: First, for A3C, starting from the third convolutional layer, T 2 has lower MC-hardness for all the threshold levels (FIG7 . Second, T 2 performs well on all the layers when retraining ResNet-50, for all α ≤ 0.9 ( FIG2 ; the difference is especially visible at α = 0.9.For A3C we use all four retraining procedures. The difference between T 1 and T 2 are shown in FIG7 . For T 3 we tried replacing the first convolutional layer with two convolutional layers using a different kernel size, as well as replacing a fully connected layer with two fully connected layers of varying sizes. The were worse than using the same architecture and we were often unable to retrieve 100% of the original performance. With T 4 we do not see any statistically significant difference in retraining time between the initialization schemes glorot normal, msra, and caffe. FIG4 and FIG2 for T 1, the model hardness for threshold α = 0.5 and α = 0.8 is much lower for ResNet50 than for AlexNet. However, to get the original model performance (α = 1), both models need about 40% of the original training cost. As mentioned above, T 2 works better than T 1 on ResNet50 for α ≤ 0.9. An intact skip connection helps retraining for α ≤ 0.9 and T 1, but not T 2, as illustrated in the experiment S4 B1 -W FIG2. A noticeable outlier is S4 B1 at α = 0.9; it is unclear what causes this effect, but it reproduced every time we ran this experiment. Residual neural networks use skip connections across two or more layers BID17. This causes the features in those layers to be additive with respect to the incoming features, rather than replacing them as in non-residual networks. Thus lower-level and higher-level representations tend to be more spread out across the network, rather than being confined to lower and higher layers, respectively. This would explain why model completion in residual networks is more independent of the location of the layer.3. For A3C lower layers are often harder to complete than upper layers. FIG7 shows that for A3C the lower layers are harder to complete than the higher layers since for each value of α the MC-hardness decreases from left to right. However, this effect is much smaller for Rainbow (Figure 5) and AlexNet FIG4 ).In nonresidual networks lower convolutional layers typically learn much simpler and more general features that are more task independent BID49. Moreover, noise perturbations of lower layers have a significantly higher impact on the performance of deep learning models since noise grows exponentially through the network layers BID35. Higher level activations are functions of the lower level ones; if a lower layer is reset, all subsequent activations will be invalidated. This could imply that the gradients on the higher layers are incorrect and thus slow down training.4. The absolute number of parameters has a minimal effect on the hardness of model completion. If information content is spread uniformly across the model, then model completion should be a linear function in the number of parameters that we remove. However, the number of parameters in deep models usually vary greatly between layers; the lower-level convolutional layers have 2-3 orders of magnitude fewer parameters than the higher level fully connected layers and LSTMs (see Appendix B).In order to test this explicitly, we performed an experiment on AlexNet both increasing and decreasing the total number of feature maps and fully connected units in every layer by 50%, ing in approximately an order of magnitude difference in terms of parameters between the two models. We found that there is no significant difference in MC-hardness across all threshold levels.5. RL models are harder to complete than SL models. Across all of our experiments, the model completion of individual layers for threshold α = 1 in SL FIG4 and FIG2 is easier than the model completion in RL FIG7, Figure 5, and Figure 6 ). In many cases the same holds from lower thresholds as well. By resetting one layer of the model we lose access to the agent's ability to generate useful experience from interaction with the environment. As we retrain the model, the agent has to re-explore the environment to gather the right experience again, which takes extra training time. While this effect is also present during the training procedure T, it is possible that resetting one layer makes the exploration problem harder than acting from a randomly initialized network.6. When completing RL models access to the right experience matters. To understand this effect better, we allow the retraining procedure access to Rainbow's replay buffer. At the start of retraining this replay buffer is filled with experience from the fully trained policy. Figure 5 shows that the model completion hardness becomes much easier with access to this replay buffer: the three left bar plots are lower than the three right. This is supported by the benefits of kickstarting BID39, where a newly trained agent gets access to an expert agent's policy. Moreover, this is consistent with findings by BID20, who show performance benefits by adding expert trajectories to the replay buffer. Our shed some initial glimpse on the model completion problem and its hardness. Our findings include: residual networks are easier to complete than non-residual networks, lower layers are often harder to complete than higher layers, and RL models are harder to complete than SL models. Nevertheless several question remain unanswered: Why is the difference in MC-hardness less pronounced between lower and higher layers in Rainbow and AlexNet than in A3C? Why is the absolute number of parameters insubstantial? Are there retraining procedures that are faster than T 1? Furthermore, our definition of hardness of the model completion problem creates an opportunity to modulate the hardness of model completion. For example, we could devise model architectures with the explicit objective that model completion be easy (to encourage robustness) or hard (to increase security when sharing governance through model splitting). Importantly, since Equation 1 can be evaluated automatically, we can readily combine this with architecture search BID50.Our experiments show that when we want to recover 100% of the original performance, model completion may be quite costly: ∼ 40% of the original training costs in many settings; lower performance levels often retrain significantly faster. In scenarios where a model gets trained over many months or years, 40% of the cost may be prohibitively expensive. However, this number also has to be taken with a grain of salt because there are many possible retraining procedures that we did not try. The security properties of model splitting as a method for shared governance require further investigation: in addition to more effective retraining procedures, an attacker may also have access to previous activations or be able to inject their own training data. Yet our experiments suggest that model splitting could be a promising method for shared governance. In contrast to MPC and HE it has a substantial advantage because it is cost-competitiveness with normal training and inference. Learning rate Training batches Retraining batches 5e − 2 0 0 5e − 3 60e3 30e3 5e − 4 90e3 45e3 5e − 5 105e3 72.5e3 Table 1: AlexNet: Learning schedule for training and retraining procedures. 1e − 1 0 / 1e − 2 30e3 0 1e − 3 45e3 20e3 AlexNet We train this model for 120e3 batches, with batch size of 256. We apply batch normalization on the convolutional layers and 2 -regularization of 1e-4. Optimization is done using Momentum SGD with momentum of 0.9 and the learning rate schedule which is shown in Table 1. Note that the learning schedule during retraining is 50% faster than during training (for T 1 and T 2).For both retraining procedures T 1 and T 2, we perform reset for each of the first 5 convolutional layers and the following 2 fully connected layers. TAB3 shows the number of trainable parameters for each of the layers. We perform all training and retraining procedures for 60e3 batches, with batch size of 64 and 2 -regularization of 1e-4. Optimization is done using Momentum SGD with momentum of 0.9 and the learning rate schedule shown in TAB1.For our experiments, we reinitialize the very first convolutional layer, as well as the first ResNet block for each of the four subsequent network sections. In the'S4 B1 -W' experiment, we leave out resetting the learned skip connection. Finally, we also reset the last fully connected layer containing logits. A3C Each agent is trained on a single Atari level for 5e7 environment steps, over 10 seeds. We use the standard Atari architecture consisting of 3 convolutional layers, 1 fully connected layer and 2 fully connected'heads' for policy and value function. The number of parameters for each of those layers is shown in TAB5. For optimization, we use RMSProp optimizer with = 0.1, decay of 0.99 and α = 6e-4 that is linearly annealed to 0. For all the other hyperparameters we refer to BID31. Finally, while calculating reported statistics we removed the following Atari levels, due to poor behaviour of the trained agent: Montezuma's Revenge, Venture, Solaris, Enduro, Battle Zone, Gravitar, Kangaroo, Skiing, Krull, Video pinball, Freeway, Centipede, and Robotank. Rainbow Each agent is trained on a single Atari level for 20e6 environment frames, over 3 seeds. Due to agent behaving randomly, we remove the following Atari games from our MC-hardness calculations: Montezuma's Revenge, Venture, and Solaris. For our experiments, we use the same network architecture and hyperparameters as reported in BID19 and target the first 3 convolutional layers. TAB6 has the total number of parameters for each of the 3 layers. IMPALA We train a single agent over a suite of 28 DeepMind Lab levels for a total of 1 billion steps over all the environments, over 5 seeds. During training we apply population based training (PBT; BID23 with population of size 12, in order to evolve the entropy cost, learning rate and for RMSProp. For language modelling a separated LSTM channel is used. In the we report, we removed two DeepMind Lab levels due to poor behavior of the trained agent:'language_execute_random_task' and'psychlab_visual_search'. All the other hyperparameters are retained from BID11. For our experiments, we reinitialize the first convolutional layer TAB7. 1.00 1.00 0.00 1.00 1.00

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We study empirically how hard it is to recover missing parts of trained models

Represent the short summary for finding the corresponding scientific article (long):

Contrary to previous beliefs, the training performance of deep networks, when measured appropriately, is predictive of test performance, consistent with classical machine learning theory.