Question: Represent the scientific article to find a short TLDR.

Question: We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not deterministic if depth and width simultaneously tend to infinity. Moreover, we prove that for such deep and wide networks, the NTK has a non-trivial evolution during training by showing that the mean of its first SGD update is also exponential in the ratio of network depth to width. This is sharp contrast to the regime where depth is fixed and network width is very large. Our suggest that, unlike relatively shallow and wide networks, deep and wide ReLU networks are capable of learning data-dependent features even in the so-called lazy training regime. Modern neural networks are typically overparameterized: they have many more parameters than the size of the datasets on which they are trained. That some setting of parameters in such networks can interpolate the data is therefore not surprising. But it is a priori unexpected that not only can such interpolating parameter values can be found by stochastic gradient descent (SGD) on the highly non-convex empirical risk but also that the ing network function generalizes to unseen data. In an overparameterized neural network N (x) the individual parameters can be difficult to interpret, and one way to understand training is to rewrite the SGD updates ∆θ p = − λ ∂L ∂θ p, p = 1,..., P of trainable parameters θ = {θ p} P p=1 with a loss L and learning rate λ as kernel gradient descent updates for the values N (x) of the function computed by the network: Here B = {(x 1, y 1),..., (x |B|, y |B|)} is the current batch, the inner product is the empirical 2 inner product over B, and K N is the neural tangent kernel (NTK): Relation is valid to first order in λ. It translates between two ways of thinking about the difficulty of neural network optimization: (i) The parameter space view where the loss L, a complicated function of θ ∈ R #parameters, is minimized using gradient descent with respect to a simple (Euclidean) metric; (ii) The function space view where the loss L, which is a simple function of the network mapping x → N (x), is minimized over the manifold M N of all functions representable by the architecture of N using gradient descent with respect to a potentially complicated Riemannian metric K N on M N. A remarkable observation of is that K N simplifies dramatically when the network depth d is fixed and its width n tends to infinity. In this setting, by the universal approximation theorem , the manifold M N fills out any (reasonable) ambient linear space of functions. The in then show that the kernel K N in this limit is frozen throughout training to the infinite width limit of its average E[K N] at initialization, which depends on the depth and non-linearity of N but not on the dataset. This mapping between parameter space SGD and kernel gradient descent for a fixed kernel can be viewed as two separate statements. First, at initialization, the distribution of K N converges in the infinite width limit to the delta function on the infinite width limit of its mean E[K N]. Second, the infinite width limit of SGD dynamics in function space is kernel gradient descent for this limiting mean kernel for any fixed number of SGD iterations. As long as the loss L is well-behaved with respect to the network outputs N (x) and E[K N] is non-degenerate in the subspace of function space given by values on inputs from the dataset, SGD for infinitely wide networks will converge with probability 1 to a minimum of the loss. Further, kernel method-based theorems show that even in this infinitely overparameterized regime neural networks will have non-vacuous guarantees on generalization . But replacing neural network training by gradient descent for a fixed kernel in function space is also not completely satisfactory for several reasons. First, it suggests that no feature learning occurs during training for infinitely wide networks in the sense that the kernel E[K N] (and hence its associated feature map) is data-independent. In fact, empirically, networks with finite but large width trained with initially large learning rates often outperform NTK predictions at infinite width. One interpretation is that, at finite width, K N evolves through training, learning data-dependent features not captured by the infinite width limit of its mean at initialization. In part for such reasons, it is important to study both empirically and theoretically finite width corrections to K N. Another interpretation is that the specific NTK scaling of weights at initialization (b; a; ; 2018; a; b) and the implicit small learning rate limit obscure important aspects of SGD dynamics. Second, even in the infinite width limit, although K N is deterministic, it has no simple analytical formula for deep networks, since it is defined via a layer by layer recursion. In particular, the exact dependence, even in the infinite width limit, of K N on network depth is not well understood. Moreover, the joint statistical effects of depth and width on K N in finite size networks remain unclear, and the purpose of this article is to shed light on the simultaneous effects of depth and width on K N for finite but large widths n and any depth d. Our apply to fully connected ReLU networks at initialization for which our main contributions are: 1. In contrast to the regime in which the depth d is fixed but the width n is large, K N is not approximately deterministic at initialization so long as d/n is bounded away from 0. Specifically, for a fixed input x the normalized on-diagonal second moment of K N satisfies Thus, when d/n is bounded away from 0, even when both n, d are large, the standard deviation of K N (x, x) is at least as large as its mean, showing that its distribution at initialization is not close to a delta function. See Theorem 1. 2. Moreover, when L is the square loss, the average of the SGD update ∆K N (x, x) to K N (x, x) from a batch of size one containing x satisfies where n 0 is the input dimension. Therefore, if d 2 /nn 0 > 0, the NTK will have the potential to evolve in a data-dependent way. Moreover, if n 0 is comparable to n and d/n > 0 then it is possible that this evolution will have a well-defined expansion in d/n. See Theorem 2. In both statements above, means is bounded above and below by universal constants. We emphasize that our hold at finite d, n and the implicit constants in both and in the error terms Under review as a conference paper at ICLR 2020 2 ) are independent of d, n. Moreover, our precise , stated in §2 below, hold for networks with variable layer widths. We have denoted network width by n only for the sake of exposition. The appropriate generalization of d/n to networks with varying layer widths is the parameter which in light of the estimates in and plays the role of an inverse temperature. A number of articles (; ; ;) have followed up on the original NTK work Jacot et al. (2018) can be promoted to d/n expansions. Also, the sum-over-path approach to studying correlation functions in randomly initialized ReLU nets was previously taken up for the forward pass by and for the backward pass by and. Consider a ReLU network N with input dimension n 0, hidden layer widths n 1,..., n d−1, and output dimension n d = 1. We will assume that the output layer of N is linear and initialize the biases in N to zero. Therefore, for any input x ∈ R n0, the network N computes given by where for i = 1,..., d − 1 and µ is a fixed probability measure on R that we assume has a density with respect to Lebesgue measure and satisfies: µ is symmetric around 0, The three assumptions in hold for virtually all standard network initialization schemes with the exception of orthogonal weight initialization. But believe our extend hold also for this case but not do take up this issue. The on-diagonal NTK is and we emphasize that although we have initialized the biases to zero, they are not removed them from the list of trainable parameters. Our first is the following: Theorem 1 (Mean and Variance of NKT on Diagonal at Init). We have Moreover, we have that E[K N (x, x) 2 ] is bounded above and below by universal constants times In particular, if all the hidden layer widths are equal (i.e. n i = n, for i = 1, . . ., d − 1), we have where f g means f is bounded above and below by universal constants times g. This shows that in the deep and wide double scaling limit the NTK does not converge to a constant in probability. This is contrast to the wide and shallow regime where n i → ∞ and d < ∞ is fixed. Our next shows that when L is the square loss K N (x, x) is not frozen during training. To state it, fix an input x ∈ R n0 to N and define ∆K N (x, x) to be the update from one step of SGD with a batch of size 1 containing x (and learning rate λ). Theorem 2 (Mean of Time Derivative of NTK on Diagonal at Init). We have that E λ −1 ∆K N (x, x) is bounded above and below by universal constants times times a multiplicative error of size 1/n i, as in Theorem 1. In particular, if all the hidden layer widths are equal (i.e. n i = n, for i = 1, . . ., d − 1), we find Observe that when d is fixed and n i = n → ∞, the pre-factor in front of exp (5β) scales like 1/n. This is in keeping with the from and. Moreover, it shows that if d, n, n 0 grow in any way so that dβ/n 0 = d 2 /nn 0 → 0, the update ∆K N (x, x) to K N (x, x) from the batch {x} at initialization will have mean 0. It is unclear whether this will be true also for larger batches and when the arguments of K N are not equal. In contrast, if n i n and β = d/n is bounded away from 0, ∞, and the n 0 is proportional to d, the average update E[∆K N (x, x)] has the same order of magnitude as E[K N (x)]. The remainder of this article is structured as follows. First, we give an outline of the proofs of Theorems 1 and 2 in §3 and particularly in §3.1, which gives an in-depth but informal explanation of our strategy for computing moments of K N and its time derivative. Next, in the Appendix Section §A, we introduce some notation about paths and edges in the computation graph of N. This notation will be used in the proofs of Theorems 1 and 2 presented in the Appendix Section §B- §D. The computations in §B explain how to handle the contribution to K N and ∆K N coming only from the weights of the network. They are the most technical and we give them in full detail. Then, the discussion in §C and §D show how to adapt the method developed in §B to treat the contribution of biases and mixed bias-weight terms in K N, K 2 N and ∆K N. Since the arguments are simpler in these cases, we omit some details and focus only on highlighting the salient differences. 3 OVERVIEW OF PROOF OF THEOREMS 1 AND 2 The proofs of Theorems 1 and 2 are so similar that we will prove them at the same time. In this section and in §3.1 we present an overview of our argument. Then, we carry out the details in Appendix Sections §B- §D below. Fix an input x ∈ R n0 to N. Recall from that where we've set and have suppressed the dependence on x, N. Similarly, we have where we have introduced and have used that the loss on the batch {x} is given by 2 for some target value N * (x). To prove Theorem 1 we must estimate the following quantities: To prove Theorem 2, we must control in addition The most technically involved computations will turn out to be those involving only weights: namely, the terms. These terms are controlled by writing each as a sum over certain paths γ that traverse the network from the input to the output layers. The corresponding for terms involving the bias will then turn out to be very similar but with paths that start somewhere in the middle of network (corresponding to which bias term was used to differentiate the network output). The main about the pure weight contributions to K N is the following Proposition 3 (Pure weight moments for K N, ∆K N). We have Finally, We prove Proposition 3 in §B below. The proof already contains all the ideas necessary to treat the remaining moments. In §C and §D we explain how to modify the proof of Proposition 3 to prove the following two Propositions: Proposition 4 (Pure bias moments for K N, ∆K N). We have Moreover, Finally, with probability 1, we have ∆ bb = 0. Proposition 5 (Mixed bias-weight moments for K N, ∆K N). We have Further, E[∆ wb] is bounded above and below by universal constants times The statements in Theorems 1 and 2 that hold for general n i now follow directly from Propositions 3-5. The asymptotics when n i n follow from some routine algebra. Before turning to the details of the proof of Propositions 3-5 below, we give an intuitive explanation of the key steps in our sum-over-path analysis of the moments of Since the proofs of all three Propositions follow a similar structure and Proposition 3 is the most complicated, we will focus on explaining how to obtain the first 2 moments of K w. Since the biases are initialized to zero and K w involves only derivatives with respect to the weights, for the purposes of analyzing K w the biases play no role. Without the biases, the output of the neural network, N (x) can be express as a weighted sum over paths in the computational graph of the network: where Γ 1 a is the collection of paths in N starting at neuron a and the weight of a path wt(γ) is defined in in the Appendix and includes both the product of the weights along γ and the condition that every neuron in γ is open at x. The path γ begins at some neuron in the input layer of N and passes through a neuron in every subsequent layer until ending up at the unique neuron in the output layer (see). Being a product over edge weights in a given path, the derivative of wt(γ) with respect to a weight W e on an edge e of the computational graph of N is: There is a subtle point here that wt(γ) also involves indicator functions of the events that neurons along γ are open at x. However, with probability 1, the derivative with respect to W e of these indicator functions is identically 0 at x. The details are in Lemma 11. Because K w is a sum of derivatives squared (see), ignoring the dependence on the network input x, the kernel K w roughly takes the form where the sum is over collections (γ 1, γ 2) of two paths in the computation graph of N and edges e in the computational graph of N that lie on both (see Lemma 6 for the precise statement). When computing the mean, E[K w], by the mean zero assumption of the weights W e (see), the only contribution is when every edge in the computational graph of N is traversed by an even number of paths. Since there are exactly two paths, the only contribution is when the two paths are identical, dramatically simplifying the problem. This gives rise to the simple formula for E[K w] (see). The expression w is more complex. It involves sums over four paths in the computational graph of N as in the second statement of Lemma 6. Again recalling that the moments of the weights have mean 0, the only collections of paths that contribute to E[K 2 w] are those in which every edge in the computational graph of N is covered an even number of times: However, there are now several ways the four paths can interact to give such a configuration. It is the combinatorics of these interactions, together with the stipulation that the marked edges e 1, e 2 belong to particular pairs of paths, which complicates the analysis of E[K 2 w]. We estimate this expectation in several steps: 1. Obtain an exact formula for the expectation in: where F (Γ, e 1, e 2) is the product over the layers = 1,..., d in N of the "cost" of the interactions of γ 1,..., γ 4 between layers − 1 and. The precise formula is in Lemma 7. 2. Observe the dependence of F (Γ, e 1, e 2) on e 1, e 2 is only up to a multiplicative constant: F (Γ, e 1, e 2) F * (Γ). The precise relation is. This shows that, up to universal constants, γ1,γ2 togethe at layer 1 γ3,γ4 togethe at layer 2. This is captured precisely by the terms I j, II j defined in,. 3. Notice that F * (Γ) depends only on the un-ordered multiset of edges E = E Γ ∈ Σ 4 even determined by Γ (see for a precise definition). We therefore change variables in the sum from the previous step to find where Jacobian(E, e 1, e 2) counts how many collections of four paths Γ ∈ Γ 4 even that have the same E Γ also have paths γ 1, γ 2 pass through e 1 and paths γ 3, γ 4 pass through e 2. Lemma 8 gives a precise expression for this Jacobian. It turns outs, as explained just below Lemma 8, that Jacobian(E, e 1, e 2) 6 #loops(E), where a loop in E occurs when the four paths interact. More precisely, a loop occurs whenever all four paths pass through the same neuron in some layer (see Figures 1 and 2). 4. Change variables from unordered multisets of edges E ∈ Σ 4 even in which every edge is covered an even number of times to pairs of paths V ∈ Γ 2. The Jacobian turns out to be 2 −#loops(E) (Lemma 9), giving 5. Just like F * (V), the term 3 #loops(V) is again a product over layers in the computational graph of N of the "cost" of interactions between our four paths. Aggregating these two terms into a single functional F * (E) and factoring out the 1/n terms in F * (V) we find that: where the 1/n terms cause the sum to become an average over collections V of two independent paths in the computational graph of N, with each path sampling neurons uniformly at random in every layer. The precise , including the dependence on the input x, is in. 6. Finally, we use Proposition 10 to obtain for this expectation estimates above and below that match up multiplicative constants. Figure 1: Cartoon of the four paths γ 1, γ 2, γ 3, γ 4 between layers 1 and 2 in the case where there is no interaction. Paths stay with there original partners γ 1 with γ 2 and γ 3 with γ 4 at all intermediate layers. Figure 2: Cartoon of the four paths γ 1, γ 2, γ 3, γ 4 between layers 1 and 2 in the case where there is exactly one "loop" interaction between the marked layers. Paths swap away from their original partners exactly once at some intermediate layer after 1, and then swap back to their original partners before 2. Taken together Theorems 1 and 2 show that in fully connected ReLU nets that are both deep and wide the neural tangent kernel K N is genuinely stochastic and enjoys a non-trivial evolution during training. This suggests that in the overparameterized limit n, d → ∞ with d/n ∈ (0, ∞), the kernel K N may learn data-dependent features. Moreover, our show that the fluctuations of both K N and its time derivative are exponential in the inverse temperature β = d/n. It would be interesting to obtain an exact description of its statistics at initialization and to describe the law of its trajectory during training. Assuming this trajectory turns out to be data-dependent, our suggest that the double descent curve Belkin et al. (2018; 2019); that trades off complexity vs. generalization error may display significantly different behaviors depending on the mode of network overparameterization. However, it is also important to point out that the in;; show that, at least for fully connected ReLU nets, gradient-based training is not numerically stable unless d/n is relatively small (but not necessarily zero). Thus, we conjecture that there may exist a "weak feature learning" NTK regime in which network depth and width are both large but 0 < d/n 1. In such a regime, the network will be stable enough to train but flexible enough to learn data-dependent features. In the language of Chizat & Bach (2018b) one might say this regime displays weak lazy training in which the model can still be described by a stochastic positive definite kernel whose fluctuations can interact with data. Finally, it is an interesting question to what extent our hold for non-linearities other than ReLU and for network architectures other than fully connected (e.g. convolutional and residual). Typical ConvNets, for instance, are significantly wider than they are deep, and we leave it to future work to adapt the techniques from the present article to these more general settings. In this section, we introduce some notation, adapted in large part from , that will be used in the proofs of Theorems 1 and 2. For n ∈ N, we will write [n]:= {1, . . ., n}. It will also be convenient to denote k | every entry in a appears an even number of times}. Given a ReLU network N with input dimension n 0, hidden layer widths n 1,..., n d−1, and output dimension n d = 1, its computational graph is a directed multipartite graph whose vertex set is the disjoint union Definition 1 (Path in the computational graph of N)., a path γ in the computational graph of N from neuron z(1, α 1) to neuron z(2, α 2) is a collection of neurons in layers 1,..., 2: Further, we will write Given a collection of neurons γj is a path starting at neuron z(j,αj) ending at the output neuron z(d,1) Note that with this notation, we have Correspondingly, we will write If each edge e in the computational graph of N is assigned a weight W e, then associated to a path γ is a collection of weights: ) Definition 2 (Weight of a path in the computational graph of N). Fix 0 ≤ ≤ d, and let γ be a path in the computation graph of N starting at layer and ending at the output. The weight of a this path at a given input x to N is where, is the event that all neurons along γ are open for the input x. Here y is as in. Next, for an edge e ∈ [n i−1] × [n i] in the computational graph of N we will write for the layer of e. In the course of proving Theorems 1 and 2, it will be useful to associate to every Γ ∈ Γ k (n) an unordered multi-set of edges E Γ. Definition 3 (Unordered multisets of edges and their endpoints). For n, n, ∈ N set to be the unordered multiset of edges in the complete directed bi-paritite graph K n,n oriented from [n] to [n]. For every E ∈ Σ k (n, n) define its left and right endpoints to be where L(E), R(E) are unordered multi-sets. Using this notation, for any collection of edges between layers − 1 and that are present in Γ. Similarly, we will write Z,even }. We will moreover say that for a path γ an edge e = (α, β) ∈ [n i−1] × [n i] in the computational graph of N belongs to γ (written e ∈ γ) if Finally, for an edge e = (α, β) ∈ [n i−1] × [n i] in the computational graph of N, we set for the normalized and unnormalized weights on the edge corresponding to e (see). We begin with the well-known formula for the output of a ReLU net N with biases set to 0 and a linear final layer with one neuron: The weight of a path wt(γ) was defined in and includes both the product of the weights along γ and the condition that every neuron in γ is open at x. As explained in §A, the inner sum in is over paths γ in the computational graph of N that start at neuron a in the input layer and end at the output neuron and the random variables W γ are the normalized weights on the edge of γ between layer i − 1 and layer i (see). Differentiating this formula gives sum-over-path expressions for the derivatives of N with respect to both x and its trainable parameters. For the NTK and its first SGD update, the is the following: Lemma 6 (weight contribution to K N and ∆K N as a sum-over-paths). With probability 1, where the sum is over collections Γ of two paths in the computation graph of N and edges e that lie on both paths. Similarly, almost surely,, and plus a term that has mean 0. a, e ∈ γ, etc is defined in §A. We prove Lemma 6 in §B.1 below. Let us emphasize that the expressions for K 2 w and ∆ ww are almost identical. The main difference is that in the expression for ∆ ww, the second path γ 2 must contain both e 1 and e 2 while γ 4 has no restrictions. Hence, while for K 2 w the contribution from a collection of four paths Γ = (γ 1, γ 2, γ 3, γ 4) is the same as from the collection Γ = (γ 2, γ 1, γ 4, γ 3), for ∆ ww the contributions are different. This seemingly small discrepancy, as we shall see, causes the normalized expectation E[∆ ww]/E[K w] to converge to zero when d < ∞ is fixed and n i → ∞ (see the 1/n factors in the statement of Theorem 2). In contrast, in the same regime, the normalized second moment E[K Lemma 7 (Expectation of K w, K 2 w, ∆ ww as sums over 2, 4 paths). We have, where Similarly, where Finally, Lemma 7 is proved in §B.2. The expression is simple to evaluate due to the delta function in H(Γ, e). We obtain: where in the second-to-last equality we used that the number of paths in the comutational graph of N from a given neuron in the input to the output neuron equals i=1,...,d n i and in the last equality we used that n d = 1. This proves the first equality in Theorem 1. It therefore remains to evaluate and. Since they are so similar, we will continue to discuss them in parallel. To start, notice that the expression F (Γ, e 1, e 2) appearing in and satisfies where For the remainder of the proof we will write Thus, in particular, The advantage of F * (Γ) is that it does not depend on e 1, e 2. Observe that for every a = (α 1, α 2, α 3, α 4) ∈ [n 0] 4 even, we have that either α 1 = α 2, α 1 = α 3, or α 1 = α 4. Thus, by symmetry, the sum over Γ 4 even (n) in and we find and similarly, where F * (Γ)# {edges e 1, e 2 | e 1 ∈ γ 1 ∩ γ 2, e 2 ∈ γ 3 ∩ γ 4} F * (Γ)# {edges e 1, e 2 | e 1 ∈ γ 1 ∩ γ 2, e 2 ∈ γ 2, γ 3, e 1 = e 2}. To evaluate I j, II j let us write for the indicator function of the event that paths γ α, γ β pass through the same edge between layers i − 1, i in the computational graph of N. Observe that and # {edges e 1, e 2 | e 1 ∈ γ 1 ∩ γ 2, e 2 ∈ γ 2, γ 3, e 1 = e 2} = i2. Thus, we have where i2. To simplify I j,i1,i2 and II j,i1,i2 observe that F * (Γ) depends only on Γ only via the unordered edge multi-set (i.e. only which edges are covered matters; not their labelling) defined in Definition 3. Hence, we find that for j = 1, 2, 3, 4, i 1, i 2 = 1,..., d, The counts in I j, *,i1,i2 and II j, *,i1,i2 have a convenient representation in terms of Informally, the event C(E, i 1, i 2) indicates the presence of a "collision" of the four paths in Γ before the earlier of the layers i 1, i 2, while C(E, i 1, i 2) gives a "collision" between layers i 1, i 2; see Section 3.1 for the intuition behind calling these collisions. We also write Finally, for E ∈ Σ 4 a,even (n), we will define That is, a loop is created at layer i if the four edges in E all begin at occupy the same vertex in layer i − 1 but occupy two different vertices in layer i. We have the following Lemma. Lemma 8 (Evaluation of Counting Terms in and). Suppose E ∈ Σ 4 aj,even for some j = 1, 2, 3, 4. For each i 1, i 2 ∈ {1, . . ., d}, Similarly, We prove Lemma 8 in §B.3 below. Assuming it for now, observe that and that the conditions L(E) = a j are the same for j = 2, 3, 4 since the equality it is in the sense of unordered multi-sets. Thus, we find that E[K 2 w] is bounded above/below by a constant times Similarly, E[∆ ww] is bounded above/below by a constant times Observe that every unordered multi-set four edge multiset E ∈ Σ 4 even can be obtained by starting from some V ∈ Γ 2, considering its unordered edge multi-set E V and doubling all its edges. This map from Γ 2 to Σ 4 even is surjective but not injective. The sizes of the fibers is computed by the following Lemma., where as in, Lemma 9 is proved in §B.4. Using it and that 0 ≤ C(E, i 1, i 2) ≤ 1, the relation shows that E[K 2 w] is bounded above/below by a constant times Similarly, E[∆ ww] is bounded above/below by a constant times where, in analogy to, we have. Since the number of V in Γ 2 (n) with specified V equals, we find that so that for each and similarly, Here, E x is the expectation with respect to the probability measure on V = (v 1, v 2) ∈ Γ 2 obtained by taking v 1, v 2 independent, each drawn from the products of the measure We are now in a position to complete the proof of Theorems 1 and 2. To do this, we will evaluate the expectations E x above to leading order in i 1/n i with the help of the following elementary which is proven as Lemma 18 in. Proposition 10. Let A 0, A 1,..., A d be independent events with probabilities p 0,..., p d and B 0,..., B d be independent events with probabilities q 0,..., q d such that Denote by X i the indicator that the event A i happens, X i:= 1 {Ai}, and by Y i the indicator that B i happens, Then, if γ i ≥ 1 for every i, we have: where by convention α 0 = γ 0 = 1. In contrast, if γ i ≤ 1 for every i, we have: We first apply Proposition 10 to the estimates above for we find that Since the contribution for each layer in the product is bounded above and below by constants, we have that 2 is bounded below by a constant times and above by a constant times Here, note that the initial condition given by x and the terminal condition that all paths end at one neuron in the final layer are irrelevant. The expression is there precisely 3 ≤ 1, and K i = 1. Thus, since for i = 1,..., d − 1, the probability of X i is 1/n i + O(1/n 2 i), we find that where in the last inequality we used that 1 + x ≥ e When combined with this gives the lower bound in Proposition 3. The matching upper bound is obtained from in the same way using the opposite inequality from Proposition 10. To complete the proof of Proposition 3, we prove the analogous bounds for E[∆ ww] in a similar fashion. Namely, we fix 1 ≤ i 1 < i 2 ≤ d and write The set A is the event that the first collision between layers i 1, i 2 occurs at layer. We then have On the event A, notice that F * (V) only depends on the layers 1 ≤ i ≤ i 1 and layers < i ≤ d because the event A fixes what happens in layers i 1 < i ≤. Mimicking the estimates, and the application of Proposition 10 and using independence, we get that: Finally, we compute: Under review as a conference paper at ICLR 2020 Combining this we obtain that E[∆ ww]/ x 4 2 is bounded above and below by constants times This completes the proof of Proposition 3, modulo the proofs of Lemmas 6-9, which we supply below. Fix an input x ∈ R n0 to N. We will continue to write as in y (i) for the vector of pre-activations as layer i corresponding to x. We need the following simple Lemma. Lemma 11. With probability 1, either there exists i so that we have y For each fixed Γ this event defines a co-dimension 1 set in the space of all the weights. Hence, since the joint distribution of the weights has a density with respect to Lebesgue measure (see just before), the union of this (finite number) of events has measure 0. This shows that on the even that y = 0 for every, y (i) j = 0 with probability 1. Taking the union over i, j completes the proof. Lemma 11 shows that for our fixed x, with probability 1, the derivative of each ξ (i) j in vanishes. Hence, almost surely, for any edge e in the computational graph of N: This proves the formulas for K N, K 2 N. To derive the for ∆K N, we write where the loss L on a single batch containing only x is 1 2 (N (x) − N * (x)) 2. We therefore find Using and again applying Lemma 11, we find that with probability 1 Under review as a conference paper at ICLR 2020 Thus, almost surely To complete the proof of Lemma 6 it therefore remains to check that this last term has mean 0. To do this, recall that the output layer of N is assumed to be linear and that the distribution of each weight is symmetric around 0 (and hence has vanishing odd moments). Thus, the expectation over the weights in layer d has either 1 or 3 weights in it and so vanishes. Lemma 7 is almost a corollary of of Theorem 3 in and Proposition 2 in. The difference is that, in; , the biases in N were assumed to have a non-degenerate distribution, whereas here we've set them to zero. The nondegeneracy assumption is not really necessary, so we repeat here the proof from To compute the inner expectation, write F j for the sigma algebra generated by the weight in layers up to and including j. Let us also define the events: where we recall from that x (j) are the post-activations in layer j. Supposing first that e is not in layer d, the expectation becomes We have Thus, the expectation in becomes Note that given F d−2, the pre-activations y. Recall that by assumption, the weight matrix. This replacement leaves the product. On the event S d−1 (which occurs whenever y = 0 with probability 1 since we assumed that the distribution of each weight has a density relative to Lebesgue measure. Hence, symmetrizing over ± W (d), we find that. Similarly, if e is in layer i, then we automatically find that γ 1 (i − 1) = γ 2 (i − 1) and γ 1 (i) = γ 2 (i), giving an expectation of 1/n i−1 1. Proceeding in this way yields which is precisely. The proofs of and are similar. We have As before let us first assume that edges e 1, e 2 are not in layer d. Then, The the inner expectation is 1 each weight appears an even number of times Again symmetrizing with respect to ± W (d) and using that the pre-activation of different neurons are independent given the activations in the previous layer we find that, on the event {y where L is the event that |Γ(d − 1)| = |Γ(d)| = 1 and e 1, e 2 are not in layer d − 1. Proceeding in this way one layer at a time completes the proofs of and. Fix j = 1,..., 4, edges e 1, e 2 with (e 1) ≤ (e 2) in the computational graph of N and E ∈ Σ 4 aj,even. The key idea is to decompose E into loops. To do this, define For each i = 1,..., d there exists unique k = 1,..., #loops(E) so that We will say that two layers i, j = 1,..., d belong to the same loop of E if exists k = 1,..., #loops(E) so that We proceed layer by layer to count the number of Γ ∈ Γ 4 aj,even satisfying Γ = a j and E Γ = E. To do this, suppose we are given Γ(i − 1) ∈ [n i−1] 4 and we have L(E(i)) = 2. Then Γ(i − 1) is some permutation of (α 1, α 1, α 2, α 2) with α 1 = α 2. Moreover, for j = 1, 2 there is a unique edge (with multiplicity 2) in E(i) whose left endpoint is α j. Therefore, Γ(i − 1) determines Γ(i) when L(E(i)) = 2. In contrast, suppose L(E(i)) = 1. If R(E(i)) = 1, then E(i) consists of a single edge with multiplicity 4, which again determines Γ(i − 1), Γ(i). In short, Γ(i) determines Γ(j) for all j belonging to the same loop of E as i. Therefore, the initial condition Γ = a j determines Γ(i) for all i ≤ i 1 and the conditions e 1 ∈ γ 1, e 2 ∈ γ 2 determine Γ in the loops of E containing the layers of e 1, e 2. Finally, suppose L(E(i)) = 1 and R(E(i)) = 2 (i.e. i = i k (E) for some k = 1,..., d) and that e 1, e 2 are not contained in the same loop of E layer i. Then all 4 2 = 6 choices of Γ(i) satisfy Γ(i) = R(E(i)), accounting for the factor of 6 #loops(E). The concludes the proof in the case j = 1. the only difference in the cases j = 2, 3, 4 is that if γ 1 = γ 2 (and hence γ 3 = γ 4), then since (e 1) ≤ (e 2) in order to satisfy e 1 ∈ γ 1, γ 2 we must have that i 1 (E) < (e 1). The proof of Lemma 9 is essentially identical to the proof of Lemma 8. In fact it is slightly simpler since there are no distinguished edges e 1, e 2 to consider. We omit the details. In this section, we seek to estimate The approach is essentially identical to but somewhat simpler than our proof of Proposition 3 in §B. We will therefore focus here on explaining the salient differences. Our starting point is the following analog of Lemma 6, which gives a sum-over-paths expression for the bias contribution K b to the neural tangent kernel. To state it, let us define, for any collection Z = (z 1, . . ., z k) ∈ Z k of k neurons in N 1 {y Z >0}:= k j=1 1 {yz j >0}, to be the event that the pre-activations of the neurons z k are positive. Lemma 12 (K b as a sum over paths). With probability 1, where Z 1, Γ 2 (Z,Z), wt(γ) are defined in §A. Further, almost surely, The proof of this is a small modification of the proof of Lemma 6 and hence is omitted. Taking expectations, we therefore obtain the following analog to Lemma 7. Moreover, E[K The proof is identical to the argument used in §B.2 to establish Lemma 7, so we omit the details. The relation is easy to simplify: where we used that the number paths from a neuron in layer to the output of N equals Proof. The proof of Lemma 14 is a simplified version of the computation of E[K 2 w] (starting around and ending at the end of the proof of Proposition 3). Specifically, note that for Γ = (γ 1, . . ., γ 4) ∈ Γ 4 Z,even with (z 1) ≤ (z 2), the delta functions 1 {γ 1(i)=γ2(i)} 1 {γ 1(i −1)=γ2(i −1)} in the definition of H(Γ) ensures that γ 1, γ 2 go through the same neuron in layer (z 2). To condition on the index of this neuron, we recall that we denote by z(j, β) neuron number β in layer j. We have where Z = (z( (z 2), β), z((z 2), β), z 2, z 2 ) and. Since the inner sum in is independent of β by symmetry, we find where Z = (1, 1, z 2, z 2). The inner sum in is now precisely one of the terms I j from without counting terms involving edges e 1, e 2, except that the paths start at neuron 1 in layer (z 2). The changes of variables from Γ ∈ Γ 4 even to E ∈ Σ 4 even to V ∈ Γ 2 that we used to estimate the I j's are no far simpler. In particular, Lemma 8 still holds but without any of the A(E, i 1, i 2), C(E, i 1, i 2), C(E, i 1, i 2) terms. Thus, we find that where for the second estimate we applied Lemma 9 and have written Z = (1, z 2). Thus, as in the derivation of, we find that

The neural tangent kernel in a randomly initialized ReLU net is non-trivial fluctuations as long as the depth and width are comparable.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the article for finding a single-sentence TLDR

Question: Most algorithms for representation learning and link prediction in relational data have been designed for static data. However, the data they are applied to usually evolves with time, such as friend graphs in social networks or user interactions with items in recommender systems. This is also the case for knowledge bases, which contain facts such as (US, has president, B. Obama,) that are valid only at certain points in time. For the problem of link prediction under temporal constraints, i.e., answering queries of the form (US, has president, ?, 2012), we propose a solution inspired by the canonical decomposition of tensors of order 4. We introduce new regularization schemes and present an extension of ComplEx that achieves state-of-the-art performance. Additionally, we propose a new dataset for knowledge base completion constructed from Wikidata, larger than previous benchmarks by an order of magnitude, as a new reference for evaluating temporal and non-temporal link prediction methods. Link prediction in relational data has been the subject of interest, given the widespread availability of such data and the breadth of its use in bioinformatics , recommender systems or Knowledge Base completion (a). Relational data is often temporal, for example, the action of buying an item or watching a movie is associated to a timestamp. Some medicines might not have the same adverse side effects depending on the subject's age. The task of temporal link prediction is to find missing links in graphs at precise points in time. In this work, we study temporal link prediction through the lens of temporal knowledge base completion, which provides varied benchmarks both in terms of the underlying data they represent, but also in terms of scale. A knowledge base is a set of facts (subject, predicate, object) about the world that are known to be true. Link prediction in a knowledge base amounts to answer incomplete queries of the form (subject, predicate, ?) by providing an accurate ranking of potential objects. In temporal knowledge bases, these facts have some temporal metadata attached. For example, facts might only hold for a certain time interval, in which case they will be annotated as such. Other facts might be event that happened at a certain point in time. Temporal link prediction amounts to answering queries of the form (subject, predicate, ?, timestamp). For example, we expect the ranking of queries (USA, president, ?, timestamp) to vary with the timestamps. As tensor factorization methods have proved successful for Knowledge Base Completion (a; ;), we express our Temporal Knowledge Base Completion problem as an order 4 tensor completion problem. That is, timestamps are discretized and used to index a 4-th mode in the binary tensor holding (subject, predicate, object, timestamps) facts. First, we introduce a ComplEx decomposition of this order 4 tensor, and link it with previous work on temporal Knowledge Base completion. This decomposition yields embeddings for each timestamps. A natural prior is for these timestamps representation to evolve slowly over time. We are able to introduce this prior as a regularizer for which the optimum is a variation on the nuclear p-norm. In order to deal with heterogeneous temporal knowledge bases where a significant amount of relations might be non-temporal, we add a non-temporal component to our decomposition. Experiments on available benchmarks show that our method outperforms the state of the art for similar number of parameters. We run additional experiments for larger, regularized models and obtain improvements of up to 0.07 absolute Mean Reciprocal Rank (MRR). Finally, we propose a dataset of 400k entities, based on Wikidata, with 7M train triples, of which 10% contain temporal validity information. This dataset is larger than usual benchmarks in the Knowledge Base completion community and could help bridge the gap between the method designed and the envisaged web-scale applications. Matrices and tensors are upper case letters. The i-th row of U is denoted by u i while it's j − th column is denoted by U:,j. The tensor product of two vectors is written ⊗ and the hadamard (elementwise) product ⊙. Static link prediction methods Standard tensor decomposition methods have lead to good (; ; ; Balažević et al., 2019) in Knowledge Base completion. The Canonical Polyadic (CP) Decomposition is the tensor equivalent to the low-rank decomposition of a matrix. A tensor X of canonical rank R can be written as: Setting U = W leads to the Distmult model, which has been successful, despite only being able to represent symmetric score functions. In order to keep the parameter sharing scheme but go beyond symmetric relations, use complex parameters and set W to the complex conjugate of U, U. Regularizing this algorithm with the variational form of the tensor nuclear norm as well as a slight transformation to the learning objective (also proposed in) leads to state of the art in. Other methods are not directly inspired from classical tensor decompositions. For example, TransE models the score as a distance of the translated subject to an object representation. This method has lead to many variations (; ;), but is limited in the relation systems it can model and does not lead to state of the art performances on current benchmarks. propose to generate the entity embeddings of a CP-like tensor decomposition by running a forward pass of a Graph Neural Network over the training Knowledge Base. The experiments included in this work did not lead to better link prediction performances than the same decomposition (Distmult) directly optimized . describes a bayesian model and learning method for representing temporal relations. The temporal smoothness prior used in this work is similar to the gradient penalty we describe in Section 3.3. However, learning one embedding matrix per timestamp is not applicable to the scales considered in this work. uses a tensor decomposition called ASALSAN to express temporal relations. This decomposition is related to RESCAL which underperforms on recent benchmarks due to overfitting (b). For temporal knowledge base completion, learns entity embeddings that change over time, by masking a fraction of the embedding weights with an activation function of learned frequencies. Based on the Tucker decomposition, ConT learns one new core tensor for each timestamp. Finally, viewing the time dimension as a sequence to be predicted, García-Durán et al. use recurrent neural nets to transform the embeddings of standard models such as TransE or Distmult to accomodate the temporal data. Published as a conference paper at ICLR 2020 by studying and extending a regularized CP decomposition of the training set seen as an order 4 tensor. We propose and study several regularizer suited to our decompositions. In this section, we are given facts (subject, predicate, object) annotated with timestamps, we discretize the timestamp range (eg. by reducing timestamps to years) in order to obtain a training set of 4-tuple (subject, predicate, object, time) indexing an order 4 tensor. We will show in Section 5.1 how we reduce each datasets to this setting. , we minimize, for each of the train tuples (i, j, k, l), the instantaneous multiclass loss: Note that this loss is only suited to queries of the type (subject, predicate, ?, time), which is the queries that were considered in related work. We consider another auxiliary loss in Section 6 which we will use on our Wikidata dataset. For a training set S (augmented with reciprocal relations ), and parametric tensor estimateX(θ), we minimize the following objective, with a weighted regularizer Ω: ]. The ComplEx decomposition can naturally be extended to this setting by adding a new factor T, we then have: We call this decomposition TComplEx. Intuitively, we added timestamps embedding that modulate the multi-linear dot product. Notice that the timestamp can be used to equivalently modulate the objects, predicates or subjects to obtain time-dependent representation: Contrary to DE-SimplE , we do not learn temporal embeddings that scale with the number of entities (as frequencies and biases), but rather embeddings that scale with the number of timestamps. The number of parameters for the two models are compared in Table 1. Some predicates might not be affected by timestamps. For example, Malia and Sasha will always be the daughters of Barack and Michelle Obama, whereas the "has occupation" predicate between two entities might very well change over time. In heterogeneous knowledge bases, where some predicates might be temporal and some might not be, we propose to decompose the tensorX as the sum of two tensors, one temporal, and the other non-temporal: Published as a conference paper at ICLR 2020 We call this decomposition TNTComplEx. suggests another way of introducing a non-temporal component, by only allowing a fraction γ of components of the embeddings to be modulated in time. By allowing this sharing of parameters between the temporal and non-temporal part of the tensor, our model removes one hyperparameter. Moreover, preliminary experiments showed that this model outperforms one without parameter sharing. Any order 4 tensor can be considered as an order 3 tensor by unfolding modes together. For a tensor X ∈ R N1×N2×N3×N4, unfolding modes 3 and 4 together will lead to tensorX ∈ R N1×N2×N3N4 . We can see both decompositions ( and) as order 3 tensors by unfolding the temporal and predicate modes together. Considering the decomposition implied by these unfoldings (see Appendix 8.1) leads us to the following weighted regularizers : The first regularizer weights objects, predicates and pairs (predicate, timestamp) according to their respective marginal probabilities. This regularizer is a variational form of the weighted nuclear 3-norm on an order 4 tensor (see subsection 3.4 and Appendix 8.3 for details and proof). The second regularizer is the sum of the nuclear 3 penalties on tensors We have more a priori structure on the temporal mode than on others. Notably, we expect smoothness of the application i → t i. In words, we expect neighboring timestamps to have close representations. Thus, we penalize the norm of the discrete derivative of the temporal embeddings: We show in Appendix 8.2 that the sum of Λ p and the variational form of the nuclear p norm lead to a variational form of a new tensor atomic norm. As was done in , we aim to use tensor nuclear p-norms as regularizers. The definition of the nuclear p-norm of a tensor of order D is: This formulation of the nuclear p-norm writes a tensor as a sum over atoms which are the rank-1 tensors of unit p-norm factors. The nuclear p-norm is NP-hard to compute . , a practical solution is to use the equivalent formulation of nuclear p-norm using their variational form, which can be conveniently written for p = D: For the equality above to hold, the infimum should be over all possible R. The practical solution is to fix R to the desired rank of the decomposition. Using this variational formulation as a regularizer leads to state of the art for order-3 tensors and is convenient in a stochastic gradient setting because it separates over each model coefficient. In addition, this formulation makes it easy to introduce a weighting as recommended in;. In order to learn under non-uniform sampling distributions, one should penalize the weighted norm: In subsection 3.3, we add another penalty in Equation which changes the norm of our atoms. In subsection 3.2, we introduced another variational form in Equation which allows to easily penalize the nuclear 3-norm of an order 4 tensor. This regularizer leads to different weighting. By considering the unfolding of the timestamp and predicate modes, we are able to weight according to the joint marginal of timestamps and predicates, rather than by the product of the marginals. This can be an important distinction if the two are not independent. We study the impact of regularization on the ICEWS05-15 dataset, for the TNTComplEx model. For details on the experimental set-up, see Section 5.1. The first effect we want to quantify is the effect of the regularizer Λ p. We run a grid search for the strength of both Λ p and Ω 3 and plot the convex hull as a function of the temporal regularization strength. As shown in Figure 1, imposing smoothness along the time mode brings an improvement of over 2 MRR point. The second effect we wish to quantify is the effect of the choice of regularizer Ω. A natural regularizer for TNTComplEx would be: We compare ∆ 4, ∆ 3 and ∆ 2 with Ω 3. The comparison is done with a temporal regularizer of 0 to reduce the experimental space. 2 is the common weight-decay frequently used in deep-learning. Such regularizers have been used in knowledge base completion (; 2016b;), however, showed that the infimum of this penalty is non-convex over tensors. 3 matches the order used in the Ω 3 regularizer, and in previous work on knowledge base completion . However, by the same arguments, its minimization does not lead to a convex penalty over tensors. There are two differences between ∆ 4 and Ω 3. First, whereas the first is a variational form of the nuclear 4-norm, the second is a variational form of the nuclear 3-norm which is closer to the nuclear 2-norm. Results for exact recovery of tensors have been generalized to the nuclear 2-norm, and to the extent of our knowledge, there has been no formal study of generalization properties or exact recovery under the nuclear p-norm for p greater than two. Second, the weighting in ∆ 4 is done separately over timestamps and predicates, whereas it is done jointly for Ω 3. This leads to using the joint empirical marginal as a weighting over timestamps and predicates. The impact of weighting on the guarantees that can be obtained are described more precisely in. The contribution of all these regularizers over a non-regularized model are summarized in Table 3. Note that careful regularization leads to a 0.05 MRR increase. A dataset based on Wikidata was proposed by García-Durán et al.. However, upon inspection, this dataset contains numerical data as entities, such as ELO rankings of chess players, which are not representative of practically useful link prediction problems. Also, in this dataset, temporal informations is specified in the form of "OccursSince" and "OccursUntil" statements appended to triples, which becomes unwieldy when a predicate holds for several intervals in time. Moreover, this dataset contains only 11k entities and 150k which is insufficient to benchmark methods at scale.. In order to adress these limitations, we created our own dataset from Wikidata, which we make available at dataseturl. Starting from Wikidata, we removed all entities that were instance of scholarly articles, proteins and others. We also removed disambiguation, template, category and project pages from wikipedia. Then, we removed all facts for which the object was not an entity. We iteratively filtered out entities that had degree at least 5 and predicates that had at least 50 occurrences. With this method, we obtained a dataset of 432715 entities, 407 predicates and 1724 timestamps (we only kept the years). Each datum is a triple (subject, predicate, object) together a timestamp range (begin, end) where begin, end or both can be unspecified. Our train set contains 7M such tuples, with about 10% partially specified temporal tuples. We kept a validation and test set of size 50k each. At train and test time, for a given datum (subject, predicate, object, [begin, end] ), we sample a timestamp (appearing in the dataset) uniformly at random, in the range [begin, end]. For datum without a temporal range, we sample over the maximum date range. Then, we rank the objects for the partial query (subject, predicate, ?, timestamp). We follow the experimental set-up in García-Durán et al.;. We use models from García-Durán et al. and as baselines since they are the best performing algorithms on the datasets considered. We report the filtered Mean Reciprocal Rank (MRR) defined in Nickel et al. (2016b). In order to obtaiqn comparable , we use Table 1 and dataset statistics to compute the rank for each (model, dataset) pair that matches the number of parameters used in. We also report at ranks 10 times higher. This higher rank set-up gives an estimation of the best possible performance attainable on these datasets, even though the dimension used might be impractical for applied systems. All our models are optimized with Adagrad , with a learning rate of 0.1, a batch-size of 1000. More details on the grid-search, actual ranks used and hyper-parameters are given in Appendix 8. (García-Durán et al., 2018) and DE-SimplE are the best numbers reported in the respective papers. Our models have as many parameters as DE-SimplE. Numbers in parentheses are for ranks multiplied by 10. We give on 3 datasets previously used in the litterature: ICEWS14, ICEWS15-05 and Yago15k. The ICEWS datasets are samplings from the Integrated Conflict Early Warning System (ICEWS) 1.García-Durán et al. introduced two subsampling of this data, ICEWS14 which contains all events occuring in 2014 and ICEWS05-15 which contains events occuring between 2005 and 2015. These datasets immediately fit in our framework, since the timestamps are already discretized. The Yago15K dataset (García-Durán et al., 2018) is a modification of FB15k which adds "occursSince" and "occursUntil" timestamps to each triples. Following the evaluation setting of García-Durán et al., during evaluation, the incomplete triples to complete are of the form (subject, predicate, ?, occursSince | occursUntil, timestamp) (with reciprocal predicates). Rather than deal with tensors of order 5, we choose to unfold the (occursSince, occursUntil) and the predicate mode together, multiplying its size by two. Some relations in Wikidata are highly unbalanced (eg. (?, InstanceOf, Human)). For such relations, a ranking evaluation would not make much sense. Instead, we only compute the Mean Reciprocal Rank for missing right hand sides, since the data is such that highly unbalanced relations happen on the left-hand side. However, we follow the same training scheme as for all the other dataset, including reciprocal relations in the training set. The cross-entropy loss evaluated on 400k entities puts a restriction on the dimensionality of embeddings at about d = 100 for a batch-size of 1000. We leave sampling of this loss, which would allow for higher dimensions to future work. We compare ComplEx with the temporal versions described in this paper. We report in Table 2. Note that ComplEx has performances that are stable through a tenfold increase of its number of parameters, a rank of 100 is enough to capture the static information of these datasets. For temporal models however, the performance increases a lot with the number of parameters. It is always beneficial to allow a separate modeling of non-temporal predicates, as the performances of TNTComplex show. Finally, our model match or beat the state of the art on all datasets, even at identical number of parameters. Since these datasets are small, we also report for higher ranks (10 times the number of parameters used for DE-SimplE). outweighs the Temporal MRR (T-MRR). A breakdown of the performances is available in table 4. TNTComplEx obtains performances that are comparable to ComplEx on non-temporal triples, but are better on temporal triples. Moreover, TNTComplEx can minimize the temporal cross-entropy and is thus more flexible on the queries it can answer. Training TNTComplEx on Wikidata with a rank of d = 100 with the full cross-entropy on a Quadro GP 100, we obtain a speed of 5.6k triples per second, leading to experiments time of 7.2 hours. This is to be compared with 5.8k triples per second when training ComplEx for experiments time of 6.9 hours. The additional complexity of our model does not lead to any real impact on runtime, which is dominated by the computation of the cross-entropy over 400k entities. The instantaneous loss described in equation, along with the timestamp sampling scheme described in the previous section only enforces correct rankings along the "object" tubes of our order-4 tensor. In order to enforce a stronger temporal consistency, and be able to answer queries of the type (subject, predicate, object, ?), we propose another cross-entropy loss along the temporal tubes: We optimize the sum of ℓ defined in Equation 1 andl defined in Equation 7. Doing so, we only lose 1 MRR point overall. However, we make our model better at answering queries along the time axis. The macro area under the precision recall curve is 0.92 for a TNTComplEx model learned with ℓ alone and 0.98 for a TNTComplEx model trained with ℓ +l. We plot in Figure 2 the scores along time for train triples (president of the french republic, office holder, {Jacques Chirac | Nicolas Sarkozy | François Hollande | Emmanuel Macron}, ). The periods where a score is highest matches closely the ground truth of start and end dates of these presidents mandates which is represented as a colored . This shows that our models are able to learn rankings that are correct along time intervals despite our training method only ever sampling timestamps within these intervals. Tensor methods have been successful for Knowledge Base completion. In this work, we suggest an extension of these methods to Temporal Knowledge Bases. Our methodology adapts well to the various form of these datasets: point-in-time, beginning and endings or intervals. We show that our methods reach higher performances than the state of the art for similar number of parameters. For several datasets, we also provide performances for higher dimensions. We hope that the gap between low-dimensional and high-dimensional models can motivate further research in models that have increased expressivity at lower number of parameters per entity. Finally, we propose a large scale temporal dataset which we believe represents the challenges of large scale temporal completion in knowledge bases. We give performances of our methods for low-ranks on this dataset. We believe that, given its scale, this dataset could also be an interesting addition to non-temporal knowledge base completion. Then according to , unfolding along modes 3 and 4 leads to an order three tensor of decompositionX Where • is the Khatri-Rao product , which is the column-wise Kronecker product: Note that for a fourth mode of size L: This justifies the regularizers used in Section 3.2. Consider the penalty: ∥ · ∥ τ 4 is a norm and lets us rewrite: ). Following the proof in which only uses homogeneity of the norms, we can show that Ω(U, V, W, T) is a variational form of an atomic norm with atoms: We consider the regularizer: ). Let D subj (resp. obj, pred/time) the diagonal matrix containing the cubic-roots of the marginal probabilities of each subject (resp. obj, pred/time) in the dataset. We denote by • the Kathri-Rao product between two matrices (the columnwise Kronecker product). Summing over the entire dataset, we obtain the penalty: ). Dropping the weightings to simplify notations, we state the equivalence between this regularizer and a variational form of the nuclear 3-norm of an order 4 tensor: The proof follows , noting that ∥u Statistics of all the datasets used in this work are gathered in Table 6: Results for TA (García-Durán et al., 2018) and DE-SimplE are the best numbers reported in the respective papers.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose new tensor decompositions and associated regularizers to obtain state of the art performances on temporal knowledge base completion.

Represent the short summary for finding the corresponding scientific article (long)

We propose nearest neighbor overlap, a procedure which quantifies similarity between embedders in a task-agnostic manner, and use it to compare 21 sentence embedders.