Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Given samples from a group of related regression tasks, a data-enriched model describes observations by a common and per-group individual parameters. In high-dimensional regime, each parameter has its own structure such as sparsity or group sparsity. In this paper, we consider the general form of data enrichment where data comes in a fixed but arbitrary number of tasks $G$ and any convex function, e.g., norm, can characterize the structure of both common and individual parameters. 	 We propose an estimator for the high-dimensional data enriched model and investigate its statistical properties. We delineate the sample complexity of our estimator and provide high probability non-asymptotic bound for estimation error of all parameters under a condition weaker than the state-of-the-art. We propose an iterative estimation algorithm with a geometric convergence rate. Overall, we present a first through statistical and computational analysis of inference in the data enriched model. 	 Over the past two decades, major advances have been made in estimating structured parameters, e.g., sparse, low-rank, etc., in high-dimensional small sample problems BID13 BID6 BID14. Such estimators consider a suitable (semi) parametric model of the response: y = φ(x, β *)+ω based on n samples {(x i, y i)} n i=1and β * ∈ R p is the true parameter of interest. The unique aspect of such high-dimensional setup is that the number of samples n < p, and the structure in β *, e.g., sparsity, low-rank, makes the estimation possible (; BID7 BID5). In several real world problems, natural grouping among samples arises and learning a single common model β 0 for all samples or many per group individual models β g s are unrealistic. The middle ground model for such a scenario is the superposition of common and individual parameters β 0 + β g which has been of recent interest in the statistical machine learning community BID16 and is known by multiple names. It is a form of multi-task learning (; BID17 when we consider regression in each group as a task. It is also called data sharing BID15 since information contained in different group is shared through the common parameter β 0 . And finally, it has been called data enrichment BID10 BID0 because we enrich our data set with pooling multiple samples from different but related sources. In this paper, we consider the following data enrichment (DE) model where there is a common parameter β * 0 shared between all groups plus individual per-group parameters β * g which characterize the deviation of group g: y gi = φ(x gi, (β * 0 + β * g)) + ω gi, g ∈ {1, . . ., G}, where g and i index the group and samples respectively. Note that the DE model is a system of coupled superposition models. We specifically focus on the high-dimensional small sample regime for where the number of samples n g for each group is much smaller than the ambient dimensionality, i.e., ∀g: n g p. Similar to all other highdimensional models, we assume that the parameters β g are structured, i.e., for suitable convex functions f g's, f g (β g) is small. Further, for the technical analysis and proofs, we focus on the case of linear models, i.e., φ(x, β) = x T β. The seamlessly extend to more general non-linear models, e.g., generalized linear models, broad families of semi-parametric and single-index models, non-convex models, etc., using existing , i.e., how models like LASSO have been extended (e.g. employing ideas such as restricted strong convexity ).In the context of Multi-task learning (MTL), similar models have been proposed which has the general form of y gi = x T gi (β * 1g + β * 2g) + ω gi where B 1 = [β 11, . . ., β 1G] and B 2 = [β 21, . . ., β 2G] are two parameter matrices . To capture relation of tasks, different types of constraints are assumed for parameter matrices. For example, BID11 assumes B 1 and B 2 are sparse and low rank respectively. In this parameter matrix decomposition framework for MLT, the most related work to ours is the one proposed by BID17 where authors regularize the regression with B 1 1,∞ and B 2 1,1 where norms are p, q-norms on rows of matrices. Parameters of B 1 are more general than DE's common parameter when we use f 0 (β 0) = β 0 1. This is because B 1 1,∞ regularizer enforces shared support of β * 1g s, i.e., supp(β * 1i) = supp(β * 1j) but allows β * 1i = β * 1j. Further sparse variation between parameters of different tasks is induced by B 2 1,1 which has an equivalent effect to DE's individual parameters where f g (·)s are l 1 -norm. Our analysis of DE framework suggests that it is more data efficient than this setup of BID17 ) because they require every task i to have large enough samples to learn its own common parameters β i while DE shares the common parameter and only requires the total dataset over all tasks to be sufficiently large. The DE model where β g's are sparse has recently gained attention because of its application in wide range of domains such as personalized medicine BID12, sentiment analysis, banking strategy BID15, single cell data analysis , road safety , and disease subtype analysis BID12. In spite of the recent surge in applying data enrichment framework to different domains, limited advances have been made in understanding the statistical and computational properties of suitable estimators for the data enriched model. In fact, non-asymptotic statistical properties, including sample complexity and statistical rates of convergence, of regularized estimators for the data enriched model is still an open question BID15 ). To the best of our knowledge, the only theoretical guarantee for data enrichment is provided in where authors prove sparsistency of their proposed method under the stringent irrepresentability condition of the design matrix for recovering supports of common and individual parameters. Existing support recovery guarantees , sample complexity and l 2 consistency BID17 of related models are restricted to sparsity and l 1 -norm, while our estimator and norm consistency analysis work for any structure induced by arbitrary convex functions f g. Moreover, no computational , such as rates of convergence of the optimization algorithms associated with proposed estimators, exist in the literature. We denote sets by curly V, matrices by bold capital V, random variables by capital V, and vectors by small bold v letters. We take DISPLAYFORM0 Given G groups and n g samples in each as DISPLAYFORM1, we can form the per group design matrix X g ∈ R ng×p and output vector y g ∈ R ng.The total number of samples is n = G g=1 n g. The data enriched model takes the following vector form: DISPLAYFORM2 where each row of X g is x T gi and ω T g = (ω g1, . . ., ω gng) is the noise vector. A random variable V is sub-Gaussian if its moments satisfies ∀p ≥ 1: DISPLAYFORM3 is sub-Gaussian if the one-dimensional marginals v, u are sub-Gaussian random variables for all u ∈ R p. The sub-Gaussian norm of v is defined as, 2018), where the expectation is over g ∼ N (0, I p×p), a vector of independent zeromean unit-variance Gaussian. DISPLAYFORM4 We propose the following Data Enrichment (DE) estimatorβ for recovering the structured parameters where the structure is induced by convex functions f g (·): DISPLAYFORM0 We present several statistical and computational for the DE estimator of the data enriched model:• The DE estimator succeeds if a geometric condition that we call Data EnRichment Incoherence Condition (DERIC) is satisfied, FIG1. Compared to other known geometric conditions in the literature such as structural coherence BID16 and stable recovery conditions BID18, DERIC is a weaker condition, FIG1.• Assuming DERIC holds, we establish a high probability non-asymptotic bound on the weighted sum of parameterwise estimation error, δ g =β g − β * g as: DISPLAYFORM1 where n 0 n is the total number of samples, γ max g∈ [G] n ng is the sample condition number, and C g is the error cone corresponding to β * g exactly defined in Section 2. To the best of our knowledge, this is the first statistical estimation guarantee for the data enrichment. • We also establish the sample complexity of the DE estimator for all parameters as ∀g ∈ [G]: DISPLAYFORM2 We emphasize that our proofs that the recovery of the common parameter β 0 by DE estimator benefits from all of the n pooled samples.• We present an efficient projected block gradient descent algorithm DICER, to solve DE's objective which converges geometrically to the statistical error bound of. To the best of our knowledge, this is the first rigorous computational for the high-dimensional data-enriched regression. A compact form of our proposed DE estimator is: DISPLAYFORM0 where y = (y DISPLAYFORM1 Example 1. (L 1 -norm) When all parameters β g s are s gsparse, i.e.,|supp(β * g)| = s g by using l 1 -norm as the sparsity inducing function, DE instantiates to the spare DE: DISPLAYFORM2 Consider the group-wise estimation error δ g =β g − β * g. Sinceβ g = β * g + δ g is a feasible point of, the error vector δ g will belong to the following restricted error set: DISPLAYFORM3 We denote the cone of the error set as C g Cone(E g) and the spherical cap corresponding to it as DISPLAYFORM4 following two subsets of C play key roles in our analysis: DISPLAYFORM5 DISPLAYFORM6 Using optimality ofβ, we can establish the following deterministic error bound. DISPLAYFORM7 The main assumptions of Theorem 1 is known as Restricted Eigenvalue (RE) condition in the literature of high dimensional statistics; ): DISPLAYFORM0 Here, we show that for the design matrix X defined in, the RE condition holds with high probability under a suitable geometric condition we call Data EnRichment Incoherence Condition (DERIC) and for enough number of samples. For the analysis, similar to existing work (; BID19 BID16, we assume the design matrix to be isotropic sub-Gaussian. 1 Definition 1. We assume x gi are i.i.d. random vectors from a non-degenerate zero-mean, isotropic sub-Gaussian distribution. In other words, DISPLAYFORM1 Further, we assume noise ω gi are i.i.d. zero-mean, unit-variance sub-Gaussian with |||ω gi ||| ψ2 ≤ K.). There exists a non-empty set I ⊆ [G] \ of groups where for some scalars 0 <ρ ≤ 1 and λ min > 0 the following holds: i∈I n i ≥ ρn. 2. ∀i ∈ I, ∀δ i ∈ C i, and δ 0 ∈ C 0: DISPLAYFORM0 Using DERIC and the small ball method BID19, a recent tool from empirical process theory in the following theorem, we elaborate the sample complexity required for satisfying the RE condition: Theorem 2. Let x gi s be random vectors defined in Definition 1. Assume DERIC condition of Definition 2 holds for error cones C g s and ψ I = λ minρ /3. Then, for all δ ∈ H, when we have enough number of samples as ∀g DISPLAYFORM1, with probability at least 1 − e −nκmin/4 we have inf δ∈H DISPLAYFORM2 4 is the lower bound of the RE condition. Example 2. (L 1 -norm) The Gaussian width of the spherical cap of a p-dimensional s-sparse vector is ω(A) = Θ(√ s log p) ). Therefore, the number of samples per group and total required for satisfaction of the RE condition in the sparse DE estimator FORMULA10 is ∀g ∈ [G]: n g ≥ m g = Θ(s g log p). Here, we provide a high probability upper bound for the deterministic upper bound of Theorem 1 and derive the final estimation error bound. Theorem 3. Assume x gi and ω gi distributed according to Definition 1 and τ > 0, then with probability at least 1 DISPLAYFORM0 we have: DISPLAYFORM1 The following corollary characterizes the general error bound and from the direct combination of Theorem 1, Theorem 2, and Theorem 3. Corollary 1. For x gi and ω gi described in Definition 1 and τ > 0 when we have enough number of samples ∀g ∈ [G]: n g > m g which lead to κ > 0, the following general error bound holds with high probability for estimator: DISPLAYFORM2 Example 3. (L 1 -norm) For the sparse DE estimator of FORMULA10, of Theorem 2 and 3 translates to the following: For enough number of samples as ∀g DISPLAYFORM3, the error bound of simplifies to: DISPLAYFORM4 Therefore, individual errors are bounded as δ g 2 = O((max g∈ [G] s g ) log p/n g ) which is slightly worse than DISPLAYFORM5 for g=1 to G do 5: DISPLAYFORM6 end for 7: DISPLAYFORM7 O(s g log p/n g), the well-known error bound for recovering an s g -sparse vector from n g observations using LASSO or similar estimators BID8 BID4 BID9 BID2. Note that max g∈ [G] s g (instead of s g) is the price we pay to recover the common parameter β 0. We propose Data enrIChER (DICER) a projected block gradient descent algorithm, Algorithm 1, where Π Ω fg is the Euclidean projection onto the set DISPLAYFORM0 To analysis convergence properties of DICER, we should upper bound the error of each iteration. Let's δ (t) = β (t) − β * be the error of iteration t of DICER, i.e., the distance from the true parameter (not the optimization minimum,β). We show that δ (t) 2 decreases exponentially fast in t to the statistical error δ 2 = β − β * 2. DISPLAYFORM1 2, updates of the Algorithm 1 obey the following with high probability: DISPLAYFORM2 where r(τ) < 1. Corollary 2. For enough number of samples, iterations of DE algorithm with step sizes µ 0 = Θ(1 n) and µ g = Θ(1 √ nng) geometrically converges to the following with high probability: DISPLAYFORM3 which is a scaled variant of statistical error bound determined in Corollary 1. In this Section we present detail proof for each theorem and proposition. To avoid cluttering, during our proofs, we state some needed as lemmas and provide their proof in the next Section B. Proof. Starting from the optimality inequality, for the lower bound with the set H we get: DISPLAYFORM0 2 is known as Restricted Eigenvalue (RE) condition. The upper bound will factorize as: DISPLAYFORM1 Putting together inequalities FORMULA6 and FORMULA8 completes the proof. A.2. Proof of Proposition 1 Proposition 1. Assume observations distributed as defined in Definition 1 and pair-wise SC conditions are satisfied. Consider each superposition model in isolation; to recover the common parameter β * 0 requires at least one group i to have n i = O(ω 2 (A 0)). To recover the rest of individual parameters, we need ∀g = i: DISPLAYFORM2 Proof. Consider only one group for regression in isolation. Note that y g = X g (β * g + β * 0) + ω g is a superposition model and as shown in BID16 ) the sample complexity required for the RE condition and subsequently recovering DISPLAYFORM3 Let's simplify the LHS of the RE condition: DISPLAYFORM0 where the first inequality is due to Lyapunov's inequality. To avoid cluttering we denote δ 0g = δ 0 + δ g where δ 0 ∈ C 0 and δ g ∈ C g. Now we add and subtract the corresponding per-group marginal tail function, DISPLAYFORM1 where ξ g > 0. Let ξ g = δ 0g 2 ξ then the LHS of the RE condition reduces to: DISPLAYFORM2 For the ease of exposition we have written the LHS of as the difference of two terms, i.e., t 1 (X) − t 2 (X) and in the followings we lower bound the first term t 1 and upper bound the second term t 2.A.3.1. LOWER BOUNDING THE FIRST TERM Our main is the following lemma which uses the DERIC condition of the Definition 2 and provides a lower bound for the first term t 1 (X):Lemma 1. Suppose DERIC holds. Let ψ I = λminρ 3. For any δ ∈ H, we have: DISPLAYFORM3 which implies that t 1 (X) = inf δ∈H G g=1 n G n ξ g Q 2ξg (δ 0g) satisfies the same RHS bound of. Let's focus on the second term, i.e., t 2 (X). First we want to show that the second term satisfies the bounded difference property defined in Section 3.2. of BID3. In other words, by changing each of x gi the value of t 2 (X) at most change by one. First, we rewrite t 2 as follows: DISPLAYFORM0 where g (x 11, . . ., x jk, . . ., DISPLAYFORM1 To avoid cluttering let's X = {x 11, . . ., x jk, . . ., x Gn G}. We want to show that t 2 has the bounded difference property, meaning: DISPLAYFORM2 for some constant c i. Note that for bounded functions f, g: X → R, we have | sup X f − sup X g| ≤ sup X |f − g|. Therefore: DISPLAYFORM3 Note that for δ ∈ H we have δ 0 2 + ng n δ g 2 ≤ 1 which in δ 0 2 ≤ 1 and δ g 2 ≤ n ng. Now, we can invoke the bounded difference inequality from Theorem 6.2 of BID3 which says that with probability at least 1 − e −τ 2 /2 we have: DISPLAYFORM4 Having this concentration bound, it is enough to bound the expectation of the second term. Following lemma provides us with the bound on the expectation. Lemma 2. For the random vector x of Definition 1, we have the following bound: DISPLAYFORM5 Set n 0 = n. Putting back bounds of t 1 (X) and t 2 (X) together from Lemma 1 and 2, with probability at least 1 − e − τ 2 2 we have: DISPLAYFORM0 Note that all κ g s should be bounded away from zero. To this end we need the follow sample complexities: DISPLAYFORM1 Taking ξ = α 6 we can simplify the sample complexities to the followings: DISPLAYFORM2 Finally, to conclude, we take τ = √ nκ min /2. Proof. From now on, to avoid cluttering the notation assume ω = ω 0. We massage the equation as follows: DISPLAYFORM0, δg δg 2 n ng ω g 2 and a g = ng n δ g 2. Then the above term is the inner product of two vectors a = (a 0, . . ., a G) and b = (b 0, . . ., b G) for which we have: DISPLAYFORM1 Now we can go back to the original form: DISPLAYFORM2, u g and e g (τ) = DISPLAYFORM3 Then from, we have: DISPLAYFORM4 To simplify the notation, we drop arguments of h g for now. From the union bound we have: DISPLAYFORM5 where σ = max g∈ [G] σ g and the last inequality is a of the following lemma:Lemma 3. For x gi and ω gi defined in Definition 1 and τ > 0, with probability at least 1 − DISPLAYFORM6 we have: DISPLAYFORM7 where σ g, η g, ζ g and g are group dependent constants. Proof. To analysis convergence properties of DICER, we should upper bound the error of each iteration. Let's δ (t) = β (t) −β * be the error of iteration t of DICER, i.e., the distance from the true parameter (not the optimization minimum,β). We show that δ (t) 2 decreases exponentially fast in t to the statistical error δ 2 = β − β * 2. We first start with the required definitions for our analysis. Definition 3. We define the following positive constants as functions of step sizes µ g > 0: DISPLAYFORM0 where DISPLAYFORM1 is the intersection of the error cone and the unit ball. In the following theorem, we establish a deterministic bound on iteration errors δ (t) g 2 which depends on constants defined in Definition 3.Theorem 5. For Algorithm 1 initialized by β = 0, we have the following deterministic bound for the error at iteration t + 1: DISPLAYFORM2 DISPLAYFORM3 The RHS of FORMULA2 consists of two terms. If we keep ρ < 1, the first term approaches zero fast, and the second term determines the bound. In the following, we show that for specific choices of step sizes µ g s, the second term can be upper bounded using the analysis of Section 4. More specifically, the first term corresponds to the optimization error which shrinks in every iteration while the second term is constant times the upper bound of the statistical error characterized in Corollary 1. Therefore, if we keep ρ below one, the estimation error of DE algorithm geometrically converges to the approximate statistical error bound. One way for having ρ < 1 is to keep all arguments of max(· · ·) defining ρ strictly below 1. To this end, we first establish high probability upper bound for ρ g, η g, and φ g (in the Appendix A.6) and then show that with enough number of samples and proper step sizes µ g, ρ can be kept strictly below one with high probability. In the following lemma we establish a recursive relation between errors of consecutive iterations which leads to a bound for the tth iteration. Lemma 4. We have the following recursive dependency between the error of t + 1th iteration and tth iteration of DE: DISPLAYFORM4 By recursively applying the of Lemma 4, we get the following deterministic bound which depends on constants defined in Definition 3: DISPLAYFORM5 where DISPLAYFORM6 µg φ g. We have: DISPLAYFORM7 A.6. Proof of Theorem 4Proof. First we need following two lemmas which are proved separately in the following sections. Lemma 5. Consider a g ≥ 1, with probability at least 1 − 6 exp −γ g (ω(A g) + τ ) 2 the following upper bound holds: DISPLAYFORM8 Lemma 6. Consider a g ≥ 1, with probability at least 1 − 4 exp −γ g (ω(A g) + τ ) 2 the following upper bound holds: DISPLAYFORM9 Note that Lemma 3 readily provides a high probability upper bound for η g (1/(a g n g)) as DISPLAYFORM10 where DISPLAYFORM11 Remember the following two to upper bound ρ g s and φ g s from Lemmas 5 and 6: DISPLAYFORM12 First we want to keep ρ 0 + G g=1 ng n φ g of FORMULA2 strictly below 1. DISPLAYFORM13 Remember that a g ≥ 1 was arbitrary. So we pick it as a g = 2 n ng 1 + c 0g DISPLAYFORM14 (because we need a g ≥ 1) and the condition becomes: DISPLAYFORM15 We want to upper bound the RHS by 1/θ f which will determine the sample complexity for the shared component: DISPLAYFORM16 Note that any lower bound on the RHS of will lead to the correct sample complexity for which the coefficient of δ DISPLAYFORM17 (determined in) will be below one. Since a 0 ≥ 1 we can ignore the first term by assuming max g∈[G] \ b g ≤ 1 and the condition becomes: DISPLAYFORM18 which can be simplified to: DISPLAYFORM19 DISPLAYFORM20 Secondly, we want to bound all of ρ g + µ 0 n ng φg µg terms of for µ g = 1 agng by 1: DISPLAYFORM21 The condition becomes: DISPLAYFORM22 Remember that we chose a g = 2b DISPLAYFORM23. We substitute the value of a g by keeping in mind the constraints for the b g and the condition reduces to: DISPLAYFORM24 Note that any positive lower bound of the d g will satisfy the condition in and the is a valid sample complexity. In the following we show that d g > 1.We have a 0 ≥ 1 condition from, so we take DISPLAYFORM25 and look for a lower bound for d g: DISPLAYFORM26 (a g from) = 2b DISPLAYFORM27 The term inside of the last bracket is always positive and therefore a lower bound is one, i.e., d g ≥ 1. From the condition we get the following sample complexity: DISPLAYFORM28 Now we need to determine b g from previous conditions, knowing that a 0 = 4 max g∈ DISPLAYFORM29. We have 0 < b g ≤ 1 in and we take the largest step by setting b g = 1.Here we summarize the setting under which we have the linear convergence: DISPLAYFORM30 Now we rewrite the same analysis using the tail bounds for the coefficients to clarify the probabilities. To simplify the notation, let r g1 = and r g (τ) = r g1 + ng n ag a0 r g2, ∀g ∈ [G] \, and r(τ) = max g∈[G] r g. All of which are computed using a g s specified in. Basically r is an instantiation of an upper bound of the ρ defined in using a g s in.We are interested to upper bound the following probability: DISPLAYFORM31 where the first inequality comes from the deterministic bound of, We first focus on bounding the first term P (ρ ≥ r(τ)): DISPLAYFORM32 Now we focus on bounding the second term: DISPLAYFORM33, g ∈ [G] and a g ≥ 1: DISPLAYFORM34 where we used the intermediate form of Lemma 3 for τ > 0. Putting all of the bounds,, and back into the: DISPLAYFORM35 where υ = max(28, σ) and γ = min g∈ [G] γ g and τ = t + max(, γ −1/2) log(G + 1) where = k max g∈[G] η g. Note that τ = t + C log(G + 1) increases the sample complexities to the followings: DISPLAYFORM36 and it also affects step sizes as follows: DISPLAYFORM37 Here, we present proofs of each lemma used during the proofs of theorems in Section A. Proof. LHS of FORMULA10 is the weighted summation of ξ g Q 2ξg (δ 0g) = δ 0g 2 ξP(| x,, δ 0g / δ 0g 2 | > 2ξ) = δ 0g 2 ξQ 2ξ (u) where ξ > 0 and u = δ 0g / δ 0g 2 is a unit length vector. So we can rewrite the LHS of as: DISPLAYFORM0 With this observation, the lower bound of the Lemma 1 is a direct consequence of the following two :Lemma 7. Let u be any unit length vector and suppose x obeys Definiton 1. Then for any u, we have DISPLAYFORM1 Lemma 8. Suppose Definition 2 holds. Then, we have: DISPLAYFORM2 Proof. Consider the following soft indicator function which we use in our derivation: DISPLAYFORM3 0, |s| ≤ a (|s| − a)/a, a ≤ |s| ≤ 2a 1, 2a < |s|Now: DISPLAYFORM4 Eψ ξg (x, δ 0g) − ψ ξg (x gi, δ 0g)≤ 2E sup DISPLAYFORM5 where gi are iid copies of Rademacher random variable which are independent of every other random variables and themselves. Now we add back 1 n and expand δ 0g = δ 0 + δ g: DISPLAYFORM6 1 √ n g gi x gi, δ g (n 0 := n, 0i := 0, x 0i := x i) = 2 √ n E sup DISPLAYFORM7 Note that the h gi is a sub-Gaussian random vector which let us bound the E sup using the Gaussian width in the last step. Proof. To avoid cluttering let h g (ω g, X g) = n ng ω g 2 sup ug∈Ag X T g ωg ωg 2, u g, e g = ζ g kω(A g) + g √ log G + τ, where s g = n ng(2K 2 + 1)n g.P (h g (ω g, X g) > e g s g ) = P h g (ω g, X g) > e g s g n n g ω g 2 > s g P n n g ω g 2 > s g+ P h g (ω g, X g) > e g s g n n g ω g 2 < s g P n n g ω g 2 < s g ≤ P n n g ω g 2 > s g + P h g (ω g, X g) > e g s g n n g ω g 2 < s g ≤ P ω g 2 > (2K 2 + 1)n g + P sup DISPLAYFORM8, u g > e g ≤ P ω g 2 > (2K 2 + 1)n g + sup DISPLAYFORM9 Let's focus on the first term. Since ω g consists of i.i.d. centered unit-variance sub-Gaussian elements with |||ω gi ||| ψ2 < K, ω 2 gi is sub-exponential with |||ω gi ||| ψ1 < 2K 2. Let's apply the Bernstein's inequality to ω g DISPLAYFORM10 We also know that E ω g 2 2 ≤ n g ) which gives us: DISPLAYFORM11 Finally, we set τ = 2K 2 n g: P ω g 2 > (2K 2 + 1)n g ≤ 2 exp (−ν g n g) = 2 (G + 1) exp (−ν g n g + log(G + 1))Now we upper bound the second term of. Given any fixed v ∈ S p−1, X g v is a sub-Gaussian random vector with X T g v ψ2 ≤ C g k. From Theorem 9 of for any v ∈ S p−1 we have: DISPLAYFORM12 where φ g = sup ug∈Ag u g 2 and in our problem φ g = 1. We now substitute t = τ + g log(G + 1) where g = θ g C g k. DISPLAYFORM13

We provide an estimator and an estimation algorithm for a class of multi-task regression problem and provide statistical and computational analysis..

represent the short summary for finding the corresponding scientific article (long)


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Autonomous vehicles are becoming more common in city transportation. Companies will begin to find a need to teach these vehicles smart city fleet coordination. Currently, simulation based modeling along with hand coded rules dictate the decision making of these autonomous vehicles. We believe that complex intelligent behavior can be learned by these agents through Reinforcement Learning. In this paper, we discuss our work for solving this system by adapting the Deep Q-Learning (DQN) model to the multi-agent setting. Our approach applies deep reinforcement learning by combining convolutional neural networks with DQN to teach agents to fulfill customer demand in an environment that is partially observ-able to them. We also demonstrate how to utilize transfer learning to teach agents to balance multiple objectives such as navigating to a charging station when its en-ergy level is low. The two evaluations presented show that our solution has shown hat we are successfully able to teach agents cooperation policies while balancing multiple objectives. Many business problems that exist in todays environment consist of multiple decisions makers either collaborating or competing towards a particular goal. In this work, the challenge is applying multi-agent systems for autonomous fleet control. As Autonomous Vehicles (AVs) are becoming more prevalent, companies controlling these fleets such as Uber/Lyft will need to teach these agents to make optimal decisions. The goal of this work is to train these agents/cars optimal relocation strategies that will maximize the efficiency of the fleet while satisfying customer trip demand. Traditional solutions will use discrete event simulation modeling to optimize over a chosen objective function. This approach requires various hand coded rules as well as assumptions to help the model converge on a solution. This becomes an extremely difficult problem when there are many outside environment dynamics that can influence an agents/cars decision making (E.g. Charging, Parking). Furthermore, a solution to a particular environment may become outdated with new incoming information (E.g. New Demand Distribution).An algorithm that can adapt and learn decision making organically is needed for these types of problems and recent works in Reinforcement Learning and particularly Deep Reinforcement Learning has shown to be effective in this space. Deep Minds recent success with Deep Q Learning (DQN) was proven to be very successful in learning human level performance for many Atari 2600 games which was difficult before this because of its highly dimension unstructured data. In this work, we will pull from prior work in Multi-Agent Deep Reinforcement Learning (MA-DRL) and extend this to our multi-agent system of cars and fleet coordination. We will represent the city environment that holds the cars and customers as an image-like state representation where each layer holds specific information about the environment. We then will introduce our work with applying this to a partially observable environment where agents can only see a certain distance from them and show how this helps with scaling up. Along with that, we will show how we took advantage of Transfer Learning to teach agents multiple objects in particular charging an import aspect of AVs. Our show that we are successfully able to teach coordination strategies with other cars so that they can optimize the utility of each car. Finally, we are also able to teach agents the second object of keeping itself alive while not losing the previous objective of picking up customers. The domain of Deep Reinforcement Learning has garnered increased attention recently because of its effectiveness in solving highly dimensional problem spaces. Although its sub-field of multiagent systems presents a difficult challenge in how we represents other agents in the environment. Since these agents are non-stationary, how do we train agents to intelligently cooperate and find an optimal policy when each agent's rewards depend on one another? [Tampuu, Ardi] builds on DeepMind success with the Atari game pong and tests cooperation and competitive policies with each player. [Foerster, Jakob] proposes two novel approaches (RIAL,DIAL) which allows error to backpropagate through multiple agents. Although, this approach has only been tested on riddles where there is a small number of agents and does not seem to scale well to larger problems. [Palmer, Gregory] solves the experience replay memory becoming outdated problem of multi-agent system. [Egorov, Maxim] introduces a novel approach to solve multi-agent systems through convolutional neural networks, stochastic policies, residual networks and is what we will building our solution for the large scale autonomous fleet control on. We will extend some of this work to partially observable and multi-objective case of our agents. The first step in training a Reinforcement Learning model to to create an environment that is representative of the real world characteristics of your problem. In our case, we needed to create an environment that represents the city dynamics that are associated with the Uber/Lyft ride sharing platform. The general goal of each agent is to travel to customers on the map and fulfill that demand. The different objects and locations that are present in this environment are as follows and can be referenced in the example figure 1.Car (Blue Circle): This is the agent or decision maker that will be acting in the environment. Its actions are right, left, up, down, and stay. The car also has an associated energy level that decrements every timestep until it runs out of batter. Agents can occupy the same space as other agents without collision. Agents can not travel outside of the defined map or into obstacles on the map. The goal location of where the Agents need to travel to. Once an agents location matched a customers location, then the customer will be considered fulfilled. Customers also have a drop-off location and travel time associated with them that determines how long an agent/car is removed from the system while in transit. Each customer begins with a random wait time of 5-15 timesteps which decrements every step. Once the wait time has been expired the customer is expire and be removed from the map. Finally, customers are generated from a distribution of demand data that we created based on real trip data for the test city location. Each timestep, new customers are generated by pulling from this distribution. Obstacles (Gray Square): These are the locations on the map that the agents cannot travel to. Customer also will not appear at these locations. Charging Stations (Yellow Square): These are the locations that agents can travel to refill their energy levels. Agents must choose the Stay action at these locations for of their battery to be refilled. Customers can appear at these locations. Open Road (Green Square): These are the standard spaces that agents and customers can exist on. Each timestep the order of activity are as follows, advance each agent sequentially, resolve their pickups or collisions, decremented car energy levels and customer wait times, remove customers that have waited too long or have been picked up, and finally generate new customers from the distribution. Each one of these transitions will have a reward associated with each agent. These events and and the reward structure are shown in TAB0.After each timestep the rewards are aggregated and attributed to that specific agent for that initial state. To point out, you may see there is a small negative reward for standard movement. This was essential to incentive agents to find the quickest path to the customers. Another important reward structure decision was the small positive reward for charging. We found that there was not enough signal to just have a large negative penalty for losing all of its energy. We had to incentivize being in that charging station space without giving a strong enough reward to detract from the agents picking up the customers. Now that we defined our various objects and reward structure, we need to represent the environment as a vector that the Deep Reinforcement Learning model can learn from. Along with the suggestions of previous work, we found that the best way to do this while keeping spatial representations was with an image-like structure. Just like how images provide 3 matrices stacked on top of each other to represent rgb values, the same can be done for our environment. Our state representation has 5 layers that encode a different piece of information and in a tensor of 5 X W x H. The following are what each channels represents. Self Layer: Simply encodes the location of the agent of interest. A value of 1 is given to the (x,y) location where the agent exists. Each self layer is unique to each agent. Other Agents Layer: This encodes the locations of all the other agents excluding yourself. A value of 1 is given to locations where other agents exists and if there are two agents at a location the value 1 is still used. Customer Layer: This encodes the locations of all the customers on the map or within each agent's vision. Here an integer value that represents that customers wait time remaining is placed at the location for that specific customer. When customers are removed from the map than the value will return to 0.Obstacles Layer: This encodes the locations of obstacles and charging locations. The value 1 is used to represent locations where the agent cant go to and the value 2 is used to encode the location of the charging station. Extra Agent Information: This encodes the energy and priority of the agent of interest. For energy we placed the integer value of how much energy is left on the location of the matrix and the priority on the location As we will mention later, in the partially observable version of this environment, we limit this state space representation to just be of that specific agents vision distance v. Figure 3, is the visual matrix of how the example map presented above translates to these layers. In the following sections we will build on the simple Deep Reinforcement Learning technique Deep Q Learning (DQN) and walk through the adaptations that make it the final partially observable multi Agent System (PO-MA-DRL). We will describe how these methods relate to our environment and how they affected our implementation decisions for this environment. The goal of reinforcement learning is to learn an optimal decision making policy that will maximize the future expected reward. One of the most common Reinforcement Learning algorithms is QLearning which represents the maximum discounted reward when we perform action a in state s, and continue optimally from that point on. Q-Learning is a form of model-free learning, meaning that it does not have to learn a model of the environment to optimally settle on a policy. The following formula is the Q-Function which is recursively calculated with transitions of (s,a,r,s). In our case, s represents the image like array of the state representation holding all the agents, customers, and obstacles. The values a and r are the straightforward action taken by the agent and the associate reward. s represents the image-like state after the agent has moved to its next location and collision detection or customer pick up has been resolved. DISPLAYFORM0 This formula is essentially the Bellman equation which is simply the current reward plus the discounted future reward of the next time step if you were to take the best action. This can be implemented as a large table of states by actions and can be recursively solved if the environment is explored enough. If iterated enough times, the Q-Function should approach the true value of taking a certain action in a state and an optimal decision policy can be looked up by taking the action that will return that maximum value. When environments become complex with very large state representations, like Atari games, then it becomes computationally difficult to converge on a reasonable Q-Function because the look-up tables can become very large. This is where the recent success in Deep Q Learning has come into play. Neural networks do a great job in generalizing highly dimensional data in a lower space. We can essentially represent that large Q-Function table with a neural network. The Q(a,s) value can be estimated with the net and the loss of this network simply is as follows. DISPLAYFORM1 Along the lines of [Egorov, Maxim], we also decided to use Convolutional Neural Networks (CNNs) to interpret these image like state representations. CNNs have shown to be successful for image classification tasks because of its ability to understand spatial relationships within the image. The same can be said with the importance of the geospatial location of agents in relation to other agents and customers. In addition, CNN's allow us to scale our environment to large sizes without the exponential increase in processing time that may have occurred with fully connected networks. Finally what makes the convergence of this deep Q-Learning algorithm possible is the addition of Experience Replay, e-greedy exploration, and target networks. Experience Replay allows us to store all of our ¡s,a,r,s¿ transitions in a bucket that we can access. During training, mini batches of transitions are pulled from the bucket and used to fit on the network. E-greedy exploration, allows us to initially randomly take actions but over time the agent will take actions it know in the maximum expected reward. Lastly, target networks, is a separate network that is a mere copy of the previous network, but frozen in time. This network is updated at slower intervals than the actual network as to stabilize the convergence of the Q-values during training. The biggest challenge with multi-agent systems is how we are going to optimize one's own control policy while understanding the intentions of the other agents. Referring back to our image representation of the state, we have one layer that represents yourself and another layer that represents the other agents. Now when we defined s as the updated image-like state after the agent has moved earlier, we must now modify that to account for the other agents in the system. The final value s represents the image-like state after all the other agents have moved. So, each agent is not looking at the other agents as a stationary object, rather as intentions that need to be accounted for when making its own optimal decision. When we increment the state from s to s, we must perform all the conflict resolutions of agent-agent and agent-customer interactions before we can resolve the final state. All the agents move sequentially rather than simultaneously which means that agents with the higher priority will be first to pick up customers. We decided to have all the agents controlled by a single network as a sort of fleet manage rather than have each agent train its own network. While we lose the ability to teach agents different personality and pickup styles, for the purpose of scaling up to a large number of agents this was not feasible. Another incentivization used to create signals for agents to act efficiently with one another is the penalty for missing a customer. Each agent feels the negative reward of missing a customer even if it is on the other side of the map. This incentivises the agents to learn a type of divide and conquer technique as to minimize the probability of missing a future customer. In the Results/Tests section we will show how effective this has been in a simple bridge example. A final consideration was how do we update the replay memory with transitions from multiple agents. We decided to increase the Replay size proportional to the number of agents so that the refresh rate of the memory would be consistent to if it were a single agent. As proposed by [Palmer, Gregory], there can be more efficient ways to update the experience replay for multi-agent systems but we leave that for later work. Partial observability is the final addition to our methods and is where our work differs from some of the related works. When attempted to train our MA-DRL with actual data from a pilot city of Autonomous Vehicles, the problem just became too large with over 200+ cars and a 25x25 grid size. Using CNNs were not enough to scale the processing of the image-like representations. To solve this, we made the agents partially observable to the entire environment or map. We limited how far an agent can see around them as v (e.g. +10 spaces away) which limits how big the state image-like representation can be. Now if the map explodes to a 100x100, the transition states will always be of size v x v and training time will not exponentially explode. Making these agents partially observable also greatly helped with the missed customer penalty mentioned in the earlier section. As the map becomes extremely large, there are dozens of customers that appear and disappear every timestep. Since we penalized every single agent in the system for missed customers, the signals became very convoluted and they were not able to recognize the importance of picking up customers. When we limited the state representation to just be of size v x v, the penalties attributed for missed customers also only applied to that partially observable view. Some of the initial concerns were that if an agent did not have a complete view of the map it may wander off in a undesirable location. Although, if enough iterations are done, each agent should have a good understanding of the map and where they are over a few timesteps. The drawback here is that the agents will not be able to generalize to other unseen maps that may have very different geographies. The final method we will be talking about is the use of Transfer Learning to teach agents multiple objectives. In our AV fleet control problem, agents/cars have the main objective of fulfilling customer demand but have the secondary objective to charge its battery when needed. As state earlier in the reward structure, agents received a positive reward when picking up customers but a large negative reward when it loses all of its energy. We found that when training on this reward structure, the agents did not seem to converge on any successful strategy. In some iterations, the agents only figured out how to go to the charging locations and stay there. In other cases, the agents just continued picking up customers until all the agents died. To solve this, we took advantage of transfer learning to consecutively teach these objectives. We begun with training the standard PO-MA-DRL model where the agents were given an infinite amount of energy. On the next iteration, the map and reward structure stayed consistent but this time we have each agent a random energy level from (50 minutes to 150 minutes). Now the agents were beginning to receive the large negative rewards but over time they were able to balance these two objectives and charge themselves when needed. In the second part of the Results/Tests we will show the of this improvement. In this section we provide two experimental ; first, we wanted to test the effectiveness of agent communication through divide and conquer techniques. Second, we tested the effectiveness of implementing transfer learning for teaching the agents to charge. In both tests, we compared the agents with baseline policies that we created. These baseline policies were build with Dijkstra's algorithm which calculates the shortest distance to each customer and chooses the action that will get to the customer the quickest. The goal of these experiments are to show the power of a DRL compared to a rules based Dijkstras approach. The goal of this evaluation is to simply test the ability of the agents to learning coordination policies as to maximize the reward of all the agents in the system. Our environment consisted of a 7x7 map with 2 agents similar to the example in figure 1. Our experiment randomly placed these agents on the board and randomized their priority. The map is in the form of a bridge in the middle and two opposite sides of the nap where customers appear randomly. The optimal cooperation policy here would be for the agents to split up and pick up customers on opposite sides of the map. We tested if our model can perform this divide and conquer coordination technique versus the baseline model which would just travel to the closest customer. FIG1 shows the average customers fulfilled of the model versus the baseline. We can see that as time (game length) increases the models reward begins to diverge from the baseline. This shows that the policies that the two model agents are performing more efficiently than the baselines. We inspected the individual games that were ing and found that the agents did indeed perform a dived and conquer technique. When randomly placed on the same side of the map, one agent would take the initiative and travel to the other side of the map. When both agents started on different sides, they would just stay there and pick up their respective customers. This simple example, successfully demonstrates that an agent can create a policy based on the other agent in the environment and the map structure to improve its overall expected future reward. The goal of this experiment was to demonstrate the ability to teach multiple objectives to agents through transfer learning and show its strength over a rules based approach. Our environment was a 10x10 bridge map with 4 agents like the one shown in figure 1. As described in the transfer learning section above, we initially trained a model to pick up customers with infinite energy. We then took that same trained model, gave each agent a random amount of energy and activated the penalty for losing all of your energy. When fine tuning, we found that we needed to set the epsilon for the e-greedy exploration to.5. Which meant that it would take the best action 50 percent of the time and a random otherwise. We linearly decayed this value to.01 over 3000 iterations. We found this approach to be successful in teaching the agent to keep itself alive and still pick up customers versus training a model altogether to learn to manage both objectives from scratch. In our experiment we wanted to demonstrate that the RL model was successful in learning how to balance these two objectives effectively. Would the model stray too closely to keeping itself alive and only pick up a few customers? Or would it pick up many customers and sparingly recharge itself. We compared our model to two baseline agents. The first baseline agent would be a conservative agent where it would go back to the charging station if its energy was below 40 percent. The second, was a more aggressive baseline agent and would only go charge its energy when it was below 10 percent. Rule based simulation models would have to set these thresholds according to business requirements and goals. Setting a hard threshold is not the most optimal though as in some scenarios it may be more beneficial to be more aggressive and get the customers near you. In others, it helps to be more conservative as customers may be far away. We can see that the conservative baseline model successfully goes and charges it's battery when low but ends up picking up less customers than the aggressive baseline. Unfortunately, the aggressive baseline loses its energy more frequently and has experience around 11 deaths over the course of the trials. Our RL model shows that it is able to balance the deaths and customers fulfilled metrics and have an overall significant improvement in net reward. While it still may die from time to time, the overall performance boost of balancing these two objectives far exceeds the general baseline models. Deep Reinforcement Learning provides a great approach to teach agents how to solve complex problems that us as humans may never be able to solve. For instance, Deep Mind has been successful in teach an agent to defeat the world champion in Go. More specifically, multi-Agent Reinforcement Learning problems provide an interesting avenue to investigate agent to agent communication and decision protocols. Since agents must rationalize about the intentions of other agents the dimensionality of the problem space becomes difficult to solve. In our use case, we wanted to see if we can scale a DRL solution up to an actual ride sharing environment that maintains the same dynamics as it would in real life. For this to be possible, we were tasked with the problem of teaching these agents effective cooperation strategies that would optimize the reward of the system along with the problem of teaching these same agents multiple objectives. This work, demonstrated how we successfully applied a partially observable multi-agent deep reinforcement solution to this ride sharing problem. Along with that, we showed how we can effectively take advantage of transfer learning to adapt decision policies to account for multiple objectives.

Represent the short summary for finding the corresponding scientific article (long).

Utilized Deep Reinforcement Learning to teach agents ride-sharing fleet style coordination.

Represent the TLDR for finding the corresponding scientific article

Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.


Question: Represent the article for finding a single-sentence TLDR

Question: Stability is a key aspect of data analysis. In many applications, the natural notion of stability is geometric, as illustrated for example in computer vision. Scattering transforms construct deep convolutional representations which are certified stable to input deformations. This stability to deformations can be interpreted as stability with respect to changes in the metric structure of the domain. In this work, we show that scattering transforms can be generalized to non-Euclidean domains using diffusion wavelets, while preserving a notion of stability with respect to metric changes in the domain, measured with diffusion maps. The ing representation is stable to metric perturbations of the domain while being able to capture''high-frequency'' information, akin to the Euclidean Scattering. Convolutional Neural Networks (CNN) are layered information processing architectures. Each of the layers in a CNN is itself the composition of a convolution operation with a pointwise nonlinearity where the filters used at different layers are the outcome of a data-driven optimization process BID22. Scattering transforms have an analogous layered architecture but differ from CNNs in that the convolutional filters used at different layers are not trained but selected from a multi-resolution filter bank BID25 BID3. The fact that they are not trained endows scattering transforms with intrinsic value in situations where training is impossible -and inherent limitations in the converse case. That said, an equally important value of scattering transforms is that by isolating the convolutional layered architecture from training effects it permits analysis of the fundamental properties of CNN information processing architectures. This analysis is undertaken in BID25; BID3 where the fundamental is about the stability of scattering transforms with respect to deformations in the underlying domain that are close to translations. In this paper we consider graphs and signals supported on graphs such as brain connectivity networks and functional activity levels BID17, social networks and opinions BID19, or user similarity networks and ratings in recommendation systems BID18. Our specific goals are: (i) To define a family of graph-scattering transforms. (ii) To define a notion of deformation for graph signals. (iii) To study the stability of graph scattering transforms with respect to this notion of deformation. To accomplish goal (i) we consider the family of graph diffusion wavelets which provide an appropriate construction of a multi-resolution filter bank BID8. Our diffusion scattering transforms are defined as the layered composition of diffusion wavelet filter banks and pointwise nonlinearities. To accomplish goal (ii) we adopt the graph diffusion distance as a measure of deformation of the underlying domain BID27. Diffusion distances measure the similarity of two graphs through the time it takes for a signal to be diffused on the graph. The major accomplishment of this paper is to show that the diffusion graph scattering transforms are stable with respect to deformations as measured with respect to diffusion distances. Specifically, consider a signal x supported on graph G whose diffusion scattering transform is denoted by the operator Ψ G. Consider now a deformation of the signal's domain so that the signal's support is now described by the graph G whose diffusion scattering operator is Ψ G. We show that the operator norm distance Ψ G − Ψ G is bounded by a constant multiplied by the diffusion distance between the graphs G and G. The constant in this bound depends on the spectral gap of G but, very importantly, does not depend on the number of nodes in the graph. It is important to point out that finding stable representations is not difficult. E.g., taking signal averages is a representation that is stable to domain deformations -indeed, invariant. The challenge is finding a representation that is stable and rich in its description of the signal. In our numerical analyses we show that linear filters can provide representations that are either stable or rich but that cannot be stable and rich at the same time. The situation is analogous to (Euclidean) scattering transforms and is also associated with high frequency components. We can obtain a stable representation by eliminating high frequency components but the representation loses important signal features. Alternatively, we can retain high frequency components to have a rich representation but that representation is unstable to deformations. Diffusion scattering transforms are observed to be not only stable -as predicted by our theoretical analysis -but also sufficiently rich to achieve good performance in graph signal classification examples. Since graph and graph signals are of increasing interest but do not have the regular structure that would make use of CNNs appealing, it is pertinent to ask the question of what should be an appropriate generalization of CNNs to graphs and the graph signals whose topology they describe BID2. If one accepts the value of convolutions as prima facie, a natural solution is to replace convolutions with graph shift invariant filters which are known to be valid generalizations of (convolutional) time invariant filters BID4. This idea is not only natural but has been demonstrated to work well in practical implementations of Graph Neural Networks (GNNs) BID9 BID11 BID13 BID16 BID20. Same as Euclidean scattering transforms, our graph scattering transforms differ from GNNs in that they do not have to be trained. The advantages and limitations of the absence of training notwithstanding, our work also sheds light on the question of why graph convolutions are appropriate generalizations of regular domain convolutions for signal classification problems. Our work suggests that the value of GNNs stems from their stability relative to deformations of the underlying domain that are close to permutations -which is the property that a pair of graphs must satisfy to have small diffusion distance. The stability obtained in this paper build on the notion of scattering transforms. These scattering representations were introduced by BID25 and further developed in BID3 with computer vision applications. Since, these representations have been extended to handle transformations on more complex groups, such as roto-translations BID31 BID28, and to domains such as audio processing BID0 and quantum chemistry BID10.Similarly as in this work, extensions of scattering to general graphs have been considered in BID5 and BID34. BID5 focuses on Haar wavelets that hierarchically coarsen the graph, and relies on building multiresolution pairings. The recent BID34 is closest to our work. There, the authors define graph scattering using spectrally constructed wavelets from BID15, and establish some properties of the ing representation, such as energy conservation and stability to spectral perturbations. In contrast, our stability are established with respect to diffusion metric perturbations, which are generally weaker, in the sense that they define a weaker topology (see Section 3). We use diffusion wavelets BID8 ) to obtain multi-resolution graph filter banks that are localized in frequency as well as in the graph domain, while spanning the whole spectrum. Diffusion wavelets serve as the constructive basis for the obtained stability . Our work is also closely related to recent analysis of stability of Graph Neural Networks in the context of surface representations in BID21. In our work, however, we do not rely on extrinsic deformations and exploit the specific multiresolution structure of wavelets. This section introduces our framework and states the desired stability properties of signal representations defined on general non-Euclidean domains. Motivated by computer vision applications, our analysis starts with the notion of deformation stability. If x(u) ∈ L 2 (Ω) is an image defined over an Euclidean domain Ω ⊂ R d, we are interested in signal representations Φ: L 2 (Ω) → R K that are stable with respect to small deformations. If DISPLAYFORM0 ) denotes a change of variables with a differentiable field τ: Ω → Ω such that ∇τ < 1, then we ask DISPLAYFORM1 τ:= ∇τ ∞ denoting a uniform bound on the operator norm of ∇τ. In this setting, a notorious challenge to achieving while keeping enough discriminative power in Φ(x) is to transform the high-frequency content of x in such a way that it becomes stable. Scattering transforms BID25 BID3 provide such representations by cascading wavelet decompositions with pointwise modulus activation functions. We briefly summarize here their basic definition. Given a mother wavelet ψ ∈ L 1 (Ω) with at least a vanishing moment ψ(u)du = 0 and with good spatial localization, we consider rotated and dilated versions ψ j,c (u) = 2 −jd ψ(2 −j R c u) using scale parameter j and angle θ ∈ {2πc/C} c=0,...,C−1. A wavelet decomposition operator is defined as a filter bank spanning all scales up to a cutoff 2 J and all angles: DISPLAYFORM2 This filter bank is combined with a pointwise modulus activation function ρ(z) = |z|, as well as a low-pass average pooling operator U computing the average over the domain. The ing representation using m layers becomes DISPLAYFORM3 The ing signal representation has the structure of a CNN, in which feature maps are not recombined with each other, and trainable filters are replaced by multiscale, oriented wavelets. It is shown in BID25 that for appropriate signal classes and wavelet families, the ing scattering transform satisfies a deformation stablity condition of the form, which has been subsequently generalised to broader multiresolution families BID33. In essence, the mechanism that provides stability is to capture high-frequency information with the appropriate spatio-temporal tradeoffs, using spatially localized wavelets. Whereas deformations provide the natural framework to describe geometric stability in Euclidean domains, their generalization to non-Euclidean, non-smooth domains is not straightforward. Let x ∈ L 2 (X). If X is embedded into a low-dimension Euclidean space Ω ⊂ R d, such as a 2-surface within a three-dimensional space, then one can still define meaningful deformations on X via extrinsic deformations of Ω BID21.However, in this work we are interested in intrinsic notions of geometric stability, that do not necessarily rely on a pre-existent low-dimensional embedding of the domain. The change of variables ϕ(u) = u − τ (u) defining the deformation can be seen as a perturbation of the Euclidean metric in DISPLAYFORM0 with dμ(u) = |I − ∇τ (u)|dµ(u), and |I − ∇τ (u)| ≈ 1 if ∇τ is small, where I is the identity. Therefore, a possible way to extend the notion of deformation stability to general domains L 2 (X) is to think of X as a metric space and reason in terms of stability of Φ: L 2 (X) → R K to metric changes in X. This requires a representation that can be defined on generic metric spaces, as well as a criteria to compare how close two metric spaces are. Graphs are flexible data structures that enable general metric structures and modeling non-Euclidean domains. The main ingredients of the scattering transform can be generalized using tools from computational harmonic analysis on graphs. We note that, unlike the case of Euclidean domains, where deformations are equivalent whether they are analyzed from the function domain or its image, in the case of graphs, we focus on deformations on the underlying graph domain, while keeping the same function mapping (i.e. we model deformations as a change of the underlying graph support and analyze how this affects the interaction between the function mapping and the graph).In particular, diffusion wavelets BID8 provide a simple framework to define a multi-resolution analysis from powers of a diffusion operator defined on a graph. A weighted, undirected graph G = (V, E, W) with |V | = n nodes, edge set E and adjacency matrix W ∈ R n×n defines a diffusion process A in its nodes, given in its symmetric form by the normalized adjacency DISPLAYFORM0 where d i = (i,j)∈E W i,j denotes the degree of node i. Denote by d = W 1 the degree vector containing d i in the ith element. By construction, A is well-localized in space (it is nonzero only where there is an edge connecting nodes), it is self-adjoint and satisfies A ≤ 1, where A is the operator norm. Let λ 0 ≥ λ 1 ≥... λ n−1 denote its eigenvalues in decreasing order. Defining DISPLAYFORM1, one can easily verify that the normalized squared root degree vector DISPLAYFORM2 1 is the eigenvector with associated eigenvalue λ 0 = 1. Also, note that λ n−1 = −1 if and only if G has a connected component that is non-trivial and bipartite BID6.In the following, it will be convenient to assume that the spectrum of A (which is real and discrete since A is self-adjoint and in finite-dimensions) is non-negative. Since we shall be taking powers of A, this will avoid folding negative eigenvalues into positive ones. For that purpose, we adopt the so-called lazy diffusion, given by T = 1 2 (I + A). In Section 4 we use this diffusion operator to define both a multiscale wavelet filter bank and a low-pass average pooling, leading to the diffusion scattering representation. This diffusion operator can also be used to construct a metric on G. The so-called diffusion distances BID27 measure distances between two nodes x, x ∈ V in terms of their associated diffusion at time s: In this work, we build on this diffusion metric to define a distance between two graphs G, G. Assuming first that G and G have the same size, the simplest formulation is to compare the diffusion metric generated by G and G up to a node permutation: DISPLAYFORM3 DISPLAYFORM4 where Π n is the space of n × n permutation matrices. The diffusion distance is defined at a specific time s. As s increases, this distance becomes weaker 1, since it compares points at later stages of diffusion. The role of time is thus to select the smoothness of the'graph deformation', similarly as ∇τ measures the smoothness of the deformation in the Euclidean case. For convenience, we denote d(G, G) = d 1/2 (G, G) and use the distance at s = 1/2 as our main deformation measure. The quantity d defines a distance between graphs (seen as metric spaces) yielding a stronger topology than other alternatives such as the Gromov-Hausdorff distance, defined as DISPLAYFORM5. We choose d(G, G) in this work for convenience and mathematical tractability, but leave for future work the study of stability relative to d s GH. Finally, 1 In the sense that it defines a weaker topology, i.e., limm→∞ d DISPLAYFORM6 we consider for simplicity only the case where the sizes of G and G are equal, but definition (3.1) can be naturally extended to compare variable-sized graphs by replacing permutations by softcorrespondences (see BID1 . Our goal is to build a stable and rich representation Φ G (x). The stability property is stated in terms of the diffusion metric above: For a chosen diffusion time s, ∀ x ∈ R n, G = (V, E, W), G = (V, E, W) with |V | = |V | = n, we want DISPLAYFORM0 This representation can be used to model both signals and domains, or just domains G, by considering a prespecified x = f (G), such as the degree, or by marginalizing from an exchangeable distribution, DISPLAYFORM1 The motivation of FORMULA12 is two-fold: On the one hand, we are interested in applications where the signal of interest may be measured in dynamic environments that modify the domain, e.g. in measuring brain signals across different individuals. On the other hand, in other applications, such as building generative models for graphs, we may be interested in representing the domain G itself. A representation from the adjacency matrix of G needs to build invariance to node permutations, while capturing enough discriminative information to separate different graphs. In particular, and similarly as with Gromov-Hausdorff distances, the definition of d(G, G) involves a matching problem between two kernel matrices, which defines an NP-hard combinatorial problem. This further motivates the need for efficient representations of graphs Φ G that can efficiently tell apart two graphs, and such that (θ) = Φ G − Φ G(θ) can be used as a differentiable loss for training generative models. Let T be a lazy diffusion operator associated with a graph G of size n such as those described in Section 3.3. Following BID8, we construct a family of multiscale filters by exploiting the powers of the diffusion operator T 2 j. We define DISPLAYFORM0 This corresponds to a graph wavelet filter bank with optimal spatial localization. Graph diffusion wavelets are localized both in space and frequency, and favor a spatial localization, since they can be obtained with only two filter coefficients, namely h 0 = 1 for diffusion T 2 j−1 DISPLAYFORM1 The finest scale ψ 0 corresponds to one half of the normalized Laplacian operator DISPLAYFORM2, here seen as a temporal difference in a diffusion process, seeing each diffusion step (each multiplication by ∆) as a time step. The coarser scales ψ j capture temporal differences at increasingly spaced diffusion times. For j = 0,..., J n − 1, we consider the linear operator DISPLAYFORM3 which is the analog of the wavelet filter bank in the Euclidean domain. Whereas several other options exist to define graph wavelet decompositions BID29 BID12, and GNN designs that favor frequency localization, such as Cayley filters BID24, we consider here wavelets that can be expressed with few diffusion terms, favoring spatial over frequential localization, for stability reasons that will become apparent next. We choose dyadic scales for convenience, but the construction is analogous if one replaces scales 2 j by γ j for any γ > 1 in.If the graph G exhibits a spectral gap, i.e., β G = sup i=1,... n−1 |λ i | < 1, the following proposition proves that the linear operator Ψ defines a stable frame. Proposition 4.1. For each n, let Ψ define the diffusion wavelet decomposition and assume β G < 1. Then there exists a constant 0 < C(β) depending only on β such that for any x ∈ R n satisfying x, v = 0, DISPLAYFORM4 This proposition thus provides the Littlewood-Paley bounds of Ψ, which control the ability of the filter bank to capture and amplify the signal x along each'frequency' (i.e. the ability of the filter to increase or decrease the energy of the representation, relative to the energy of the x). We note that diffusion wavelets are neither unitary nor analytic and therefore do not preserve energy. However, the frame bounds in Proposition 4.1 provide lower bounds on the energy lost, such that the smaller 1 − β is, the less "unitary" our diffusion wavelets are. It also informs us about how the spectral gap β determines the appropriate diffusion scale J: The maximum of p(u) = (u r − u 2r) 2 is at u = 2 −1/r, thus the cutoff r * should align with β as r * = −1 log 2 β, since larger values of r capture energy in a spectral range where the graph has no information. Therefore, the maximum scale can be adjusted as J = 1 + log 2 r * = 1 + log 2 −1 log 2 β.Recall that the Euclidean Scattering transform is constructed by cascading three building blocks: a wavelet decomposition operator, a pointwise modulus activation function, and an averaging operator. Following the Euclidean scattering, given a graph G and x ∈ L 2 (G), we define an analogous Diffusion Scattering transform Φ G (x) by cascading three building blocks: the Wavelet decomposition operator Ψ, a pointwise activation function ρ, and an average operator U which extracts the average over the domain. The average over a domain can be interpreted as the diffusion at infinite time, thus U x = lim t→∞ T t x = v T, x. More specifically, we consider a first layer transformation given by DISPLAYFORM5 followed by second order coefficients DISPLAYFORM6 and so on. The representation obtained from m layers of such transformation is thus DISPLAYFORM7 5 STABILITY OF GRAPH DIFFUSION SCATTERING Given two graphs G, G of size n and a signal x ∈ R n, our objective is to bound DISPLAYFORM0. Let π * the permutation minimising the distortion between G and G in. Since all operations in Φ are either equivariant or invariant with respect to permutations, we can assume w.l.o.g. that π = 1, so that the diffusion distance can be directly computed by comparing nodes with the given order. A key property of G that drives the stability of the diffusion scattering is given by its spectral gap 1 − β G = 1 − sup i=1,... n−1 |λ i | ≥ 0. In the following, we use 2 operator norms, unless stated otherwise. Lemma 5.1. DISPLAYFORM1 Remark: If diffusion distance is measured at time different from s = 1/2, the stability bound would be modified due to scales j such that 2 j < s. The following lemma studies the stability of the low-pass operator U with respect to graph perturbations. Lemma 5.2. Let G, G be two graphs with same size, denote by v and v their respective squaredroot degree vectors, and by β, β their spectral gap. Then DISPLAYFORM2 Spectral Gap asymptotic behavior Lemmas 5.1 and 5.2 leverage the spectral gap of the lazy diffusion operator associated with G. In some cases, such as regular graphs, the spectral gap vanishes asymptotically as n → ∞, thus degrading the upper bound asymptotically. Improving the bound by leveraging other properties of the graph (such as regular degree distribution) is an important open task. The scattering transform coefficients Φ G (x) obtained after m layers are given by equation 11, for low-pass operator U such that U x = v, x so that U = v T.From Lemma 5.1 we have that, DISPLAYFORM0 We also know, from Proposition 4.1 that Ψ conforms a frame, i.e. C(β) x 2 ≤ Ψx 2 ≤ x 2 for known constant C(β) given in Prop. 4.1. Additionally, from Lemma 5.2 we get that DISPLAYFORM1 The objective now is to prove stability of the scattering coefficients Φ G (x), that is, to prove that DISPLAYFORM2 This is captured in the following Theorem: Theorem 5.3. Let G, G be two graphs and let d(G, G) be their distance measured as in equation 4. Let T G and T G be the respective diffusion operators. Denote by U G, ρ G and Ψ G and by U G, ρ G and Ψ G the low pass operator, pointwise nonlinearity and the wavelet filter bank used on the scattering transform defined on each graph, respectively, cf. equation 11. Assume ρ G = ρ G and that ρ G is non-expansive. Let β − = min(β G, β G), β + = max(β G, β G) and assume β + < 1. Then, we have that, for each k = 0,..., m − 1, the following holds BID3, it is straightforward to compute the stability bound on the scattering coefficients as follows. Corollary 5.4. In the context of Theorem 5.3, let x ∈ R n and let Φ G (x) be the scattering coefficients computed by means of equation 11 on graph G after m layers, and let Φ G (x) be the corresponding coefficients on graph G. Then, DISPLAYFORM3 DISPLAYFORM4 Corollary 5.4 satisfies equation 5. It also shows that the closer the graphs are in terms of the diffusion metric, the closer their scattering representations will be. The constant is given by topological properties, the spectral gaps of G and G, as well as design parameters, the number of layers m. We observe that the stability bound grows the smaller the spectral gap is and also as more layers are considered. The spectral gap is tightly linked with diffusion processes on graphs, and thus it does emerge from the choice of a diffusion metric. Graphs with values of beta closer to 1, exhibit weaker diffusion paths, and thus a small perturbation on the edges of these graphs would lead to a larger diffusion distance. The contrary holds as well. In other words, the tolerance of the graph to edge perturbations (i.e., d(G, G) being small) depends on the spectral gap of the graph. We also note that, as stated at the end of Section 5.1, the spectral gap appears in our upper bounds, but it is not necessarily sharp. In particular, the spectral gap is a poor indication of stability in regular graphs, and we believe our bound can be improved by leveraging structural properties of regular domains. Finally, we note that the size of the graphs impacts the stability inasmuch as it impacts the distance measure d(G, G). This is expected, since graphs of different size can be compared, as mentioned in Section 3.3. Different from BID34, our focus is on obtaining graph wavelet banks that are localized in the graph domain to improve computational efficiency as discussed in BID9. We also notice that the scattering transform in BID34 is stable with respect to a graph measure that depends on the spectrum of the graph through both eigenvectors and eigenvalues. More specifically, it is required that the spectrum gets concentrated as the graphs grow. However, in general, it is not straightforward to relate the topological structure of the graph with its spectral properties. As mentioned in Section 3.3, the stability is computed with a metric d(G, G) which is stronger than what could be hoped for. Our metric is permutation-invariant, in analogy with the rigid translation invariance in the Euclidean case, and stable to small perturbations around permutations. The extension of to weaker metrics, using e.g. multiscale deformations, is left for future work. By denoting T j = T 2 j, observe that one can approximate the diffusion wavelets from as a cascade of low-pass diffusions followed by a high-pass filter at resolution 2 j: DISPLAYFORM0 This pyramidal structure of multi-resolution analysis wavelets -in which each layer now corresponds to a different scale, shows that the diffusion scattering is a particular instance of GNNs where each layer j is generated by the pair of operators {I, T j−1}. If x (j) ∈ R n×dj denotes the feature representation at layer j using d j feature maps per node, the corresponding update is given by DISPLAYFORM1 where θ DISPLAYFORM2 In this case, a simple modification of the previous theorem shows that the ing GNN representation DISPLAYFORM3 2 ) j≤J is also stable with respect to d(G, G), albeit this time the constants are parameter-dependent:Corollary 5.5. The J layer GNN with parameters Θ = (θ DISPLAYFORM4 This bound is thus learning-agnostic and is proved by elementary application of the diffusion distance definition. An interesting question left for future work is to relate such stability to gradient descent optimization biases, similarly as in BID14 BID32, which could provide stability certificates for learnt GNN representations. In this section, we first show empirically the dependence of the stability with respect to the spectral gap, and then we illustrate the discriminative power of the diffusion scattering transform in two different classification tasks; namely, the problems of authorship attribution and source localization. Consider a small-world graph G with N = 200 nodes, edge probability p SW and rewiring probability q SW = 0.1. Let x ∼ N (0, I) be a random graph signal defined on top of G and Φ G (x) the corresponding scattering transform. Let G be another realization of the small-world graph, and let Φ G (x) be the scattering representation of the same graph signal x but on the different support G. We can then proceed to compute Φ G (x) − Φ G (x). By changing the value of p SW we can change value of the spectral gap β and study the dependence of the difference in representations as a function of the spectral gap. Results shown in FIG1 are obtained by varying p SW from 0.1 to 0.9. For each value of p SW we generate one graph G and 50 different graphs G; and for each graph G we compute Φ G (x) − Φ G (x) for 1, 000 different graph signal realizations x. The average across all signal realizations is considered the estimate of the representation difference, and then the mean as well as the variance across all graphs are computed and shown in the figure (error bars). DISPLAYFORM0 as a function of the spectral gap (changing p SW from 0.1 to 0.9 led to values of spectral gap between 0.5 and close to 1). First and foremost we observe that, indeed, as β reaches one, the stability gets worse and the representation difference increases. We also observe that, for deeper scattering representations, the difference also gets worse, although it is not a linear behaviour as predicted in equation 16, which suggest that the bound is not tight. For classifying we train a SVM linear model fed by features obtained from different representations. We thus compare with two non-trainable linear representations of the data: a data-based method (using the graph signals to feed the classifier) and a graph-based method (obtaining the GFT coefficients as features for the data). Additionally, we consider the graph scattering with varying depth to analyze the richness of the representation. Our aim is mainly to illustrate that the scattering representation is rich enough, relative to linear representations, and is stable to graph deformations. First, we consider the problem of authorship attribution where the main task is to determine if a given text was written by a certain author. We construct author profiles by means of word adjacency networks (WAN). This WAN acts as the underlying graph support for the graph signal representing the word count (bag-of-words) of the target text of unknown authorship. Intuitively, the choice of words of the target text should reflect the pairwise connections in the WAN, see BID30 for detailed construction of WANs. In particular, we consider all works by Jane Austen. To illustrate the stability , we construct a WAN with 188 nodes (functional words) using a varying number of texts to form the training set, obtaining an array of graphs that are similar but not exactly the same. For the test set, we include 154 excerpts by Jane Austen and 154 excerpts written by other contemporary authors, totaling 308 data points. FIG1 shows classification error as a function of the number of training samples used. We observe that graph scattering transforms monotonically improve while considering more training data, whereas other methods vary more erratically, showing their lack of stability (their representations vary more wildly when the underlying graph support changes). This shows that scattering diffusion transforms strike a good balance between stability and discriminative power. For the second task, let G be a 234-node graph modeling real-world Facebook interactions BID26. In the source localization problem, we observe a diffusion process after some unknown time t, that originated at some unknown node i, i.e. we observe x = W t δ i, where δ i is the signal with all zeros except a 1 on node i. The objective is to determine which community the source node i belongs to. These signals can be used to model rumors that percolate through the social network by interaction between users and the objective is to determine which user group generated said rumor (or initiated a discussion on some topic). We generate a training sample of size 2, 000, for nodes i chosen at random and diffusion times t chosen as random as well. The GFT is computed by projecting on the eigenbasis of the operator T. We note that, to avoid numerical instabilities, the diffusion is carried out using the normalized operator (W/λ max (W)) and t ≤ t max = 20. The representation coefficients (graph signals, GFT or scattering coefficients) obtained from this set are used to train different linear SVMs to perform classification. For the test set, we draw 200 new signals. We compute the classification errors on the test set as a measure of usefulness of the obtained representations. Results are presented in FIG1, where perturbations are illustrated by dropping edges with probability p (adding or removing friends in Facebook). Again, it is observed that the scattering representation exhibits lower variations when the underlying graph changes, compared to the linear approaches. Finally, to remark the discriminative power of the scattering representation, we observe that as the graph scattering grows deeper, the obtained features help in more accurate classification. We remark that in regimes with sufficient labeled examples, trainable GNN architectures will generally outperform scattering-based representations. In this work we addressed the problem of stability of graph representations. We designed a scattering transform of graph signals using diffusion wavelets and we proved that this transform is stable under deformations of the underlying graph support. More specifically, we showed that the scattering transform of a graph signal supported on two different graphs is proportional to the diffusion distance between those graphs. As a byproduct of our analysis, we obtain stability bounds for Graph Neural Networks generated by diffusion operators. Additionally, we showed that the ing descriptions are also rich enough to be able to adequately classify plays by author in the context of authorship attribution, and identify the community origin of a signal in a source localization problem. That said, there are a number of directions to build upon from these . First, our stability bounds depend on the spectral gap of the graph diffusion. Although lazy diffusion prevents this spectral gap to vanish, as the size of the graph increases we generally do not have a tight bound, as illustrated by regular graphs. An important direction of future research is thus to develop stability bounds which are robust to vanishing spectral gaps. Next, and related to this first point, we are working on extending the analysis to broader families of wavelet decompositions on graphs and their corresponding graph neural network versions, including stability with respect to the GromovHausdorff metric, which can be achieved by using graph wavelet filter banks that achieve bounds analogous to those in Lemmas 5.1 and 5.2.A PROOF OF PROPOSITION 4.1Since all operators ψ j are polynomials of the diffusion T, they all diagonalise in the same basis. Let T = V ΛV T, where V T V = I contains the eigenvectors of T and Λ = diag(λ 0, . . ., λ n−1) its eigenvalues. The frame bounds C 1, C 2 are obtained by evaluating Ψx 2 for x = v i, i = 1,..., n− 1, since v 0 corresponds to the square-root degree vector and x is by assumption orthogonal to v 0.We verify that the spectrum of ψ j is given by (p j (λ 0),..., p j (λ n−1)), where DISPLAYFORM0 2. It follows from the definition that DISPLAYFORM1.., n − 1 and therefore DISPLAYFORM2 We check that DISPLAYFORM3 2. One easily verifies that Q(x) is continuous in since it is bounded by a geometric series. Also, observe that DISPLAYFORM4 since x ∈. By continuity it thus follows that DISPLAYFORM5 which in g (t) ≤ rβ r−1 B − A, proving.By plugging FORMULA5 into we thus obtain DISPLAYFORM6 (1−β 2) 3. Finally, we observe that DISPLAYFORM7 Without loss of generality, assume that the node assignment that minimizes T G − ΠT G Π T is the identity. We need to bound the leading eigenvectors of two symmetric matrices T G and T G with a spectral gap. As before, let DISPLAYFORM8 Since we are free to swap the role of v and v, the follows. DISPLAYFORM9 First, note that ρ G = ρ G = ρ since it is a pointwise nonlinearity (an absolute value), and is independent of the graph topology. Now, let's start with k = 0. In this case, we get U G x − U G x which is immediately bounded by Lemma 5.2 satisfying equation 15.For k = 1 we have DISPLAYFORM10 where the triangular inequality of the norm was used, together with the fact that ρu − ρu ≤ ρ(u − u) for any real vector u since ρ is the pointwise absolute value. Using the submultiplicativity of the operator norm, we get DISPLAYFORM11 From Lemmas 5.1 and 5.2 we have that Ψ G − Ψ G ≤ ε Ψ and U G − U G ≤ ε U, and from Proposition 4.1 that Ψ G ≤ 1. Note also that U G = U G = 1 and that ρ = 1. This yields DISPLAYFORM12 satisfying equation 15 for k = 1.For k = 2, we observe that DISPLAYFORM13 The first term is bounded in a straightforward fashion by DISPLAYFORM14 analogy to the development for k = 1. Since U G = 1, for the second term, we focus on DISPLAYFORM15 We note that, in the first term in equation 33, the first layer induces an error, but after that, the processing is through the same filter banks. So we are basically interested in bounding the propagation of the error induced in the first layer. Applying twice the fact that ρ(u) − ρ(u) ≤ ρ(u − u) we get DISPLAYFORM16 And following with submultiplicativity of the operator norm, DISPLAYFORM17 For the second term in equation 33, we see that the first layer applied is the same in both, namely ρΨ G so there is no error induced. Therefore, we are interested in the error obtained after the first layer, which is precisely the same error obtained for k = 1. Therefore, DISPLAYFORM18 Plugging equation 35 and equation 36 back in equation 31 we get DISPLAYFORM19 satisfying equation 15 for k = 2.For general k we see that we will have a first term that is the error induced by the mismatch on the low pass filter that amounts to ε U, a second term that accounts for the propagation through (k − 1) equal layers of an initial error, yielding ε Ψ, and a final third term that is the error induced by the previous layer, (k − 1)ε Ψ. More formally, assume that equation 15 holds for k − 1, implying that DISPLAYFORM20 Then, for k, we can write DISPLAYFORM21 Again, the first term we bound it in a straightforward manner using submultiplicativity of the operator norm DISPLAYFORM22 For the second term, since U G = 1 we focus on DISPLAYFORM23 The first term in equation 42 computes the propagation in the initial error caused by the first layer. Then, repeatedly applying ρ(u) − ρ(u) ≤ ρ(u − u) in analogy with k = 2 and using submultiplicativity, we get DISPLAYFORM24 The second term in equation 42 is the bounded by equation 38, since the first layer is exactly the same in this second term. Then, combining equation 43 with equation 38, yields DISPLAYFORM25 Overall, we get DISPLAYFORM26 which satisfies equation 15 for k. Finally, since this holds for k = 2, the proof is completed by induction. E PROOF OF COROLLARY 5.4From Theorem 5.3, we have DISPLAYFORM27 and, by definition (, Sec. 3 .1), DISPLAYFORM28 so that DISPLAYFORM29 Then, applying the inequality of Theorem 5.3, we get DISPLAYFORM30 Now, considering each term, such that DISPLAYFORM31 + m−1 k=0 2 3/2 k β 2 + (1 + β 2 +) (1 − β −)(1 − β 2 +) 3 d

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Stability of scattering transform representations of graph data to deformations of the underlying graph support.

We propose a differentiable family of "kaleidoscope matrices," prove that all structured matrices can be represented in this form, and use them to replace hand-crafted linear maps in deep learning models.