Question: represent the scientific article to find a short tldr

Question: We address the following question: How redundant is the parameterisation of ReLU networks? Specifically, we consider transformations of the weight space which leave the function implemented by the network intact. Two such transformations are known for feed-forward architectures: permutation of neurons within a layer, and positive scaling of all incoming weights of a neuron coupled with inverse scaling of its outgoing weights. In this work, we show for architectures with non-increasing widths that permutation and scaling are in fact the only function-preserving weight transformations. For any eligible architecture we give an explicit construction of a neural network such that any other network that implements the same function can be obtained from the original one by the application of permutations and rescaling. The proof relies on a geometric understanding of boundaries between linear regions of ReLU networks, and we hope the developed mathematical tools are of independent interest. Ever since its early successes, deep learning has been a puzzle for machine learning theorists. Multiple aspects of deep learning seem at first sight to contradict common sense: single-hidden-layer networks suffice to approximate any continuous function , yet in practice deeper is better; the loss surface is highly non-convex, yet it can be minimised by first-order methods; the capacity of the model class is immense, yet deep networks tend not to overfit . Recent investigations into these and other questions have emphasised the role of overparameterisation, or highly redundant function representation. It is now known that overparameterised networks enjoy both easier training (; ;), and better generalisation (; ;). However, the specific mechanism by which over-parameterisation operates is still largely a mystery. In this work, we study one particular aspect of over-parameterisation, namely the ability of neural networks to represent a target function in many different ways. In other words, we ask whether many different parameter configurations can give rise to the same function. Such a notion of parameterisation redundancy has so far remained unexplored, despite its potential connections to the structure of the loss landscape, as well as to the literature on neural network capacity in general. Specifically, we consider feed-forward ReLU networks, with weight matrices W 1,..., W L, and biases b 1,..., b L,. We study parameter transformations which preserve the output behaviour of the network h(z) = W L σ(W L−1 σ(. . . W 1 z + b 1 . . .) + b L−1 ) + b L for all inputs z in some domain Z. Two such transformations are known for feed-forward ReLU architectures: 1. Permutation of units (neurons) within a layer, i.e. for some permutation matrix P, 2. Positive scaling of all incoming weights of a unit coupled with inverse scaling of its outgoing weights. Applied to a whole layer, with potentially different scaling factors arranged into a diagonal matrix M, this can be written as Our main theorem applies to architectures with non-increasing widths, and shows that there are no other function-preserving parameter transformations besides permutation and scaling. Stated formally: Theorem 1. Consider a bounded open nonempty domain Z ⊆ R d0 and any architecture For this architecture, there exists a ReLU network h θ: Z → R, or equivalently a setting of the weights θ (W 1, b 1, . . ., W L, b L), such that for any'general' ReLU network h η: Z → R (with the same architecture) satisfying h θ (z) = h η (z) for all z ∈ Z, there exist permutation matrices P 1,... P L−1, and positive diagonal matrices M 1,..., M L−1, such that where η (W 1, b 1, . . ., W L, b L) are the parameters of h η. In the above,'general' networks is a class of networks meant to exclude degenerate cases. We give a more precise definition in Section 3; for now it suffices to note that almost all networks are general. The proof of the relies on a geometric understanding of prediction surfaces of ReLU networks. These surfaces are piece-wise linear functions, with non-differentiabilities or'folds' between linear regions. It turns out that folds carry a lot of information about the parameters of a network, so much in fact, that some networks are uniquely identified (up to permutation and scaling) by the function they implement. This is the main insight of the theorem. In the following sections, we introduce in more detail the concept of a fold-set, and describe its geometric structure for a subclass of ReLU networks. The paper culminates in a proof sketch of the main . The full proof, including proofs of intermediate , is included in the Appendix. The functional equivalence of neural networks is a well-researched topic in classical connectionist literature. The problem was first posed by , and soon resolved for feed-forward networks with the tanh activation function by , who showed that any smooth transformation of the weight space that preserves the function of all neural networks is necessarily a composition of permutations and sign flips. For the same class of networks, showed a somewhat stronger : knowledge of the input-output mapping of a neural network determines both its architecture and its weights, up to permutations and sign flips. Similar have been proven for single-layer networks with a saturating activation function such as sigmoid or RBF (Kůrková &), as well as single-layer recurrent networks with a smooth activation function (a; b). To the best of our knowledge, no such theoretical exist for networks with the ReLU activation, which is non-saturating, asymmetric and non-smooth. Broadly related is the recent work by and who study whether two neural networks (ReLU or otherwise) that are close in the functional space have parameterisations that are close in the weight space. This is called inverse stability. In contrast, we are interested in ReLU networks that are functionally identical, and ask about all their possible parameterisations. In terms of proof technique, our approach is based on the geometry of piece-wise linear functions, specifically the boundaries between linear regions. The intuition for this kind of analysis has previously been presented by and , and somewhat similar proof techniques to ours have been used by in the context of counting the number of linear decision regions. Finally, the sets of equivalent parametrisations can be viewed as symmetries in the weight space, with implications for optimisation. Multiple authors, including e.g.;; , have observed that the naive loss gradient is sensitive to reparametrisation by scaling, and proposed alternative, scaling-invariant optimisation procedures. We will omit the subscript θ when it is clear from the context. In this work, we restrict our attention to so-called general ReLU networks. Intuitively, a general network is one that satisfies a number of non-degeneracy properties, such as all weight matrices having non-zero entries and full rank, no two network units exactly cancelling each other out, etc. It can be shown 1 that almost all ReLU networks are general. In other words, a sufficient condition for a ReLU network to be general with probability one is that its weights are sampled from a distribution with a density. More formally, a general ReLU network is one that satisfies the following three conditions. 1. For any unit (l, i), the local optima of h 1:l i do not have value exactly zero. 2. For all k ≤ l and all diagonal matrices (I k, . . ., I l) with entries in {0, 1}, General networks are convenient to study, as they exclude many degenerate special cases. The second important class of ReLU networks are so-called transparent networks. Their significance as well as their name will become clear in the next section. For now, we state the definition. In words, we require that for any input, at least one unit on each layer is active. In this section we introduce the concept of fold-sets, which is key to our understanding of ReLU networks and their prediction surfaces. Since ReLU networks are piece-wise linear functions, a great deal about them is revealed by the boundaries between individual linear regions. A network's fold-set is simply the union of all these boundaries. More formally, if Z is an open set, and f: Z → R is any continuous, piece-wise linear function, we define the fold-set of f, denoted by F(f), as the set of all points at which f is non-differentiable. It turns out there is a class of networks whose fold-sets are especially easy to understand; these are the ones we have termed transparent. For transparent networks, we have the following characterisation of the fold-set (which also motivates the name 'transparent'). To appreciate the significance of the lemma, suppose we are given some transparent ReLU network function h and we want to infer its parameters. This lemma shows that the knowledge of the endto-end mapping h h 1:L in fact gives us information about the network's hidden units h 1:l i (hence 'transparent'). Moreover, this information is very explicit: we observe the units' zero-level sets, which in the case of a linear unit on a full-dimensional space already determines the unit's parameters up to scaling 2. Of course, dealing with piece-wise linearity and disambiguating the union into its constituent zero-level sets remains a challenge for upcoming sections. In this section, we provide a geometric description of fold-sets of transparent networks. Intuitively, the fold-sets look like the sets shown in Figure 1. The first-layer units of a network are linear, so the component i z | h 1:1 i (z) = 0 of the fold-set is a union of hyperplanes, illustrated by the blue lines in Figure 1. These hyperplanes partition the input space into a number of regions that each correspond to a different activation pattern. For a fixed activation pattern, or equivalently on each region, the second-layer units are linear, so their zero-level sets i z | h 1:2 i (z) = 0 are composed of piece-wise hyperplanes on the partition induced by the first-layer units. This is shown by the orange lines in Figure 1. More generally, the l th -layer zero-level sets i z | h 1:l i (z) = 0 consist of piece-wise hyperplanes on the partition induced by all lower-layer units. This yields a fold-set that looks like the set in the right pane of Figure 1, but potentially much more complicated. We now define these concepts more precisely. Piece-wise hyperplane. Let P be a partition of Z. We say H ⊆ Z is a piece-wise hyperplane with respect to partition P, if H is nonempty and there exist (w, b) = and P ∈ P such that H = {z ∈ P | w z + b = 0}. The final ingredient we will need to be able to reason about the parameterisation of ReLU networks is a more precise characterisation of the fold-set, in particular, the dependence structure between individual piece-wise hyperplanes. For example, consider the piece-wise linear surface in Figure 1 and compare it to the one in Figure 2. Suppose as before that the blue hyperplanes come from first-layer units, the orange hyperplanes come from second-layer units, and the black hyperplanes come from third-layer units. The difference between Figure 1 and Figure 2 is that if we observe only the fold-set, i.e. only the union of the zero-level sets over all layers (as shown in the right pane of Figure 2), then in the case of Figure 2, it is impossible to know which folds come from which layers. For instance, the blue folds and the orange folds could be assigned to the first and second layer almost arbitrarily; there is not enough information (i.e. intersection) in the fold-set to tell which is which. In contrast, the piece-wise linear surface in the right pane of Figure 1 could in principle be disambiguated into first-, second-and third-layer folds by the following procedure: 1. Take the largest possible union of hyperplanes that is a subset of the fold-set, and assign the hyperplanes to layer one. 2. Take all piece-wise hyperplanes with respect to the partition induced by the first-layer folds, and assign them to layer two. 3. Take all piece-wise hyperplanes with respect to the partition induced by the first-and second-layer folds, and assign them to layer three. This procedure is not guaranteed to assign all folds to their original layers because it ignores how piece-wise hyperplanes are connected; for example for the piece-wise linear surface in Figure 1, the procedure yields the layer assignment shown in Figure 3. However, it is sufficient for our purposes, and it is easier to work with mathematically. Formally, for a piece-wise linear surface S, we denote k S:= {S ⊆ S | S is a piece-wise linear surface of order at most k}. One can show 4 that k S is itself a piece-wise linear surface of order at most k, so one can think of k S as the'largest possible' subset of S that is a piece-wise linear surface of order at most k. For the piece-wise linear surface in Figure 3, the set 1 S consists of the blue hyperplanes, 2 S consists of the blue and the orange (piece-wise) hyperplanes, and 3 S = S. This definition allows us to uniquely decompose S into its piece-wise hyperplanes. Let S = l∈[κ],i∈[n l] H l i be any representation of S in terms of its piece-wise hyperplanes. We say the rep-. One can show 5 that such a representation exists and is unique up to subscript indexing. Importantly, it assigns a unique'layer' to each piece-wise hyperplane, its superscript. In other words, for architectures with non-increasing widths, there exists a ReLU network h such that knowledge of the input-output mapping h determines the network's parameters uniquely up to permutation and scaling. The idea behind the proof is as follows. Suppose we are given the function h. Then we also know its fold-set F(h), and if h is general and transparent, the fold-set is a piece-wise linear surface (by Lemma 2) of the form As we have mentioned earlier, this union of zero-level sets contains a lot of information about the network's parameters, provided we can disambiguate the union to obtain the zero-level sets of individual units. This disambiguation of the union is crucial, but is impossible in general. To see why, consider the first-layer units: given F(h), we want to identify i z | h 1:1 is a union of d 1 hyperplanes, we are done. In general however, F(h) may contain more than d 1 hyperplanes, such as for example in Figure 2. In such a setting it is impossible to tell which hyperplanes come from the first layer. The key insight here is the following: even though, say, a last-layer unit can create a fold that looks like a hyperplane, this hyperplane cannot have any dependencies, or descendants in the dependency graph. This follows from the fact that the layer is the last. More generally, if a (piece-wise) hyperplane has a chain of descendants of length m, it must come from a layer that is at least m layers below the last one. Formally, we have the following lemma. Lemma 3. Let h: Z → R be a general ReLU network. Denote S: Main proof idea. This lemma motivates the main idea of the proof. We explicitly construct a network h such that the dependency graph of its fold-set is well connected. More precisely, we ensure that each of the hyperplanes corresponding to first-layer units has a chain of descendants of length L − 2. This implies by Lemma 3 that the first-layer hyperplanes can be identified as such, using only the information contained in the fold-set. One can show that this is sufficient to recover the parameters W 1, b 1, up to permutation and scaling. To extend the argument to higher-layers, we then consider the truncated network h l:L. In h l:L, layer l becomes the first layer, and we apply the same reasoning as above to recover W l, b l. The next lemma shows that a network with a'well connected' dependency graph exists. In what follows, f | A denotes the restriction of a function f to a domain A, and Z for all i. One can show 6 that this implies the existence of scalars m 1,... We know that We have thus shown that there exists a permutation matrix P l ∈ R d l ×d l and a nonzero-entry diagonal matrix One can also show that the scalars m i are positive. For the inductive step, let l ∈ {2, . . ., L − 1}, and assume that there exist permutation matrices P 1,..., P l−1, and positive-entry diagonal matrices M 1,..., M l−1, such that holds up to layer l − 1. Then h. Since the end-to-end mappings are the same, h whereη:= (We therefore apply the same argument to h as we presented above for the case l = 1. We obtain that there exists a permutation matrix P l ∈ R d l ×d l and a positive-entry diagonal matrix Finally, consider the last layer. We know that h Discussion of assumptions. Most of the theorem's assumptions have their origin in Lemma 4. The reason we restrict the domain of h l:L to the interior of Z l−1 is that we want h l:L to be defined on an open set (otherwise fold-sets become unwieldy). For similar reasons, we study only architectures with non-increasing widths; otherwise int Z l−1 may be empty. We conjecture that the theorem does not hold for more general architectures. If it does, the proof will likely go beyond fold-sets. To guarantee transparency, our construction is such that for each input z ∈ Z and layer l ∈ [L − 1], either h 1:l 1 (z) > 0 or h 1:l 2 (z) > 0. Transparency could in principle be achieved with just a single unit, but it would have to be positive everywhere. This is why we impose d l ≥ 2. Guaranteeing transparency for the first layer (whose inputs are not constrained to the positive quadrant) also necessitates boundedness of Z. Boundedness can be lifted if we consider a slightly modified definition of transparency; proofs become more complicated though and we do not consider this crucial. Almost all of the proof carries over to the case of leaky ReLU activations (where σ is defined as σ(u) i = max {αu i, u i} for some small α > 0). The part that does not carry over is our proof that M l has only positive entries on the diagonal: In this part, we compare the slope of h l:L θ for inputs on the positive and negative side of a given ReLU unit, and notice that the negative-side slope is'singular' in the sense that some basis directions have zero magnitude. This particular argument does not work for the leaky ReLU, though we cannot rule out that a simple workaround exists. In this work, we have shown that for architectures with non-increasing widths, certain ReLU networks are almost uniquely identified by the function they implement. The suggests that the function-equivalence classes of ReLU networks are surprisingly small, i.e. there may be only little redundancy in the way ReLU networks are parameterised, contrary to what is commonly believed. This apparent contradiction could be explained in a number of ways: • It could be the case that even though exact equivalence classes are small, approximate equivalence is much easier to achieve. That is, it could be that h θ − h η ≤ is satisfied by a disproportionately larger class of parameters η than h θ − h η = 0. This issue is related to the so-called inverse stability of the realisation map of neural nets, which is not yet well understood. • Another possibility is that the kind of networks we consider in this paper is not representative of networks typically encountered in practice, i.e. it could be that'typical networks' do not have well connected dependency graphs, and are therefore not easily identifiable. • Finally, we have considered only architectures with non-increasing widths, whereas some previous theoretical work has assumed much wider intermediate layers compared to the input dimension. It is possible that parameterisation redundancy is much larger in such a regime compared to ours. However, gains from over-parameterisation have also been observed in practical settings with architectures not unlike those considered here. We consider these questions important directions for further research. We also hypothesise that our analysis could be extended to convolutional and recurrent networks, and to other piece-wise linear activation functions such as leaky ReLU. Definition A.1 (Partition). Let S ⊆ Z. We define the partition of Z induced by S, denoted P Z (S), as the set of connected components of Z \ S. Definition A.2 (Piece-wise hyperplane). Let P be a partition of Z. We say H ⊆ Z is a piece-wise hyperplane with respect to partition P, if H = ∅ and there exist (w, b) = and P ∈ P such that H = {z ∈ P | w z + b = 0}. Definition A.3 (Piece-wise linear surface / pwl. surface). A set S ⊆ Z is called a piece-wise linear surface on Z of order κ if it can be written as, and no number smaller than κ admits such a representation. Lemma A.1. If S 1, S 2 are piece-wise linear surfaces on Z of order k 1 and k 2, then S 1 ∪ S 2 is a piece-wise linear surface on Z of order at most max {k 1, k 2}. We can write H Given sets Z and S ⊆ Z, we introduce the notation (The dependence on Z is suppressed.) By Lemma A.1, i S is itself a pwl. surface on Z of order at most i. Lemma A.2. For i ≤ j and any set S, we have i j S = j i S = i S. Proof. We will need these definitions: j S = {S ⊆ S | S is a pwl. surface of order at most j}, i j S = {S ⊆ j S | S is a pwl. surface of order at most i}, surface of order at most j}. Consider first the equality j i S = i S. We know that j i S ⊆ i S because the square operator always yields a subset. At the same time, i S ⊆ j i S, because i S satisfies the condition for membership in. To prove the equality i j S = i S, we use the inclusion j S ⊆ S to deduce i j S ⊆ i S. Now let S ⊆ S be one of the sets under the union in, i.e. it is a pwl. surface of order at most i. Then it is also a pwl. surface of order at most j, implying S ⊆ j S. This means S is also one of the sets under the union in, proving that i S ⊆ i j S. Lemma A.3. Let Z and S ⊆ Z be sets. Then one can write k+1 S = k S ∪ i H i where H i are piece-wise hyperplanes wrt. P Z (k S). Proof. At the same time, is a pwl. surface of order at most k + 1 because k S is a pwl. surface of order at most k and H k+1 i can be decomposed into piece-wise hyperplanes wrt. Definition A.4 (Canonical representation of a pwl. surface). Let S be a pwl. surface on Z. The pwl. is a pwl. surface in canonical form, then κ is the order of S. Proof. Denote the order of S by λ. By the definition of order, λ ≤ κ, and S = λ S. Then, since It follows that κ = λ. Lemma A.5. Every pwl. surface has a canonical representation. Proof. The inclusion l∈[k],i∈[n l] H l i ⊆ k S holds for any representation. We will show the other inclusion by induction in the order of S. If S is order one, 1 S ⊆ S = i∈[n1] H 1 i holds for any representation and we are done. Now assume the lemma holds up to order κ − 1, and let S be order κ. Then by Lemma A.3, S = κ S = κ−1 S ∪ i H κ i, where H κ i are piece-wise hyperplanes wrt. P Z (κ−1 S). By the inductive assumption, κ−1 S has a canonical representation, Proof. Let k ∈ [κ]. Because both representations are canonical, we have where H k i and G k j are piece-wise hyperplanes wrt. where on both sides above we have a union of hyperplanes on an open set. The claim follows. Definition A.5 (Dependency graph of a pwl. surface). Let S be a piece-wise linear surface on Z, and let S = l∈[κ],i∈[n l] H l i be its canonical representation. We define the dependency graph of S as the directed graph that has the piece-wise hyperplanes H l i l,i as vertices, and has an edge We denote by σ the ReLU function: σ(u) i = max {0, u i} for i ∈ [dim(u)]. Definition A.6 (ReLU network). Let Z ⊆ R d0 with d 0 ≥ 2 be a nonempty open set, and let where For a ReLU network h θ: (We will omit the subscript θ when it is clear from the context.) We write f | A to denote the restriction of the function f to the domain A. (I 1, . . ., I L−1) is called an activation indicator if I l = diag(i l) ∈ R d l ×d l and i l ∈ {0, 1} d l for l ∈ [L − 1]. It is called non-trivial if i l = 0 for all l ∈ [L − 1] and non-trivial up to k if i l = 0 for all l ∈ [k]. and an activation indicator I, we introduce the notation (We will omit the argument θ when it is clear from the context.) These quantities characterise the different linear pieces implemented by the network's units. Also define I θ (z) (I Proof. Left as exercise. Definition A.8 (Fold-set). Let Z be an open set, and f: Z → R a continuous, piece-wise linear function. We define the fold-set of f, denoted by F(f), as the set of all points at which f is nondifferentiable. Definition A.9 (Positive / negative in a neighbourhood). Let Z be an open set. The function f: Z → R is positive (negative) in the neighbourhood of z ∈ Z if for any > 0 there exists z ∈ B (z) such that f (z) > 0 (f (z) < 0). Definition A.10 (Unit fold-set). Let h θ: Z → R be a ReLU network. We define the unit (l, i) fold-set of h θ, denoted F Proof. We will prove that if z satisfies any of the two conditions, then z ∈ F(σ •f), and if it violates both, then z ∈ F(σ • f) c. We begin with the latter implication. Let z be such that f (z) > 0 and z / ∈ F(f), i.e. f is differentiable at z. Since f is piece-wise linear, there exists > 0 such that all of B (z) lies inside a single linear region of f and f (B (z)) ⊆ (0, ∞]. Then, on B (z), the ReLU behaves like an identity, implying σ • f is differentiable at z, proving that z ∈ F(σ • f) c. Next, consider z such that f (z) = 0. For it to violate the second condition, there must exist a ball B (z) around z such that f (B (z)) ⊆ (−∞, 0]. (This is also true if f (z) < 0.) Then, on B (z), the ReLU behaves like a constant zero, implying that σ • f is differentiable at z. We now prove the other implication. If f (z) > 0 and z ∈ F(f), then there exists > 0 such that f (B (z)) ⊆ (0, ∞], which guarantees that the ReLU behaves like an identity on B (z). In this ball, we have σ If f (z) = 0 and f is positive in the neighbourhood of z, we distinguish several cases. If z / ∈ F(f), then there exists a ball B δ (z) on which f behaves linearly, i.e. σ(f (z)) = σ(w z + b), implying z ∈ F(σ • f). If z ∈ F(f) and, in addition, there exists a ball B δ (z) such that f (B δ (z)) ⊆ [0, ∞), then the ReLU behaves like an identity on B δ (z) and z ∈ F(σ • f). The final case is z ∈ F(f) such that f attains both positive and negative values in its neighbourhood. Since f is piece-wise linear, there exist p, n such that f (z + n) < 0 < f (z + p), and Lemma A.9. Let Z be an open set, and let f 1,..., f n: Z → R be continuous, piece-wise linear functions. For any w 1,..., Proof. Left as exercise. Lemma A.10. For all θ except a closed zero-measure set, for all activation indicators I and all k ≤ l. Proof. First, notice that is just a special case of with I l equal to the identity matrix. It therefore suffices to prove. To further simplify, we will prove the statement for a single fixed activation indicator I. Then if Θ(I) is the set of networks for which holds given I, and Θ(I) contains all networks except a closed zero-measure set, then also I Θ(I) contains all networks except a closed zero-measure set, proving the lemma. Let us hence fix I, and let k ∈ [L]. We proceed by induction. For the initial step, notice that the matrix I k W k is just W k with some rows replaced by zeroes. The rank of such a matrix is the same as the matrix obtained by removing the zero rows, which has size (rank(I k), d k−1 ). For all W k except a closed zero-measure set, this matrix has rank min {d k−1, rank(I k)}. For the inductive step, denoteW i:= I i W i · · · I k W k and We assume that rank(W i−1) = r i−1 and want to prove the same for i. Notice that for all W i except a closed zero-measure set, any r i rows of W i are linearly independent and their span intersects with ker(W i−1) only at 0. To see this, recall that by the inductive assumption, rank(W i−1) = r i−1, so ker(W i−1) has dimension d i−1 −r i−1. We can concatenate any r i -subset of rows of W i to the basis of ker(W i−1) to obtain a matrix of size (r i + d i−1 − r i−1, d i−1), which is a wide matrix, because r i ≤ r i−1. Hence, its rows are linearly independent for all W i except a closed zero-measure set. We now prove that rank(I i W iWi−1) = min rank(W i−1), rank(I i) r i. The "≤" direction is immediate. For the "≥" direction, we distinguish between two cases. If rank(I i) ≤ rank(W i−1), let v 1,... v ri be the (linearly independent) nonzero rows of I i W i. We want to show that v jW i−1 j are linearly independent, i.e. that I i W iWi−1 has at least r i linearly independent rows. If ri j=1 λ j v jW i−1 = 0, then ri j=1 λ j v j ∈ ker(W i−1), which by assumption implies λ j v j = 0. By the independence of {v j}, we obtain λ j = 0, i.e. v jW i−1 j are linearly independent, and rank(I i W iWi−1) = r i. If rank(I i) > rank(W i−1), we can reduce the problem to the case rank(I i) ≤ rank(W i−1) by observing that rank(I i W iWi−1) ≥ rank(J i W iWi−1) if J i equals I i only with some 1's replaced by 0's. We can thus take any such J i and apply the argument from the previous paragraph to obtain rank(Lemma A.11. For all θ except a closed zero-measure set, the following holds. Let (l, i), (k, j) be any units, let I be an activation indicator non-trivial up to l − 1, and let J be an activation indicator non-trivial up to k − 1, such that (l, i, I 1:l−1) = (k, j, J 1:k−1). Then, for all scalars c ∈ R, it holds that [w Proof. First, we exclude from consideration all θ = ( for some l, k, i, j, and some I non-trivial up to l − 1. Since for any fixed (l, k, i, j, I), the set of θ satisfying the above is the set of roots of a non-trivial polynomial in θ, it is zero-measure and closed. Because there are only finitely many configurations of (l, k, i, j, I), we have thus excluded a closed zero-measure set of parameters. We will denote its complement Θ *. From now on, we assume θ ∈ Θ *. Notice that the case c = 0 of the lemma is thus automatically satisfied, since w In the following, we can therefore assume c = 0 and treat (l, i, I) and (k, j, J) symmetrically. Denote by Θ ¬ ⊆ Θ * the set of parameters θ for which the lemma does not hold; we need to show that Θ ¬ is closed and zero-measure. We start by showing the latter property by contradiction. Suppose Θ ¬ is positive-measure. We know that for all θ ∈ Θ ¬, there exist triples (l, i, I), (k, j, J) as stated in the lemma, and a scalar c ∈ R such that [w Let C denote the set of all triplet-pairs ((l, i, I), (k, j, J)) satisfying the conditions of the lemma; then the previous statement can be written as Since C is finite, there exist ((l, i, I), (k, j, J)) ∈ C for which the set under the union (call it Θ) is positive-measure. We now consider two cases. If (l, i) = (k, j), then observe that Θ must contain some θ, θ such [w Notice that w Putting everything together, we have that which implies (c − c)v = [0, δ], and in particular w k j (θ, J) = 0. This contradicts the assumption that θ ∈ Θ * and completes the proof for the case (l, i) = (k, j). Definition A.11 (General ReLU network). A ReLU network is general if it satisfies Lemmas A.10, A.11 and A.12. All ReLU networks except a closed zero-measure set are general. Lemma A.13. If h is a general ReLU network, then F(h) follows from Lemma A.9. For the other inclusion, let z ∈ F(ȟ) such that I(z 1 ) =: I and I(z 2 ) =: J are independent of, and ∇ȟ. We consider three cases based on the (non-)triviality of I and J. First, suppose both I and J are trivial up to l − 1. Then by Lemma A.7, and similarly ∇ȟ 1:l−1 k (z 2 ) = 0, which contradicts ∇ȟ Hence, at least one of I, J, must be non-trivial up to l − 1. Second, say both I and J are non-trivial up to l − 1. From ∇ȟ follows that I 1:l−1 = J 1:l−1, we can therefore apply Lemma A.11 to (l, i, I) and (l, i, J). We obtain w Lemma A.14. Let h: Z → R be a ReLU network, and let Proof. We will abbreviate h λ:L | int Z λ−1 as h λ:L. Assume h is general. Then h λ:L clearly satisfies Lemma A.10, and for all (l, i), W l [i, :] = 0. Next, we prove that h λ:L satisfies Lemma A.11. Suppose this was not the case; then there exist units (λ − 1 + l, i), (λ − 1 + k, j), and non-trivial activation indicators I = (I λ, . . ., I λ−1+l), J = (J λ, . . ., I λ−1+k), with (l, i, I) = (k, j, J), and a scalar C ∈ R such that and Then for any non-trivial indicator (I 1, . . ., I λ−1) (J 1, . . ., J λ−1), we obtain by post-multiplying, and for all ι ∈ [λ − 1], The first equality means that w The last condition of generality is Lemma A.12. Suppose h λ:L does not satisfy the lemma. Then there exists a unit (l, i) such that i is positive and negative in the neighbourhood of z, i.e. there exists z ∈ int Z l−1 such that h λ:l i (z) = 0, and for some > 0 either h However, then there exists z ∈ Z such thatȟ 1:l−1 (z) = z, and for z we obtain h 1:l i (z) = 0, and by continuity, there is δ > 0 such that either h This contradicts the fact that h satisfies Lemma A.12. We have thus shown that if h is general, then h λ:L | int Z λ−1 is general. Finally, assume h is transparent, i.e. for all z ∈ Z and l Lemma A.15. a) For all ReLU networks h: In particular, for all general transparent ReLU networks, Proof. We give a proof of b) only. A proof of a) can be obtained by replacing some equalities by inclusions. We will prove by induction that F(h holds; we will prove the same statement for l + 1. By Lemma A.8 and Lemma A.13, we have Since It remains to show the reverse inclusion; we do so by contradiction.) is the partition of the input space into the linear regions of h 1:l+1 i, and Lemma A.15, the function h 1:l+1 i is also linear on the regions of, denote the slope and bias of h 1:l+1 i on P by w(P), b(P). Then The positivity condition guarantees that (w(P), is either an empty set or a piece-wise hyperplane. • the set F(h l:L | intZ l−1) is a pwl. surface whose dependency graph contains d l directed paths of length (L − 1 − l) with distinct starting vertices. We will show that this construction satisfies the lemma. The networks are transparent because of how we define W l 2: for all x ∈ X and l ∈ [L − 1], either h be its canonical representation, and let G denote its dependency graph. To find the required paths in G, we first identify some important vertices. For λ ∈ [L − l], denote This set is nonempty and open because P l+λ is nonempty and open. Next, for any unit (λ, ι), By the definition of W l+λ−1 1 and the fact that h l:l+λ−1 (Z We now show that G contains the edge H Then because of how {P l} l are defined, there existsz ∈ P l such that h l:l+λ−1 (z) =z, so it satisfies It follows thatz ∈ F λ,i). At the same time, the preimage (h l:l+λ−1) −1 (B (z)) is open by continuity, and containsz. So there exists a ball B (z) ⊆ P l such that all z ∈ B (z) satisfy (z) = 0 intersects the center of the half-ball, z. Therefore there exists a sequence of points {z n} ⊆ P l such that z n →z and We obtain thatz ∈ cl(F Proof. Because the representation is canonical, we have which implies H where P runs over the linear regions of h jι−1, but included in the same hyperplane. However, by Lemma A.11, no two piece-wise hyperplanes in S are included in a single hyperplane, so we get a contradiction. Hence, we obtain l 0 < l 1 < · · · < l m ≤ λ, which yields l 0 ≤ λ − m. Let h θ : X → R be a general ReLU network satisfying Lemma A.17, and let h η : X → R be any general ReLU network such that h θ (x) = h η (x) for all x ∈ X. Denote η (W 1, b 1, . . ., W L, b L). Then there exist permutation matrices P 1,... P L−1, and positiveentry diagonal matrices M 1,..., M L−1, such that Published as a conference paper at ICLR 2020 Finally, consider the last layer. We know that h

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We prove that there exist ReLU networks whose parameters are almost uniquely determined by the function they implement.

Represent the short summary for finding the corresponding scientific article (long)

This paper proves that skinny neural networks cannot approximate certain functions, no matter how deep they are.


Question: Represent the scientific article to find a short TLDR

Question: A general problem that received considerable recent attention is how to perform multiple tasks in the same network, maximizing both efficiency and prediction accuracy. A popular approach consists of a multi-branch architecture on top of a shared backbone, jointly trained on a weighted sum of losses. However, in many cases, the shared representation in non-optimal performance, mainly due to an interference between conflicting gradients of uncorrelated tasks. Recent approaches address this problem by a channel-wise modulation of the feature-maps along the shared backbone, with task specific vectors, manually or dynamically tuned. Taking this approach a step further, we propose a novel architecture which modulate the recognition network channel-wise, as well as spatial-wise, with an efficient top-down image-dependent computation scheme. Our architecture uses no task-specific branches, nor task specific modules. Instead, it uses a top-down modulation network that is shared between all of the tasks. We show the effectiveness of our scheme by achieving on par or better than alternative approaches on both correlated and uncorrelated sets of tasks. We also demonstrate our advantages in terms of model size, the addition of novel tasks and interpretability. Code will be released. The goal of multi-task learning is to improve the learning efficiency and increase the prediction accuracy of multiple tasks learned and performed together in a shared network. Over the years, several types of architectures have been proposed to combine multiple tasks training and evaluation. Most current schemes assume task-specific branches, on top of a shared backbone (Figure 1a) and use a weighted sum of tasks losses, fixed or dynamically tuned, to train them (; ;). Having a shared representation is more efficient from the standpoint of memory and sample complexity and can also be beneficial in cases where the tasks are correlated to each other . However, in many other cases, the shared representation can also in worse performance due to the limited capacity of the shared backbone and interference between conflicting gradients of uncorrelated tasks . The performance of the multi-branch architecture is highly dependent on the relative losses weights and the task correlations, and cannot be easily determined without a "trial and error" phase search . Another type of architecture that has been recently proposed uses task specific modules, integrated along a feed-forward backbone and producing task-specific vectors to modulate the feature-maps along it (Figure 1b). Here, both training and evaluation use a single tasking paradigm: executing one task at a time, rather than getting all the task responses in a single forward pass of the network. A possible disadvantage of using task-specific modules and of using a fixed number of branches, is that it may become difficult to add additional tasks at a later time during the system life-time. Modulation-based architectures have been also proposed by and (Figure 1c). However, all of these works modulate the recognition network channel-wise, using the same modulation vector for all the spatial dimension of the feature-maps. We propose a new type of architecture with no branching, which performs single task at a time but with no task-specific modules (Figure 1d). The core component of our approach is a top-down (TD) (a) (b) (c) (d) Figure 1: (a) Multi branched architecture, task specific branches on a top of a shared backbone, induces capacity and destructive interference problems, force careful tuning. Recently proposed architectures: (b) using tasks specific modules and (c) using channel-wise modulation modules. (d) Our architecture: a top-down image-aware full tensor modulation network with no task specific modules. modulation network, which carries the task information in combination with the image information, obtained from a first bottom-up (BU1) network, and modulates a second bottom-up (BU2) network common for all the tasks. In our approach, the modulation is channel-wise as well as spatial-wise (a full tensor modulation), calculated sequentially along the TD stream. This allows us, for example, to modulate only specific spatial locations of the image depending on the current task, and get interpretability properties by visualizing the activations in the lowest feature-map of the TD stream. In contrast to previous works, our modulation mechanism is also "image-aware" in the sense that information from the image, extracted by the BU1 stream, is accumulated by the TD stream, and affects the modulation process. The main differences between our approach and previous approaches are the following: First, as mentioned, our approach does not use multiple branches or task-specific modules. We can scale the number of tasks with no additional layers. Second, our modulation scheme includes a spatial component, which allows attention to specific locations in the image, as illustrated in figure 2a for the Multi-MNIST tasks . Third, the modulation in our scheme is also image dependent and can modulate regions of the image based on their content rather than location (relevant examples are demonstrated in figures 2b and 2c). We empirically evaluated the proposed approach on three different datasets. First, we demonstrated on par accuracies with the single task baseline on an uncorrelated set of tasks with MultiMNIST while using less parameters. Second, we examined the case of correlated tasks and outperformed all baselines on the CLEVR dataset. Third, we scaled the number of tasks and demonstrated our inherent attention mechanism on the CUB200 dataset. The choice of datasets includes cases where the tasks are uncorrelated (Multi-MNIST) and cases where the tasks are relatively correlated (CLEVR and CUB200). The demonstrate that our proposed scheme can successfully handle both cases and shows distinct advantages over the channel-wise modulation approach. Our work draw ideas from the following research lines: Multiple Task Learning (MTL) Multi task learning has been used in machine learning well before the revival of deep networks . The success of deep neural networks in single task performance (e.g. in classification, detection and segmentation) has renewed the interests of the computer vision community in the field (; ;). Although our primary application area is computer vision, multi task learning has also many application in other fields like natural language processing and even across modalities . We further refer the interested reader to a review that summarizes recent work in the field . Over the years, several types of architectures have been proposed in computer vision to combine the training and evaluation of multiple tasks. First works used several duplications (as many as the tasks) of the base network, with connections between them to pass useful information between the tasks . These works do not share computations and cannot scale with the tasks. More recent architectures, which are in common practice these days, assume task-specific branches on a top of a shared backbone, and use a weighted sum of losses to train them. The joint learning of several tasks has proven to be beneficial in several cases but can also decrease the of some of the tasks due to a limited network capacity, uncorrelated gradients from the different tasks (sometimes called destructive inference) and different learning rates . A naive implementation of multi-task learning requires careful calibration of relative losses of the different tasks. To address these problem several methods have been proposed: "Grad norm" dynamically tunes gradients magnitudes over time, to obtain similar learning rates of the different tasks. uses a joint likelihood formulation to derive task weights based on the intrinsic uncertainty in each task. applies an adaptive weighting of the different tasks, to force a pareto optimal solution to the multi task problem. Along an orthogonal line of research, other works suggested to add task-specific modules to be activated or deactivated during training and evaluation, depending on the task at hand. Liu et al. (2019b) suggests task specific attention networks in parallel to a shared recognition network. suggests adding several types of low-weight task-specific modules (e.g. residual convolutional layers, squeeze and excitation (SE) blocks and batch normalization layers) along the recognition network. Note that the SE block essentially creates a modulation vector, to be channelwise multiplied with a feature-map. Modulation vectors have been further used in for a recognition application, in for continual learning applications and in for a retrieval application and proved to decrease the destructive interference between tasks and the effect of the catastrophic forgetting phenomena. Our design, in contrast, does not use multi-branch architecture, nor task-specific modules. Our network is fully-shared between the different tasks. Compared to , we modulate the feature-maps in the recognition network channel-wise as well as spatial-wise, depending on both the task and the specific image at hand. Neuroscience research provides evidence for a top-down context, feedback and lateral processing in the primate visual pathway (; ; ; ; Piëch et al., 2013;) where top-down signals modulate the neural activity of neurons in lower-order sensory or motor areas based on the current goals. This may involve enhancement of task-relevant representations or suppression for task-irrelevant representations. This mechanism underlies humans ability to focus attention on task-relevant stimuli and ignore irrelevant distractions (; Piëch et al., 2013;). In this work, consistent with this general scheme, we suggest a model that uses top-down modulation in the scope of multi-task learning. Top down modulation networks with feedback, implemented as conv-nets, have been suggested by the computer vision community for some high level tasks (e.g. Figure 3: Several types of modulation modules, the trainable parameters are illustrated in yellow. (d) For simplicity we show only one modulation stage, where X is the input tensor to be modulated and Y is the output tensor. (a) task-dependent vector-modulation architecture , the modulation vectors are switched by the task and explicitly being optimized. (b) Hypothetical extension of (a) to spatial-wise modulation tensors, cannot be done in practice due to the huge number of parameters to optimize. (c) Our approach: We optimize the parameters in the convolutional layers along the top-down network and use the created featuremaps as the modulation tensors. re-classification , keypoints detection , crowd counting , curriculum learning etc.) and here we apply them to multi-task learning applications. We will first describe the task-dependent vector-modulation mechanism, as proposed in , illustrated in figure 3a, and then describe our architecture (figure 3c) in detail. A vector-modulation (a channel-wise modulation) of a given tensor X by a vector z is defined as the product of the elements of the vector z and the corresponding channels of the tensor X. Each element in the output tensor Y is calculated by: where X is the tensor to be modulated and Y is the modulated tensor, both in the form (H × W × C) where H, W, are the spatial dimensions of the tensors and C their channel dimension. The vector z ∈ R C has dimension equal to the number of channels of X, Y. x, y, ch are the column, row, and channel number and indicate a specific element in a tensor. In training the network, the elements of z are considered as parameters and are directly optimized (C additional parameters). In the scope of Multi Task Learning, where several tasks co-exist, the network switches on the fly between several modulation vectors z where K is the number of tasks, see figure 3a for illustration. The network performs one task at a time, and the modulated tensor Y depends on the selected task. This vector-modulation module has been used in separately for every stage in the recognition network (with additional CK parameters in every stage). Two limitations of this module are that it ignores the spatial dimensions of the image, and the lack of information from the image itself. The possible use of the same strategy to explicitly optimize spatial-aware modulation tensors (figure 3b) was discussed in but was deemed infeasible due to the large amount of added parameters (HWCK additional parameters in every stage). Our method addresses both of these issues in an efficient manner and demonstrates better accuracy, showing that spatial-wise modulation and the use of image information are beneficial to many kinds of tasks. A tensor modulation is defined by: Where Z ∈ R H×W ×C is a modulation tensor. To avoid the infeasible computation of directly optimizing Z, we propose the use of created featuremaps as the modulation tensors. Practically, we use a dedicated top-down (TD) convolutional stream, shared between the tasks, to create the modulation featuremaps, and optimize the weights of the convolutional layers instead of directly optimizing the modulation tensors (figure 3c). The number of added parameters in this case depends on the precise architecture of the TD stream but can be approximately estimated by 3 × 3 × C 2 parameters for each convolutional layer (several convolutional layers may be used in one stage). Avoiding the dependency of the number of added parameters on H, W and K allows us to apply the proposed architecture to large images and to scale the number of tasks, as illustrated our experiments. A gated modulation module with a residual connection We further define a gated modulation module with a residual connection as: where the modulation tensor Z is gated with a sigmoid or a tanh function before the multiplication and then added to the input tensor X through a residual connection. The residual gated modulation with Z is equivalent to the modulation withZ = (1 + σ(Z)). Motivated by our ablation studies 4.3.2, unless stated otherwise, we use the gated modulation as defined in Eq. 3 in all of our experiments. For simplicity we denote this operation by the symbol ⊗. An illustration of our network design is shown in figure 1d. In our design, a bottom-up (BU2) recognition network is modulated by a top-down (TD) modulation stream. The inputs to the TD stream are the current task k, and the feature-maps along the first bottom-up stream (BU1, where BU1 and BU2 share the same weights), added to the TD stream via lateral connections. The outputs of the TD stream are its feature-maps, which sequentially modulate the tensors along the recognition network (BU2). Figure 3c illustrate our architecture for one modulation step. Auxiliary losses Our architecture can be naturally decomposed into three sub-networks (BU1, TD, BU2), allowing the structural advantage of adding auxiliary losses at the end of the BU1 or TD streams. This possibility is application-dependent. In the scope of multi-task learning, the TD auxiliary loss might be beneficial because it allows the use of spatial information in a taskdependent manner. This issue is further discussed in section 4.3.4 where we demonstrate the use of a localization loss in the last TD featuremap. Applying the localization loss in train time allows us to obtain an attention map in inference time, which illustrates the relative weights assigned by the network to different locations in the image. We validate our approch on three different datasets: MultiMNIST MultiMNIST is a multi-task learning version of the MNIST dataset in which multiple MNIST images are placed on the same image. We use 2, 3 and 4 classes experiments built as suggested by. Several examples are demonstrated in Figure 2a. In the 2-classes experiment the tasks are: classifying the digit on the top-left (task-LU) and classifying the digit on the bottom-right (task-RL). We corespondently add (task-LL) and (task-RU) for classifying the digits on the bottom-left and top-right on the 3 and 4-classes experiments. The digits are independently chosen and the tasks are considered to be uncorrelated. We use 60K examples and directly apply LeNet as the underlying backbone in our experiments. CLEVR CLEVR is a synthetic dataset, consists of 70K training images and 15K validation images, mainly used as a diagnostic dataset for VQA. The dataset includes images of 3D primitives, with multiple attributes (shape, size, color and material) and a set of corresponding (questionanswer) tuples. We followed the work Liu et al. (2019a), which suggested to use CLEVR not as a VQA dataset, but rather as a referring expression dataset, and further adapt it to a multi-task learning methodology. The tasks in our setup consist of 4 questions ("Are there exactly two cylinders in the image?", "Is there a cube right to a sphere?", "Is there a red sphere?" and "Is the leftmost sphere in the image large?"), arbitrarily chosen, with various compositionary properties. CUB200 is a fine grained recognition dataset that provides 11,788 bird images (equally divided for training and testing) over 200 bird species with 312 binary attribute annotations, most of them referring to the colors of specific birds' parts. In contrast to other work that used all of the 312 attributes as a yes/no question, we re-organized the attributes as a multi-task problem of 12 tasks (for 12 annotated bird's parts) each with 16 classes (the annotated colors + an unknown class) and train using a multi-class cross-entropy loss. To demonstrate our interpretability capability, we further used the parts' location, annotated by a single point to each seen part, as an auxiliary target at the end of the TD stream. architecture We use LeNet, VGG-11 and resnet-18 as our backbone BU architectures for the Multi-MNIST, CLEVR and CUB-200 experiments correspondingly. Each of the backbones has been divided to two parts; a first part that consists mainly of the convolutional layers of the backbone and a second part with the fully connected layers (the classifier). In our architecture, both BU streams consist of the first part of the backbone and share their weights. The TD stream, unless specified otherwise, is a replica of the BU stream, in terms of layers structure and number of channels, combined with bilinear upsampling layers. The classifier is only attached to the BU2 stream. Information is passed between the BU1, TD and BU2 streams using lateral connections implemented as 1x1 convolutions. A task embedding layer (a fully connected layer) is added on the top of the TD stream. See an illustration of the full scheme in figure 1d and a detailed architecture description of the Multi-MNIST experiments in the supplementary materials. baselines We compare our method both to a "single task" approach, where each task is independently solved and to a "uniform scaling" approach, where a uniformly weighted sum of the individual losses is being minimized. We have also compared our architecture to "ch-mod", a channelwised vector modulation architecture and to a MOO (multi objective optimization approach) where the weights of loss items are dynamically tuned as suggested by. We use the Multi-MNIST dataset to demonstrate our performance in case of uncorrelated tasks for 2, 3, and 4 tasks recognition problems with no additional hardware. All models trained using a standard LeNet architecture. We used a batch size of 512 images trained on 1 GPU with learning rate of 1e −3 using the Adam optimizer. Training curves are presented in figure []. Figure 4b visualize the performance profile of the 2-classes experiment as a scatter plot of accuracies on task-LU and task-RL for the single task approach (vertical and horizontal lines correspondingly) and the multi-branched approach for several manually tuned loss weights (the blue dots). The scatter plot demonstrate a capacity problem, where better accuracies (above a certain limit) in one task cannot be achieved without being reflected as lower accuracies on another task. Our are marked as a red star, showing better accuracies than the single-task case with much less parameters. Table 1 summarizes our on the Multi-MNIST experiment while sequentially enlarging the number of tasks. We show mean ±std based on 5 experiments for each row. Our method achieves better than the single-task baseline while using much less parameters (the third column shows the number of parameters as a multiplier of the number of parameter in a standard Lenet architecture). Other approaches, including the channel-wise modulation approach, achieve lower accuracy rates. Scaling the number of tasks keeps the accuracy gap almost without additional parameters. We further conducted ablation studies on Multi-MNIST, to examine several aspects of our proposed architecture. Table 2 shows the ablation , analyzed as follows: Using spatial-wise and image-aware modulation modules. Our experiments show that extending the existing channel-wise modulation architecture to an image-aware spatial-wise modulation architecture improves the . Table 2a quantify the improvement in the compared to the channel-wise modulation baseline (, first row in the table). We show mean ± std based on 5 repetitions of the full training pipeline for each row. Using a channel-wise image-aware modulation architecture by sequentially integrating information from the featuremaps in BU1 (second row) improves the accuracies by ∼ 0.4%. Using a spatial-wise modulation without using the information from BU1 stream (third row) improve the accuracies by ∼ 2.7%. Our approach, that uses both image-aware and spatial-wise modulation, improves the accuracies by a solid gap of ∼ 3.3%. Number of channels in the TD stream. Table 2b compares the accuracies of our proposed architecture (first line, where the TD stream is a replica of the BU stream which has 1, 10 and 20 channels in its feature-maps) with cheaper architectures which use a reduced number of channels along the layers in the TD stream. Our experiments show a trend line (the accuracies decrease when the number of channels in the TD stream decreases) and that optimizing the number of channels along the TD stream in terms of efficiency-accuracies tradeoff can be done (demonstrated by the second row in the table where higher accuracy achieved while using less parameters). Connectivity type. Our architecture uses two sets of lateral connections; the first set passes information from the BU1 stream to the TD stream, and the second passes information from the TD stream to the BU2 stream. Table 2c compares the accuracy of our proposed architecture when using different connectivity types to the TD stream (first column) and to the BU2 stream (second column). Here + is an addition connectivity, × is a multiplication connectivity and ⊗ is a gated modulation with residual connection as described in Equation 3. The table shows higher accuracy when using addition connectivity along the TD stream and a small preference for the gated mod- Table 3: Performance on CLEVR, higher is better. Our approach yields better accuracies also on correlated set of tasks with no additional hardware as tasks are added. Better accuracies are demonstrated both compared to the single task and uniform scaling approaches while using less parameters. ulation connectivity over the multiplication connectivity along the BU2 stream. To better compare between the two connectivity types we carried 5 experiments and report mean ± std. We used the gated modulation connectivity type in all our experiments due to its slightly higher . Auxiliary losses. Our architecture, although usually uses only one classification loss at the end of the BU2 stream, can be easily adapted to integrate two auxiliary losses, one at the end of the BU1 stream (same classification loss) and the other on the image plane at the end of the TD stream (segmentation loss). Table 2d shows no additional improvement when using these auxiliary losses on the Multi-MNIST experiment. Note that a TD auxiliary segmentation loss (here, a binary cross entropy loss between the predicted digit and a zero-one map of the target digit) can also be used to add interpretability to our scheme. Examples are shown in the CUB200 experiment, section 4.3.4. We used the CLEVR dataset to show our performance in case of correlated tasks (the questions on CLEVR are correlated) and to demonstrate our ability to enlarge the number of tasks with no extra hardware while keeping the targets accuracies. Our are summarized in Table 3. We trained all models using a VGG-11 architecture but decreased the number of channels in the output of the last convolutional layer from 512 to 128 to allow training with larger batch size. A detailed analysis of the number of parameters can be found in the supplementary materials. We used a batch size of 128 images trained on 2 GPUs with learning rate of 1e −4 using the Adam optimizer. Table 3 shows that our are better than both single task and uniform scaling approach while using much less parameters (the third columns shows the number of parameters of each architecture as a multiplier of the number of parameters in a single task VGG-11 backbone). Here, the channelwise modulation approach uses the smallest number of parameters but also gets the worst . The table also shows that scaling the number of tasks (with no additional hardware) is not only feasible but also may improve the of each task separately. We further note that we used a TD layers that are a replica of the VGG-11 BU layers. Further reducing the number of parameters by decreasing the channel dimensions in the TD stream can be easily done but is not our main scope in this work. We used the CUB-200 dataset to further demonstrate our performance on correlated tasks in realworld images, scaling the number of tasks and using another type of backbone architecture (a Resnet backbone). In contrast to previous experiments, we did not aim at reducing the number of parameters (since we are using a Resnet backbone); rather we demonstrate better performance, and our built-in capability to visualize the attention maps at the end of the TD stream. We trained all models using a Resnet-18 architecture. We used a batch size of 128 images trained on 2 GPUs with learning rate of 1e −4 using the Adam optimizer for 200 epochs. While training our architecture we add an auxiliary loss at the end of the TD stream. The target in this case is a one-hot 224x224 mask, where only a single pixel is labeled as foreground, blurred by a Gaussian kernel with a standard deviation of 3 pixels. Training one task at a time, we minimize the cross-entropy loss over the 224x224 image at the end of the TD softmax output (which encourages a small detected area) for each visible ground-truth annotated task/part. For a fair comparison, we also compared our to the channel-wised modulation architecture trained with the same localization auxiliary loss (on the coarse map at the end of the BU2 stream, fifth line in the table). Figures 4c and 4d demonstrate the attention maps produced by our architecture in inference time. Figure 4c is an example where the predicted mask is well localized on the crown of the bird (the task) and the color is correctly predicted. Figure 4d demonstrate an error case where the breast of the bird is not well localized by the mask and as a consequence the color is wrongly predicted. More examples of interest are shown in the supplementary materials. Our quantitative are summarized in Table 4. The show better accuracy of our scheme compared to all baselines. We specifically show better accuracy compared to the channel-wise modulation scheme, indicating the preference of our image-dependent spatial-wise modulation process on the CUB200 database. We proposed a novel architecture for multi-task learning using a top-down modulation network. Compared with current approaches, our scheme does not use task-dependent branches or taskdependent modules, and the modulation process is executed spatial-wise as well as channel-wise, guided by the task and by the information from the image itself. We tested our network on three different datasets, achieving on par or better accuracies on both correlated and uncorrelated sets of tasks. We have also demonstrated inherent advantages of our scheme: adding tasks with no extra hardware that in a decrease in the total number of parameters while scaling the number of tasks, and allowing interpretability by pointing to relevant image locations. More generally, multiple-task learning algorithms are likely to become increasingly relevant, since general vision systems need to deal with a broad range of tasks, and executing them efficiently in a single network is still an open problem. In future work we plan to adapt our described architecture to a wider range of applications (e.g. segmentation, images generation) and examine possible combinations of approaches such as combining partial-branching strategy with our TD approach. We also plan to study additional aspects of multi-task learning such as scaling the number of tasks and tackling the catastrophic forgetting problem. For the MultiMNIST experiments, we use an architecture based on LeNet ). We followed and use two 5x5 convolutional layers and one fully-connected layer as the shared backbone and two other fully-connected layers as task specific branches for the multi-branched architecture (See figure 5a for details). Our architecture is illustrated in figure 5b. We use the shared backbone and a single branch as the recognition network (BU2). On the TD stream we use an embedding layer followed by two 5x5 convolutional layers. BU1 share the same weights with BU2. the three subnetworks are combined together using lateral connections, implemented as 1x1 convolutions. The networks for the CLEVR and CUB200 experiments where similarly implemented using VGG-11 and ResNet-18 backbones correspondently. The exact number of parameters used by the Multi-MNIST and by the CLEVR architectures are summarized in table 5. To demonstrate our interpretability capabilities we trained our proposed network with an auxiliary localization cross entropy loss on the last layer of the TD stream (details in section 4.3.4). Here we present several more examples of interest we did not include in the main text. More qualitative examples to demonstrate our ability to identify the relevant regions that most affected the network prediction. In all of these images the target part (the task, shown in the upper part of each image), is precisely localized and the prediction (shown in the lower part of each image) follows the ground truth. Best viewed in color while zoomed in. Figure 7: Error cases. Left images demonstrate good examples, counted as failure cases due to annotations errors. Our network successfully localize the asked part and correctly predict its color. Right images demonstrate bad localization examples. Ground truth classes were still predicted, with a very high score, maybe due to the correlated nature of the tasks. Best viewed in color while zoomed in.

Represent the TLDR for finding the corresponding scientific article.

We propose a top-down modulation network for multi-task learning applications with several advantages over current schemes.

Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise in the knowledge distillation framework and studied their effect on generalization and robustness.


Question: Represent the scientific article to find a short TLDR

Question: Clustering algorithms have wide applications and play an important role in data analysis fields including time series data analysis. The performance of a clustering algorithm depends on the features extracted from the data. However, in time series analysis, there has been a problem that the conventional methods based on the signal shape are unstable for phase shift, amplitude and signal length variations. In this paper, we propose a new clustering algorithm focused on the dynamical system aspect of the signal using recurrent neural network and variational Bayes method. Our experiments show that our proposed algorithm has a robustness against above variations and boost the classification performance. The rapid progress of IoT technology has brought huge data in wide fields such as traffic, industries, medical research and so on. Most of these data are gathered continuously and accumulated as time series data, and the extraction of features from a time series have been studied intensively in recent years. The difficulty of time series analysis is the variation of the signal in time which gives rise to phase shift, compress/stretch and length variation. Many methods have been proposed to solve these problems. Dynamic Time Warping (DTW) was designed to measure the distance between warping signals . This method solved the compress/stretch problem by applying a dynamic planning method. Fourier transfer or wavelet transfer can extract the features based on the frequency components of signals. The phase shift independent features are obtained by calculating the power spectrum of the transform . In recent years, the recurrent neural network (RNN), which has recursive neural network structure, has been widely used in time series analysis (; 1991). This recursive network structure makes it possible to retain the past information of time series. Furthermore, this architecture enables us to apply this algorithm to signals with different lengths. Although the methods mentioned above are effective solutions for the compress/stretch, phase shift and signal length variation issues respectively, little has been studied about these problems comprehensively. Let us turn our attention to feature extraction again. Unsupervised learning using a neural network architecture autoencoder (AE) has been studied as a feature extraction method (; ;). AE using RNN structure (RNN-AE) has also been proposed and it has been applied to real data such as driving data and others. RNN-AE can be also interpreted as the discrete dynamical system: chaotic behavior and the deterrent method have been studied from this point of view (; Laurent & von). In this paper, we propose a new clustering algorithm for feature extraction focused on the dynamical system aspect of RNN-AE. In order to achieve this, we employed a multi-decoder autoencoder with multiple decoders to describe different dynamical systems. We also applied the variational Bayes method (; ;) as the clustering algorithm. This paper is composed as follows: in Section 4, we explain AE from a dynamical system view, then we define our model and from this, derive its learning algorithm. In Section 5, we describe the application of our algorithm to an actual time series to show its robustness, including running two experiments using periodic data and driving data. Finally we summarize our study and describe our future work in Section 7. A lot of excellent clustering/representation algorithms of data using AE have been studied so far . integrated the distance between data and centroids into an objective function to obtain a cluster structure in the encoded data space. proposed a generative model based on the variational autoencoder (VAE) with a clustering structure as a prior distribution. achieved a high separability clustering by adding a regularization term for the orthogonality and balanced clusters of the encoded data. These, however, are regularization methods of the objective function, and focused on only the distribution of the encoded data. They did not give the clustering policy based on the decoder structure, namely, the reconstruction process of the data. From dynamical system point of view, one decoder of RNN-AE corresponds to a single dynamics in the space of latent representation. Hence, it is natural to equip RNN-AE with multiple decoders to implement multiple dynamics. Such an extension of RNN-AE, however, has yet to be proposed in related works to the best of our knowledge. 3 RECURRENT NEURAL NETWORK AND DYNAMICAL SYSTEM 3.1 RECURRENT NEURAL NETWORK USING UNITARY MATRIX RNN is a neural network designed for time series data. The architecture of the main unit is called cell, and mathematical expressions are shown in Fig. 1 and Eq.. Suppose we are given a time series, where D denotes data dimension. RNN, unlike the usual feed-forward neural network, operates the same transform matrix to the hidden valuable recursively, where σ(·) is an activation function and z t, h t, b ∈ R L. This recursive architecture makes it possible to handle signals with different lengths, although it is vulnerable to the vanishing gradient problem as with the deep neural network (DNN) (; 1991). Long short-term memory (LSTM) and gated recurrent unit (GRU) are widely known solutions to this problem (; ;). These methods have the extra mechanism called a gate structure to control output scaling and retaining/forgetting of the signal information. Though this mechanism works effectively in many application fields , the architecture of network is relatively complicated. As an alternative simpler method to solve this problem, the algorithm using a unitary matrix as the transfer matrix V was proposed in recent years. Since the unitary matrix does not change the norm of the variable vector, we avoid the vanishing gradient problem. In addition, the network architecture remains unchanged from the original RNN. In this paper, we focus on the dynamical system aspect of the original RNN. We employ the unitary matrix type RNN to take advantage of this dynamical system structure. However, to implement the above method, we need to find the transform matrix V in the space of unitary matrices is the set of complex-valued general linear matrices with size L × L and * means the adjoint matrix. Several methods to find the transform matrix from the U has been reported so far (; ; ; ;). Here, we adopt the method proposed by. The architecture of AE using RNN is shown in Fig. 2. AE is composed of an encoder unit and a decoder unit. The parameters where X is the input data and X dec is the decoded data. The input data is recovered from only the encoded signal h using the matrix (V dec, U dec), therefore h is considered as the essential information of the input signal. When focusing on the transformation Figure 2: Architecture of RNN Autoencoder of the hidden variable, this recursive operation has the same structure of a discrete dynamical system expression as described in the following equation: where f is given by Eq.. From this point of view, we can understand that RNN describes the universal dynamical system structure which is common to the all input signals. In this section, we will give the architecture of the Multi-Decoder RNN AE (MDRA) and its learning algorithm. As we discussed in the previous section, RNN can extract the dynamical system characteristics of the time series. In the case of the original RNN, the model expresses just one dynamical system, hence all input data are recovered from the encoded h by the same recovery rule. Therefore h is usually used as the feature value of the input data. In contrast, in this paper, we focus on the transformation rule itself. For this purpose, we propose MDRA which has multiple decoders to extract various dynamical system features. The architecture of MDRA is shown in Fig. 3. Let us put We will derive the learning algorithm to optimize the whole set of parameters W in the following section. We applied a clustering method to derive the learning algorithm of MDRA. Many clustering algorithms have been proposed: here we employ the variational Bayes (VB) method, because the VB method enabled us to adjust the number of clusters by tuning the hyperparameters of a prior distribution. We first define free energy, which is negative marginal log-likelihood, by the following equation, where X is data tensor defined in Section 3 and W is parameter tensor of MDRA defined above. is the set of latent variables each of which means an allocation for a decoder. That is, y n = (y n1, · · ·, y nK) T ∈ R K, where y nk = 1 if X n is allocated to the k-th decoder is the probability density function representation of MDRA parametrized by tensor W, p(α) and p(β) are its prior distributions for a probability vector α = (α 1, · · ·, α K) and a precision parameter β > 0. We applied the Gaussian mixture model as our probabilistic model. Hence p(α) and p(β) were given by Dirichlet and gamma distributions respectively which are the conjugate prior distributions of multinomial and Gaussian distributions. These specific distributions are given as follows: Here, θ 0 > 0, ν 0 > 0 and λ 0 > 0 are hyperparameters and g(h n |W k dec) = X n dec,k denotes decoder mapping of RNN from the encoded n-th data h n, H = (h 1, · · ·, h N) and T n D is the total signal dimension of input signal X n including dimension of input data. To apply the variational Bayes algorithm, we the derive the upper bound of the free energy by applying Jensen's inequality, where D KL (·∥·) is the Kullback−Leibler divergence. The upper boundF (X|W) is called the variational free energy or (negated) evidence lower bound (ELBO). The variational free energy is minimized using the variational Bayes method under the fixed parameters W. Furthermore, it is also minimized with respect to the parameters W by applying the RNN learning algorithm to the second term ofF (X|W), In this section, we derive the variational Bayes algorithm for MDRA to minimize the variational free energy. We show the outline of the derivation below (for a detailed derivation, see Appendix A). The general formula of the variational Bayes algorithm is given by. By applying the above equations to the above probabilistic models, we obtained the specific algorithm shown in Appendix A.1. Then we minimize the following term using RNN algorithm: From the above discussion, we finally obtained the following MDRA algorithm. Fig. 4 Calculate R = (r nk) by the algorithm VB part of MDRA (Algorithm 2). until the difference of variational free energyF (X|W) < Threshold We first examined the basic performance of our algorithm using periodic signals. Periodic signals are typical time series signals expressed by dynamical systems. Input signals have 2, 4, and 8 periods respectively in 64 steps. Each signal is added a phase shift (maximum one period), amplitude variation (from 50% to 100% of the maximum amplitude), additional noise (maximum 2% of maximum amplitude) and signal length variation (maximum 80% of the maximum signal length). Examples of input data are illustrated in Fig. 5. We compared RNN-AE to MDRA on its feature extraction performance using the above periodic signals. Fig. 6 and Fig. 7 show the of RNN-AE and MDRA respectively. The parameter Table 3 in Appendix B. We used multi-dimensional scaling (MDS) as the dimension reduction method to visualize the distributions of features in Fig. 6 and Fig. 7. Fig. 6 shows the distribution of the encoded data h n which is the initial value of the decoder unit in Fig. 2. We found that RNN-AE can separate the input data into three regions corresponding to each frequency. However each frequency region is distributed widely, therefore some part of the region overlap each other. Fig.7 shows the distributions of the encoded data h n and the clustering allocation weight r n extracted by MDRA. The distribution of r n, shown in the right figure of Fig. 7, is completely separated into each frequency component without overlap. This shows that the distribution of r n as the feature extraction has robustness for a phase shift, amplitude and signal length variation. We also show that MDRA can boost the classification accuracy using an actual driving data in the next section. We applied our algorithm to a driver identification problem. We use the data consisting of 3 drivers signals when driving, including speed, acceleration, braking and steering angle signals. 1 The input signal was about 10 seconds differential data (128 steps), which was cut out from the original data by a sliding window. 2 The detailed information of the input data is shown in Table 1. We also show samples of data (difference of acceleration) in Fig. 10 in Appendix C. The feature extraction by RNN-AE and MDRA are shown in Fig. 8. The parameter setting of this experiment is listed in Table 4 2 We use only the data of which the maximum acceleration difference is more than a certain threshold. and MDRA respectively, the right figure is the distribution of r n of MDRA. We can find different trends in the distributions of the latent variable h n and r n of MDRA. The distribution of r n spreads wider than that of h n. Table 2 shows the accuracy of the driver identification using the above We verified the feature extraction peformance of the MDRA using actual time series data. In Section 5.1, we saw that the periodic signals are completely classified by the frequency using clustering weight r n. In this experiment, the average clustering weights, the elements of are (3.31e-01, 8.31e-47, 8.31e-47, 3.46e-01, 8.31e-47, 3.19e-01, 8.31e-47), with only three components having effective weights. This weight narrowing-down is one of the advantages of VB learning. The left of Fig. 9 shows an enlarged view of around "freq 4" in Fig. 7 (right). We found that the distribution of "freq 4" is in fact spread linearly. The right of Fig. 9 is the ). We found that each frequency data formed several spreading clusters without overlapping. As we saw earlier, the distribution of r n has a spreading distribution compared to that of h n. We inferred that the spreading of the distribution r n was caused by extracting the diversity on the driving scenes. In addition, the identification shows that the combination of the features given by r n and h n can improve the performance. , which studied a driver identify algorithm using the AE, proposed the minimization of the error integrating the reconstruction error of AE and the classification error of deep neural network. This algorithm can avoid the over-fitting by using unlabeled data whose data collection cost is smaller than labeled data. From these , we can expect that the MDRA can contribute to not only boost identification performance but also restrain the over-fitting. In this paper, we proposed a new algorithm MDRA that can extract dynamical system features of a time series data. We conducted experiments using periodic signals and actual driving data to verify the advantages of MDRA. The show that our algorithm not only has robustness for the phase shift, amplitude and signal length variation, but also can boost classification performance. The phase transition phenomenon of the variational Bayes learning method, depending on the hyperparameters, has been reported in The hyperparameter setting of the prior distribution has a great effect on the clustering and classification performance. We intend to undertake a detailed study of the relation between the feature extraction performance and hyperparameter setting of the prior distributions in the future. In addition, where T n D means total signal dimension. Therefore, we obtain We here put Hence Next we calculate log q(α, β), Above equation can be divided into the two terms including α and β respectively, E q(yn) [y nk] = 1 · q(y nk = 1) + 0 · q(y nk = 0) = q(y nk = 1) = r nk to the above equation, we obtain On the other hand, Similarly, by applying E q(yn) [y nk] = r nk, we obtain log q(β) We finally calculate log ρ nk in Eq..We first calculate log q(α) and log q(β), Gamma(β|ν,λ). By using the expectations of β and log β by gamma distribution Similarly, q(α) turns out to be the Dirichlet distribution with parameters (θ 1, · · ·,θ K), and is calculated by the same way in the general mixture model. Therefore we finally obtain From the above , the following variational Bayes algorithm is derived. A where we used r nk = E q(yn) [y nk]. We achieve this by applying RNN algorithm. From the above discussion including Appendix A.1, we obtain the MDRA algorithm. By putting β = e x, we obtain x = log β, dβ = e x dx, In this section, we show the parameter setting of the experiments in Section 5. Here L is the dimension of hidden variable h, capacity, fft and cpx are parameters of EUNN , K is the number of the decoders, θ 0, ν 0, λ 0 are hyperparameters of prior distributions. We applied our algorithm to the driving data in the Section 5.2. We used the differential signals of speed, acceleration, braking and steering angle signals in the experiment. Fig. 10 shows examples of acceleration signals.

Novel time series data clustring algorithm based on dynamical system features.

Represent the short summary for finding the corresponding scientific article (long)!

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.