Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Generative networks are promising models for specifying visual transformations. Unfortunately, certification of generative models is challenging as one needs to capture sufficient non-convexity so to produce precise bounds on the output. Existing verification methods either fail to scale to generative networks or do not capture enough non-convexity. In this work, we present a new verifier, called ApproxLine, that can certify non-trivial properties of generative networks. ApproxLine performs both deterministic and probabilistic abstract interpretation and captures infinite sets of outputs of generative networks. We show that ApproxLine can verify interesting interpolations in the network's latent space. Neural networks are becoming increasingly used across a wide range of applications, including facial recognition and autonomous driving. So far, certification of their behavior has remained predominantly focused on uniform classification of norm-bounded balls;;;;;; b;; c;, which aim to capture invisible perturbations. However, a system's safety can also depend on its behavior on visible transformations. For these reasons, investigation of techniques to certify more complex specifications has started to take place (; a;). Of particular interest is the work of which shows that if the inputs of a network are restricted to a line segment, the verification problem can sometimes be efficiently solved exactly. The ing method has been used to certify non-norm-bounded properties of ACAS Xu networks and improve Integrated Gradients . This work We extend this technique in two key ways: (i) we demonstrate how to soundly approximate EXACTLINE, handling significantly larger networks faster than even methods based on sampling can (a form of deterministic abstract interpretation), and (ii) we use this approximation to provide guaranteed bounds on the probabilities of outputs given a distribution over the inputs (a form of probabilistic abstract interpretation). We believe this is the first time probabilistic abstract interpretation has been applied in the context of neural networks. Based on these techniques, we also provide the first system capable of certifying interesting properties of generative networks. • A verification system APPROXLINE, capable of flexibly capturing the needed non-convexity. • A method to compute tight deterministic bounds on probabilities with APPROXLINE, which is to our knowledge the first time that probabilistic abstract interpretation has been applied to neural networks. • An evaluation on autoencoders for CelebA, where we prove for the first time the consistency of image attributes through interpolations. • The first demonstration of deterministic verification of certain visible, highly non-convex specifications, such as that a classifier for "is bald" is robust to different amounts of "moustache," or that a classifier n A is robust to different head rotations, as shown in Figure 1. Figure 1: Using APPROXLINE to find probability bounds for a generative specification over flipped images. Green polygonal chains represent activation distributions at each layer exactly. Blue boxes are relaxations of segments highlighted by yellow boxes. We label regions with their probabilities. Here, we introduce the terminology of robustness verification and provide an overview of our verification technique. Let N: R m → R n be a neural network with m inputs and n output classes which classifies an input x ∈ R m to class arg max i N (x) i. Specification A robustness specification is a pair (X, Y) where X ⊆ R m is a set of input activations and Y ⊆ R n is a set of permissible outputs for those inputs. Deterministic robustness Given a specification (X, Y), a neural network N is said to be (X, Y)-robust if for all x ∈ X, we have N (x) ∈ Y. In the adversarial robustness literature, the set X is usually an l 2 -or l ∞ -ball, and Y is a set of outputs that correspond to a specific classification. In our case, X shall be a line segment connecting two encodings. The deterministic verification problem is to prove (ideally with 100% confidence) that a network is deterministically robust for a single specification. As deciding robustness is NP-hard , the problem is frequently relaxed to permit false negatives (but not false positives) and solved by sound overapproximation. Probabilistic robustness Even if N is not completely robust, it may still be useful to quantify its lack of robustness. Given a distribution µ over X, we are interested in finding provable bounds on the robustness probability Pr x∼µ [N (x) ∈ Y], which we call probabilistic [robustness] bounds. It is well known that certain generative models appear to produce interpretable transformations between outputs for interpolations of encodings in the latent space (; ; ; ; ; ; ; ; Van den ; ;). I.e., as we move from one latent vector to another, there are interpretable attributes of the outputs that gradually appear or disappear. This leads to the following verification question: given encodings of two outputs with a number of shared attributes, what fraction of the line segment between the encodings generates outputs sharing those same attributes? To anwer this question, we can verify a generator using a trusted attribute detector, or we verify an attribute detector based on a trusted generator. For both tasks, we have to analyze the outputs of neural networks restricted to line segments. computes succinct representations of a piecewise-linear neural networks restricted to such line segments AB ⊂ R m. It is visualized in the top row of Figure 1, where a line segment e 1 e 2 ⊂ R m between encodings produced by an encoder neural network n E is passed through a decoder n D and an attribute detector n A. In more detail, the primitive P(N | AB) computes a polygonal chain (P 1, . . ., P k) in R m representing the line segment AB, such that the neural network N is affine on the segment P i P i+1 for all 0 ≤ i < k. As a consequence, the polygonal chain (N (P 1),..., N (P k)) represents the image of AB under N. To compute this, one can incrementally compute normalized distances on the input segment AB for the output of each layer i of the network, N i. Specifically, we find 0 (A − B) ), and keep track of the nodes N i (A + t i,j (A − B) ). In the case of affine operations such as matrix multiplication or convolution, one can simply apply that operation to each node and leave the distances unchanged. The case of ReLU more segments may be introduced. To compute ReLU, one can apply it per-dimension, d, and check for each 0 ≤ j < k i whether This is done analogously in the case where the segment is decreasing in dimension d instead of increasing. We extend EXACTLINE to perform exact probabilistic inference by associating with each segment P j in the chain a probability distribution µ j over that segment, and a probability p j. In the case of the uniform distribution, as in Figure 1, every µ j is also a uniform distribution, and p j = t j+1 − t j. APPROXLINE Unfortunately, EXACTLINE sometimes scales poorly for tasks using generative models, because too many line segments are generated. We improve scaling by introducing a sound overapproximation, APPROXLINE. Instead of maintaining a single polygonal chain, APPROXLINE maintains a list of polygonal chains and a list of interval constraints 1, such that the neural network's activations are guaranteed to either lie on one of the polygonal chains or to satisfy one of the interval constraints. We introduce a novel relaxation heuristic, which chooses subsets of line segments in polygonal chains and replaces them with interval constraints that subsume them. Our relaxation heuristic attempts to combine line segments that are adjacent and short. To perform probabilistic inference, each element also carries its probability. For a feed-forward network, each operation acts on the elements individually, without modifying the probability associated with it. This is shown in the bottom row of Figure 1. Here, the probabilities of the segments that get approximated are shown by the yellow regions in the top row. One can observe that they remain unchanged when converted to intervals (in blue) in the bottom row. One can also observe that probabilities associated with intervals do not change, even when the intervals change substantially. Python pseudocode for propogation of APPROXLINE is shown in Appendix A. To understand the interaction between probabilistic inference and the relaxation heuristic, it is best to work through a contrived example. Suppose we have an EXACTLINE in two dimensions with the nodes,,,, with weights 0.1, 0.1, 0.5, 0.3 on its segments. The weights describe the probabilities of the output being on each segment respectively, where distributions on the segments themselves are uniform. Our relaxation heuristic might combine the line segments with nodes, and, and approximate them by a single interval constraint with lower bound and upper bound. (I.e., the first component is between 1 and 3, and the second component is between 1 and 2.) In this case, we can understand the output of approximation as an APPROXLINE: An interval constraint with center (2, 1.5), radius (1, 0.5) and weight 0.1 + 0.1 = 0.2, as well as a polygonal chain formed of the segment with weight 0.5 and the segment with weight 0.3. To compute a lower bound on the robustness probability, we would sum the probabilities of each element where it can be proven there is no violation. For example, assume that only the point is disallowed by the specification. The inferred robustness probability lower bound would be 0.8. To compute an upper bound on the robustness probability, we sum the probabilities of each element where it can be proven that at least one point is safe. Here, we obtain 1. Dvijotham et al. (2018a) verify probabilistic properties universally over sets of inputs by bounding the probability that a dual approach verifies the property. In contrast, our system verifies properties that are either universally quantified or probabilistic. However, the networks we verify are multiple orders of magnitude larger. While they only provide upper bounds on the probability that a specification has been violated, we provide extremely tight bounds on such probabilities from both sides. PROVEN uses sampling to find high confidence bounds (confidence intervals) on the probability of misclassification. While PROVEN only provides high confidence bounds (99.99%), APPROXLINE provides bounds with 100% confidence. Nevertheless, our method is much faster and produces better than a similar sampling-based technique for finding confidence intervals using (used by smoothing methods). Another line of work is smoothing, which provides a defense with high confidence statistical robustness guarantees (; ; ; ;). In contrast, APPROXLINE provides deterministic guarantees, and is not a defense. We briefly review important concepts, closely following their presentations given in previous work where applicable. In our work, we assume that we can decompose the neural network as a sequence of l piecewise-linear layers: An abstract domain is a set of symbolic representations of sets of program states. We write A n to denote an abstract domain whose elements each represent an element of P(R n), in our case a set of vectors of n neural network activations. The concretization function γ n: A n → P(R n) maps a symbolic representation a ∈ A n to its concrete interpretation as a set X ∈ P(R n) of neural network activation vectors. The concrete transformer Using this notation, the (X, Y)-robustness property of a neural network N can be written as An abstract transformer T # f: A m → A n transforms symbolic representations to symbolic representations overapproximating the effect of the function f: g. We will follow this recipe for the neural network N, abstracting it as Abstract interpretation provides a sound, typically incomplete method to certify neural network robustness. Namely, to show that a neural network N: n }. Abstract interpretation with the box domain B is equivalent to bounds propagation with standard interval arithmetic. Powerset domain Given an abstract domain A, elements of its powerset domain P(A) n are (finite) sets of elements of A n. The concretization function is given by γ n (a) = a ∈a γ n (a) (using the concretization function of the underlying domain A). We can lift any abstract transformer for A to an abstract transformer for P(A) by applying the transformer to each of the elements. Union domain Given abstract domains A and A, an element of their union domain is a tuple (a, a) with a ∈ A n and a ∈ A n. The concretization function is γ n (a, a) = γ n (a) ∪ γ n (a). We can apply abstract transformers of the same function for A and A to the tuple elements independently. We denote as D n the set of probability measures over R n. Probabilistic abstract interpretation is an instantiation of abstract interpretation where deterministic points from R n are replaced by measures from D n. I.e., a probabilistic abstract domain is a set of symbolic representations of sets of measures over program states. We again use subscript notation to determine the number of activations: a probabilistic abstract domain A n has elements that each represent an element of P(D n). The probabilistic concretization function γ n: A n → P(D n) maps each abstract element to the set of measures it represents. For a measurable function f: R m → R n, the corresponding probabilistic concrete transformer where Y ranges over measurable subsets of R n. A probabilistic abstract transformer T # f: A m → A n abstracts the probabilistic concrete transformer in the standard way: it satisfies ∀a ∈ A m. T f (γ m (a)) ⊆ γ n (T # f (a)), as in the deterministic setting. Probabilistic abstract interpretation provides a sound method to compute bounds on robustness probabilities. Namely, to show that Domain lifting Any deterministic abstract domain can be directly interpreted as a probabilistic abstract domain, where the concretization of an element is given as the set of probability measures whose support is a subset of the deterministic concretization. The original deterministic abstract transformers can still be used. Convex combinations Given two probabilistic abstract domains A and A, we can form their convex combination domain, whose elements are tuples (a, a, p) with a ∈ A n, a ∈ A n and p ∈. The concretization function is given by γ n (a, a, p) We can apply abstract transformers of the same function for A and A to the respective elements of the tuple independently, leaving p intact. Similarly, given a single probabilistic abstract domain A, elements of its convex combination domain are tuples (a, λ) where.., k}}. We can apply abstract transformers for A independently to each entry of a, leaving λ intact. Here we define APPROXLINE, its non-convex relaxations, and its usage for probabilistic inference. First, note that we can use EXACTLINE to create an abstract domain E. The elements of E n are polygonal chains (P 1, . . ., P k) in R n for some k. The concretization function γ n maps a polygonal chain (P 1, . . ., P k) in R n to the set of points in R n that lie on it. For a piecewise-linear function m to a new polygonal chain in R n by concatenating the of the EXACTLINE primitive on consecutive line segments P i P i+1, eliminating adjacent duplicate points and applying the function f to all points. The ing abstract transformers are exact, i.e., they satisfy the subset relation in ) with equality. Our abstract domain is the union of the powersets of the EXACTLINE and box domains. Therefore, an abstract element is a tuple of a set of polygonal paths and a set of boxes, whose interpretation is that the activations of the neural network in a given layer are on one of the polygonal paths or within one of the boxes. For x 1, x 2 ∈ R n, we write S(x 1, x 2) = ({(x 1, x 2)}, {}) to denote the abstract element that represents a single line segment connecting x 1 and x 2. Like EXACTLINE, we focus on the case where the abstract element describing the input activations captures such a line segment. Note that if we use the standard lifting of abstract transformers T # Li for the EXACTLINE and box domains into our union of powersets domain, propagating a segment S(is equivalent to using only the EXACTLINE domain: As the standard lifting applies the abstract transformers to all elements of both sets independently, we will simply obtain an abstract element ({(P 1, . . ., P k), {}), where (P 1, . . ., P k) is a polygonal path exactly describing the image of x 1 x 2 under N. Relaxation Therefore, our abstract transformers may, before applying a lifted abstract transformer, apply relaxation operators that turn an abstract element a into another abstract element a such that γ n (a) ⊆ γ n (a). We use two kinds of relaxation operators: bounding box operators remove a single line segment, splitting the polygonal chain into at most two new polygonal chains (at most one on each side of the removed line segment). The removed line segment is then replaced by its bounding box. Merge operators replace multiple boxes by their common bounding box. Carefully applying the relaxation operators, we can explore a rich tradeoff between the EXACTLINE domain and the box domain. Our analysis generalizes both: if we never apply any relaxation operators, the analysis reduces to EXACTLINE, and will be exact but potentially slow. If we relax the initial line segment into its bounding box, the analysis reduces to box and be will be imprecise but fast. Relaxation heuristic For our evaluation, we use the following relaxation heuristic, applied before each convolutional layer of the neural network. The heuristic is parameterized by a relaxation percentage p ∈ and a clustering parameter k ∈ N. Each chain with t > 1000 nodes is traversed from one end to the other, and each line segment is turned into its bounding box, until the chain ends, the total number of nodes visited exceeds t/k or we find a line segment whose length is strictly above the p-th percentile, computed over all segment lengths in the chain prior to applying the heuristic. All bounding boxes generated in one such step (from adjacent line segments) are then merged, the next segment (if any) is skipped, and the traversal is restarted on the remaining segments of the chain. This way, each polygonal chain is split into some new polygonal chains and a number of new boxes. The EXACTLINE domain can be extended such that it captures a single probability distribution on a polygonal chain. For each line segment (P i, P i+1) on the polygonal chain (P 1, . . ., P k) in R n, we additionally store a symbolic representation of a measure µ i on, such that where X ranges over measurable subsets of R n. I.e., we have γ n (a) = {ν}. Whenever an abstract transformer splits a line segment, it additionally splits the corresponding measure, appropriately applying affine transformations, such that the new measures each range over again. Note that if measures are uniform, it suffices to store µ i as the symbolic representation of µ i. Our probabilistic abstract domain is the convex combination of the convex combination domains of this probabilistic EXACTLINE domain and the standard lifting of the box domain as a probabilistic abstract domain. In practice, it is convenient to store an abstract element a with p probabilistic polygonal chains and q probabilistic boxes as Its concretization is then given as where Our input always captures a uniform distribution on a line segment. Relaxation and heuristic Our deterministic relaxations can be extended to work in the probabilistic setting. When we replace a line segment by its bounding box, we use the total weight in its measure as the new entry in the weight vector λ corresponding to the box. When we merge multiple boxes, their weights are added to give the weight for the the ing box. We then use the same relaxation heuristic as we described previously also in the probabilistic setting. Computing bounds Given a probabilistic abstract element a as above, describing the output distribution of the neural network, we want to compute optimal bounds on the robustness probabilities P = {ν(Y) | ν ∈ γ n (a)}. The part of the distribution tracked by the probabilistic EXACTLINE domain has all its probability mass in perfectly determined locations, while the probability mass in each box can be located anywhere inside it. We can compute bounds (l, u) = (min P, max P) = e + j∈L λ j, e + j∈U λ j, where Here, we used the deterministic box concretization γ n. We write APPROXLINE p k to denote our analysis (deterministic and probabilistic versions) where the relaxation heuristic uses relaxation percentage p and clustering parameter k. We implement APPROXLINE as in the DiffAI framework, taking advantage of the GPU parallelization provided by PyTorch . Additionally, we use our implementation of APPROXLINE to compute exact without approximation. To get exact , it suffices to set the relaxation percentage p to 0, in which case the clustering parameter k can be ignored. Verification using APPROXLINE 0 k is equivalent to EXACTLINE up to floating point error. To distinguish our GPU implementation from the original CPU implementation, we call our method EXACT instead of EXACTLINE. EXACT is additionally capable of doing exact probabilistic inference. We run on a machine with a GeForce GTX 1080 with 12 GB of GPU memory, and four processors with a total of 64 GB of RAM. For generative specifications, we use decoders from autoencoders with either 32 or 64 latent dimensions trained in two different ways: VAE and CycleAE, described below. We train them to reconstruct CelebA with image sizes 64 × 64. We always use with a learning rate of 0.0001 and a batch size of 100. The specific network architectures are described in Appendix B. Our decoder always has 74128 neurons and the attribute detector has 24676 neurons. VAE l is a variational autoencoder with l latent dimensions. CycleAE l is a repurposed CycleGAN with l latent dimensions. While these were originally designed for unsupervised style transfer between two data distributions, P and Q, we use it to build an autoencoder such that the generator behaves like a GAN and the encodings are distributed evenly among the latent space. Specifically, we use a normal distribution in l dimensions for the embedding/latent space P with a small feed forward network D P as the latent space discriminator. The distribution Q is the image distribution, and for its discriminator D Q we use the BEGAN method , which determines an example's realism based on an autoencoder (also with l latent dimensions), which is trained to reproduce the ground-truth distribution Q and adaptively to fail to reproduce the GAN generator's distribution. Attribute Detector is trained to recognize the 40 attributes provided by CelebA. Specifically, the attribute detector has a linear output. We consider the attribute i to be detected as present in the input image if and only if the i-th component of the output of the attribute detector is strictly greater than 0.5. The attribute detector is trained using Adam, minimizing the L1 loss between either 1 and the attribute (if it is present) or 0 and the attribute (if it is absent). We train it for 300 epochs. Given a generative model capable of producing interpolations between inputs which remain on the data manifold, there are many different verification goals one might pursue: E.g., check whether the generative model is correct with respect to a trusted classifier or whether a classifier is robust to interpretable interpolations between data points generated from a trusted generative model. Even trusting neither the generator nor the classifier, we might want to verify that they are consistent. We address all of these goals by efficiently computing the attribute consistency of a generative model with respect to an attribute detector: For a point picked uniformly at random between the encodings e 1 and e 2 of two ground truth inputs with matching attributes, we would like to determine the probability that its decoding will have the same attribute i. We define the attribute consistency as where t is the ground truth for attribute i. We will frequently omit the attribute detector n A and the decoder n D from C if it is clear from context which networks are being evaluated. In this section, we demonstrate that probabilistic APPROXLINE is precise and efficient enough to provide useful bounds on the attribute consistency for interesting generative models and specifications on a reasonable dataset. To this end we compare APPROXLINE to a variety of other methods which are also capable of providing probabilistic bounds. We do this for CycleAE 32 trained for 200 epochs. Specifically, suppose P is a set of unordered pairs {a, b} from the data set with a A,i > 0.5 ⇐⇒ b A,i > 0.5 for each of the k attributes i, where a A are ground truth attribute labels of a. Using each method, we find bounds on the true value of average attribute consistency asĈ where n E is the encoding network. Each method finds a probabilistic bound, [l, u], such that l ≤Ĉ ≤ u. We call u − l its width. We compare probabilistic APPROXLINE against two other probabilistic abstract domains, EXACT (=APPROXLINE 0 k), and HZono lifted probabilistically. Furthermore, we also compare against sampling with binomial confidence intervals on C using the ClopperPearson interval. For probabilistic sampling, we take samples and recalculate the ClopperPearson interval with a confidence of 99.99% until the interval width is below 0.002 (chosen to be the same as our best with APPROXLINE). To avoid an incorrect calculation, we discard this interval and prior samples, and resample using the estimated number of samples. Importantly, the probabilistic bound returned by the abstract domains is guaranteed to be correct 100% of the time, while for sampling it is only guaranteed to be correct 99.99% of the time. For all methods, we set a timeout of 60s, and report the largest possible probabilistic bound if a timeout or out-of-memory error occurs. For APPROXLINE, if an out-of-memory error occurs, we refine the hyperparameters using schedule A in Appendix C and restart (without resetting the timeout clock). Figure 2 shows the of running these on |P | = 100 pairs of matching celebrities with matching attribute labels, chosen uniformly at random from CelebA (each method uses the same P). The graph shows that while HZono is the fastest domain, it is unable to prove any specifications. Sampling and EXACT do not appear to be significantly slower than APPROXLINE, but it can be observed that the average width of the probabilistic bounds they produce is large. This is because Sampling frequently times out, and EXACT frequently exhausts GPU memory. On the other hand, APPROXLINE provides an average probabilistic bound width of less than 0.002 in under 30s with perfect confidence (compared with the lower confidence provided by sampling). Here, we demonstrate how to use our domain to check the attribute consistency of a model against an attribute detector. We do this for two possible generative specifications: (i) generating rotated heads using flipped images, and (ii) adding previously absent attributes to faces. For the in this section, we use schedule B described in Appendix C. Comparing models with turning heads It is known that VAEs are capable of generating images with intermediate poses of the subject from flipped images of the subject. An example of this transformation is shown in Figure 3b. Here, we show how one can use APPROXLINE to compare the effectiveness of different autoencoding models in performing this task. To do this, we trained all 4 architectures described above for 20 epochs. We then create a line specification over the encodings, where a and Flipped(a) are the images shown in Figure 3b. The width of the largest probabilistic bound was smaller than 3 × 10 −6, so only the lower bounds are shown. Less than 50 seconds were necessary to compute each bound, and the fastest computation was for CycleAE 64 at 30 seconds. Lower Bound on Correctness of the flipped images shown in Figure 3b. For a human face that is turned in one direction, ideally the different reconstructions will correspond to images of different orientations of the same face in 3D space. As none of the CelebA attributes correspond to pose, the attribute detector should recognize the same set of attributes for all interpolations. We used deterministic APPROXLINE to demonstrate which attributes provably remain the correct for every possible interpolation (as visualized in Appendix E). While we are able to show in the worst case, 32 out of 40 attributes are entirely robust to flipping, some attributes are not robust across interpolation. Figure 4 demonstrates the of using probabilistic APPROXLINE to find the average lower bound on the fraction of the input interpolation encodings which do in the correct attribute appearing in the output image. Verifying attribute independence Here, we demonstrate using APPROXLINE that attribute detection for one feature is invariant to a transformation in an independent feature. Specifically, we verify for a single image the effect of adding a mustache. This transformation is shown in Figure 3c. To do this, we find the attribute vector m for "mustache" (i = 22 in CelebA) using the 80k training-set images in the manner described by , and compute probabilistic bounds for C j (n E (o), n E (o) + 2m, o A,j ) for j = 22 and the image o. Using APPROXLINE we are able to prove that 30 out of the 40 attributes are entirely robust through the addition of a mustache. Among the attributes which can be proven to be robust are i = 4 for "bald" and i = 39 for "young". We are able to find that the attribute i = 24 for "NoBeard" is not entirely robust to the addition of the mustache vector. We find a lower bound on the robustness probability for that attribute of 0.83522 and an upper bound of 0.83528. In this paper we presented a highly scalable non-convex relaxation to verify neural network properties where inputs are restricted to a line segment. Our show that our method is faster and more precise than previous methods for the same networks, including sampling. This speed and precision permitted us to verify properties based on interesting visual transformations induced by generative networks for the first time, including probabilistic properties. For both models, we use the same encoders and decoders (even in the autoencoder descriminator from BEGAN), and always use the same attribute detectors. Here we use Conv s C × W × H to denote a convolution which produces C channels, with a kernel width of W pixels and height of H, with a stride of s and padding of 1. FC n is a fully connected layer which outputs n neurons. ConvT s,p C × W × H is a transposed convolutional layer with a kernel width and height of W and H respectively and a stride of s and padding of 1 and out-padding of p, which produces C output channels. • Latent Descriminator is a fully connected feed forward network with 5 hidden layers each of 100 dimensions. • Encoder is a standard convolutional neural network: x → Conv 1 32 × 3 × 3 → ReLU → Conv 2 32 × 4 × 4 → ReLU → Conv 1 64 × 3 × 3 → ReLU → Conv 2 64 × 4 × 4 → ReLU → FC 512 → ReLU → FC 512 → l. • Decoder is a transposed convolutional network which has 74128 neurons: l → FC 400 → ReLU → FC 2048 → ReLU → ConvT 2,1 16 × 3 × 3 → ReLU → ConvT 1,0 3 × 3 × 3 → x • Attribute Detector has 24676 neurons: x → Conv 2 16 × 4 × 4 → ReLU → Conv 2 32 × 4 × 4 → ReLU → FC 100 → 40. While many refinement schemes start with an imprecise approximation and progressively tighten it, we observe that being only occasionally memory limited and rarely time limited, it conserves more time to start with the most precise approximation we have determined usually works, and progressively try less precise approximations as we determine that more precise ones can not fit into GPU memory. Thus, we start searching for a probabilistic robustness bound with APPROXLINE Here we demonstrate how modifying the approximation parameters, p and N of APPROXLINE p N effect its speed and precision. Figure 5 shows the of varying these on x-axis. The bottom number, N is the number of clusters that will be ideally made, and the top number p is the percentage of nodes which are permitted to be clustered. Figure 6: Blue means that the interpolative specification visualized in Figure 3b has been deterministically and entirely verified for the attribute (horizontal) using APPROXLINE

Represent the TLDR for finding the corresponding scientific article

We verify deterministic and probabilistic properties of neural networks using non-convex relaxations over visible transformations specified by generative models

represent the tldr for finding the corresponding scientific article

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the article for finding a single-sentence TLDR

Question: Multi-view video summarization (MVS) lacks researchers’ attention due to their major challenges of inter-view correlations and overlapping of cameras. Most of the prior MVS works are offline, relying on only summary, needing extra communication bandwidth and transmission time with no focus on uncertain environments. Different from the existing methods, we propose edge intelligence based MVS and spatio-temporal features based activity recognition for IoT environments. We segment the multi-view videos on each slave device over edge into shots using light-weight CNN object detection model and compute mutual information among them to generate summary. Our system does not rely on summary only but encode and transmit it to a master device with neural computing stick (NCS) for intelligently computing inter-view correlations and efficiently recognizing activities, thereby saving computation resources, communication bandwidth, and transmission time. Experiments report an increase of 0.4 in F-measure score on MVS Office dataset as well as 0.2% and 2% increase in activity recognition accuracy over UCF-50 and YouTube 11 datasets, respectively, with lower storage and transmission time compared to state-of-the-art. The time complexity is decreased from 1.23 to 0.45 secs for a single frame processing, thereby generating 0.75 secs faster MVS. Furthermore, we made a new dataset by synthetically adding fog to an MVS dataset to show the adaptability of our system for both certain and uncertain surveillance environments. Surveillance cameras installed indoor and outdoor at offices, public places, and roads generate huge amount of video data on daily basis. This gigantic volume of data has two big issues: first one is storage consumption and second is huge computational complexity for its purposeful usage. Video summarization aims at these problems by condensing the data size via extracting key information from lengthy videos and suppressing the redundant frames. A video summary generated from a single camera is called single-view video summarization (SVS) . On the other hand, a summary generated from a camera network is known as MVS (a). SVS is intensively researched with applications to various domains including surveillance , sports , and news videos . In contrast, MVS is not studied deeply because of several challenges such as computing inter-and intra-view correlations, overlapping regions among connected cameras, and variation in light conditions among different views. The basic flow of MVS includes input acquisition, preprocessing, feature extraction, post-processing, and summary generation. The mainstream MVS methods follow traditional machine learning approaches such as clustering along with low-level features extracted from entire frame with no focus on specific targets in surveillance. The most important part of MVS is considering different objects in surveillance that can be useful for summary generation. However, the existing techniques do not focus on objects such as persons and vehicles while generating summary. Thus, the final summary may miss some important frames having persons or vehicles that need to be considered for MVS. Furthermore, all the existing techniques rely only on MVS with no further steps for analysis of the generated summary. For instance, the generated summary can be used for indexing, browsing, and activity recognition. The existing methods are functional only in certain environments with no focus on uncertain scenarios , making them inadequate in real-world environments. Finally, all the existing methods process data on local/online servers or personal computers with huge computation power. It requires extra processing time, power of transmission, and does not guarantee quick responsive action for any abnormal situations, if not handled on the edge. To ensure proper and quick responsive arrangements, activity recognition at edge is a necessary requirement of the current technological era. Activity recognition literature is mature, but with no focus on processing over the edge. Almost all the existing techniques classify activities over high computational local or cloud servers. Classifying activity on edge is an important task of surveillance in smart cities. Therefore, to tackle these challenges effectively, we present a novel framework applicable in both certain and uncertain environments for MVS and activity recognition over the edge. Figure 1: Input and output flow of our proposed framework. (a) Video frames (both certain and uncertain environment) from resource constrained devices. (b) Annotate frames by detecting objects of interest, apply keyframes selection mechanism, generate summary, encode and transmit it to master device. (c) Decode generated summary, perform features extraction, and forward it to activity prediction model at master device to get the output class with probability score. The problems aimed in this paper are different from the schemes presented in existing literature. We integrated two different domains including MVS and activity recognition under the umbrella of a unified framework in an IoT environment. We presented interconnected resource constrained IoT devices working together to achieve several targets i.e., object detection, summary generation, and activity recognition as shown in Figure 1. The overall framework consists of numerous slaves and a master resource constrained device connected through a common wireless sensor network (WSN). The slave devices are equipped with a camera to capture multi-view video data, segment it into shots, generate summary, encode a sequence of keyframes, and transmit it to the master device. The master device is equipped with an INTEL Movidius NCS to classify the ongoing activity in the acquired sequence. INTEL Movidius is a modular and deep learning accelerator in a standard USB 3.0 stick. It has a Vision Processing Unit (VPU) that is functional with ultra-low power and better performance. It enables activity recognition with significantly lower power, storage, and computational cost. Further, a widely used concept of temporal point processing is utilized for activity classification, ensuring an effective recognition model. While addressing the problems in MVS and activity recognition over resource constrained devices, we made the following contributions. • Employing an algorithm for MVS on resource constrained devices, reducing the time complexity compared to existing approaches with higher accuracy. The generated summary is further utilized to recognize the underlying activity of all the views through an auto-encoder and learned spatio-temporal features followed by different variants of SVM classifiers to demonstrate the efficiency and effectiveness of our proposed framework. • Adding uncertainties such as fog to an outdoor MVS benchmark dataset to demonstrate the working of proposed framework in any type of scenario and introduce a new trend in MVS literature for researchers. • The presented framework has high-level adaptability with special care for the capacity and traffic of WSN. It has many flavors with tradeoff among transmission time, quality of keyframes, and accuracy of activity recognition model with computationally different classifiers. In the subsequent sections, Section 2 provides a literature review and Section 3 explains the presented framework in detail. In Section 4, experimental for MVS and activity recognition are given, and Section 5 concludes the overall paper with future directions. This section is mainly divided into two sub-sections, covering the representative studies of MVS and activity recognition related to our work. The literature of MVS was initiated almost a decade ago in 2010 by with the first MVS indoor dataset. The employed MVS methods are computationally complex but they are not suitable for resource constrained devices. In contrast to MVS, activity recognition is a richer research field with a variety of techniques focusing on different applications. Majority of the MVS methods are based on handcrafted-features integrated with traditional machine learning approaches for final summary generation. The initial MVS schemes utilized features such as SIFT descriptors, Gaussian and Laplacian difference, and motion features followed by statistical learning approaches such as K-means or other clustering techniques for summary generation. The next trend of MVS (; b) used the same features by adding pre-or post-processing like subtraction along with supervised learning such as SVM or unsupervised learning. The final summary in this trend is either uniform length or of length provided by user. The next MVS trend (; a) utilized mid-level features, motion-based shot boundary detection, spatiotemporal graphs, and low-level features such as color and edge histograms. The final summary is generated through the same statistical learning-based techniques. A breakthrough in MVS field is noticed after the usage of learned features in prerequisite steps for generating summaries. This trend is followed in 2016 (a), where BVLC CaffeNet and Spatiotemporal C3D features are used for sparse coding and video representation, respectively. Besides clustering, template matching and sparse representative selection over learned embedding are used for generating final summaries. Panda et al. also used C3D features for video representation by computing inter-and intra-view similarities through sparse coefficients and summary generation using clustering. The recent success of deep learning based methods in activity recognition achieved high-level accuracies. These methods extract features from final layers of various deep learning models and apply sequential learning for activity recognition . For instance, used features extracted from a pre-trained AlexNet, followed by bi-directional LSTM for human action recognition with better . Following this, explored optical flow convolutional features with multi-layer LSTM for activity recognition. Their approach outperformed and other state-of-the-art, however, its huge running time restricts its adoptability in real-world surveillance networks. Similar to the previous methods, used different types of CNN architectures and configurations for human action representation, considering both stationary and non-stationary environments. Till date, the activity recognition methods do not perform computation over resource constrained devices with significant accuracy. Thus, to tackle this problem effectively, we optimized VGG-19 model to make it functional over resource constrained devices with details given in Section 3. The overall framework is divided into two major modules based on the functionalities of the resource constrained devices. The first module is related to multi-view video acquisition, shot segmentation, and summary generation. The second module performs computationally expensive processes involving deep features extraction, features comparison and encoding, and activity prediction. The whole framework contains a number of resource constrained devices (master and slave) connected to a WSN in IoT. This section provides details of the proposed system with major steps are given in appendix section A and visualized in Figure 2. Figure 2: The conceptual diagram of our proposed framework, where slave devices with camera capture multi-view video data, apply shot segmentation, extract keyframes, encode and transmit them to master device for computing inter-view correlations and activity recognition. Surveillance analysts are usually interested in activities of humans and vehicles; thus we consider them as objects of our interest. For this purpose, a tiny version of YOLO CNN model is employed, which is much faster and has higher accuracy compared to other detectors. This tiny object detector works well in normal scenarios but it has poor performance when videos contain fog. To overcome this limitation, we used the pre-trained weights of YOLO tiny model and retrained them over foggy data as show in experimental section 3. Fog is applied on the entire image globally without targeting our objects of interest only. Adding fog globally has an advantage of better learning, because in real-life surveillance scenarios, the fog is always observed throughout the image. The foggy data is generated from the COCO dataset by adding synthetic fog to all the images using MATLAB script with different levels of fog, ranging from 0 to 255. We conducted experiments with different values of fog parameter (f p) as shown in experimental section and selected f p = 220 as optimal. The process of foggy data creation is given in Algorithm A in appendix section. Training data for object detection specifically in foggy/normal environment has an additional advantage of reliability. Most of the available multi-view datasets are related to indoor such as Office and Bl-7f. Thus, we conducted experiments on the available multi-view outdoor datasets only, such as Road . The trained model is used over each slave device for object detection. The attached camera of device transmits frames to our customized object detection model. The frames having persons or vehicles with confidence score greater than 0.9 are stored and considered for further steps while the remaining frames are discarded. A summary is generated from the segmented frames with objects, because events occur due to their interaction. Therefore, we considered those shots that can possibly contain events such as sitting on a chair and entering a room etc. in MVS datasets. These events are possible only due to human presence. Due to this reason, we segmented those shots in which humans are detected. The shots with desired objects are larger in number, containing redundant frames and thus cannot be considered in the final summary. To avoid the redundancy and provide users a very compact representation, we compute mutual information between features of consecutive frames. This process is executed over slave devices, having limited execution resources, thus, we used low-level features intelligently. Further, instead of using only Euclidean or any other distance measuring method, we compared mutual information between two consecutive feature vectors to avoid redundancy and obtain better . The feature descriptor is acquired from each frame by computing the ORB points . The frames with persons, having exactly the same information, are discarded and only a single frame is selected from them. The selected frame is considered as a keyframe on the concerned resource constrained device. The formula used for mutual information is given in Eq. 1. Here f p is the previous frame while f n is the current frame and N is the size of feature vector. The mutual information is computed using scikit-learn library in Python. Human activity recognition is an important task in surveillance video analysis. An activity is a sequence of movements in continuous video frames, therefore, it requires extraction of spatiotemporal features. In our system, the activity needs to be recognized on master device after getting summary from the slave device. For activity recognition, we used a pre-trained VGG-19 CNN model for spatial feature representations followed by auto-encoder which captures the temporal features in the sequence. In contrast to VGG-19, the well-known AlexNet and GoogleNet are not effective to capture tiny patterns in visual data because these models have 1111 and 77 filter size with 4 and 2-pixel stride, respectively. We also made experiments on MobileNetV2 , where we extracted features from final convolution layer, but the were not convincing compared to state-of-the-art (see Table 2). VGG-16 model is recently studied for activity recognition problem in and we outperformed their method over various datasets by investigating VGG-19. It can extract discriminative visual features because it uses 33 filter in all convolutional layers with fixed 1-pixel stride. This helps in processing the smallest receptive field to grab the notions of tiny patterns. It has sixteen convolutional and three fully connected layers where we extract deep features from the final fully connected layer (FC8) having 11000 dimension. FC7 layer of VGG-19 outputs 4096 dimension representations which a large sized feature vector for a single sequence and yield huge processing for encoding of temporal representations. Thus, it cannot be considered for activity recognition over resource constrained devices in real-time scenarios. We are processing 15 fps, therefore, we extract 15000 spatial frame-level features from one second summarized frames. The features extracted in a single interval of time are compared to compute inter-view correlations. The features in the same interval from different views, are considered as overlapped and hence a single feature vector is selected from them for further processing. Our feature extraction is performed on Movidius VPU stick, therefore, the original VGG-19 model is compressed using neural computing stick software development kit (NCSDK). This optimized the original model of size 574.7 MB to 287.3 MB to Movidiuss compatible graph format. The temporal changes and sequential patterns in 15000 features are learned using our proposed auto-encoder. Motivated from the wide usage of auto-encoder network for many applications, we utilized it in our framework. Auto-encoder reduces the dimensionality by analyzing patterns in high dimensional data, therefore, we claim that it can learn the spatio-temporal patterns in deep features of 15 consecutive frames. We performed several experiments for squeezing 15000 features to low dimensions to effectively learn the temporal changes. We first encoded 15000 features to 8000, followed by its encoding to 1000 features (first setting). This kind of setting is very effective because its mean square error (MSE) for encoded features is very low, providing precise accuracy. However, its time complexity is not meeting the needs of embedded devices because the total squeezing has one extra step of encoding to 8000 features. In another setting, we encoded the 15000 high dimensional features directly to 1000 (second setting). In this setting, the accuracy is slightly compromised when using linear SVM, however, we met the computational requirements of embedded devices. We trained our auto-encoders for 40 epochs in which we utilized sparsity regularization with proportion value of 0.1 and L2 weights regularization to reduce the chance of over-fitting and avoid being stuck in local minima. To this end, MSE with support of sparsity regularization and L2 regularization is trained as a cost function of our auto-encoders. In first setting, the MSE decreased to 0.012 after 40 epochs, while in the second setting, it reduced to 0.044. The encoded features are spatiotemporal representation of an activity, which are passed to SVM classifier for activity recognition. We trained one versus all multi-class SVM because it trains N-1 (N is the number of total classes) classifiers, which is efficient for resource restricted devices. All SVMs are examined for activity recognition problem including linear, quadratic, and cubic SVM. We designed this framework considering the major limitations of WSNs and IoT devices such as computational complexity and storage capacities. We also aimed to provide an independent activity analysis system, which is malleable in both certain and uncertain environments. For this purpose, we first investigated different MVS and activity recognition methods, and studied the capabilities of WSNs. Concluding our literature study, the need of an independent system is realized, which could capture multi-view video data, analyze it, and provide a compact summary with recognized activities in lengthy surveillance videos. To this end, we made extensive experiments with different configurations and parameters, which can be interchangeably used according to users requirements and available resources. For instance, a slave node with extensively limited resources in WSN can encode its summary with PNG compression before its transmission to master node for activity recognition. With this mechanism, a real-time summary generation and transmission to master node is ensured despite the slower connectivity and limited communication bandwidth. Similarly, at master node, different options can be considered to balance execution time and accuracy of activity recognition. Various classifiers were used for experiments with different computational complexities. In addition, different experiments were performed for direct feature extraction from a sample summary as well as encoded features with different configurations for activity recognition. The complete evaluation details of these configurations are discussed in Section 4. We divide the experimental evaluation of our proposed framework into three subsections: MVS, activity recognition, and statistical analysis of data transmission. First, we explain the performance evaluation of MVS employed on resource constrained devices and compared with state-of-the-art techniques. Next, we evaluated the activity recognition module that is carried out on master device with Movidius setup. In the MVS module, we used Raspberry Pi (RPi) for our experiments, but our work is not limited to only RPi and can be applied to any other device with embedded vision. For activity recognition, we used MATLAB as a simulation tool and then transformed it to Keras deep learning framework in Python. The model trained using Keras with Tensorflow in backend can be converted into Movidius compatible graph format using NCSDK. Currently, we tested our model on RPi that can be adjustably used over other Movidius-supported devices. Sample of integrated framework for an input video from UCF-50 dataset with different views are visualized in Figure 4. Figure 4: Sample generated on UCF-50 video captured from two different views. Keyframes from both input videos are generated and then VGG-19 features are extracted. These features are compared and the selected single feature vector is encoded into 11000. Final output of activity is generated by passing this feature vector to trained model that gives probability score along with the predicted class. We used five different datasets for experiments where three of them are utilized for object detection in foggy scenarios and MVS and rest of the them are used to evaluate our activity recognition method. Office is widely used for performance evaluation of MVS methods because of its public availability along with the ground truth. It is created by utilizing four stably-held cameras fitted in an office environment. The cameras have motion with no synchronization between the captured videos with light variations. Bl-7f (a) is the most challenging dataset in MVS literature that has 19 different cameras installed at 7th floor of Taiwan University. The recorded videos contain different events closely related to each other with maximum overlapping. The installed cameras are synchronized, still, and fixed but with different light conditions in different views. The ground truth as well as the dataset is publicly available for research purposes. Road multi-view dataset is captured using handheld cameras with extreme level of shuddering effects and variable light conditions. Different from the above two datasets, it contains persons and vehicles, recorded in an outdoor day light. This dataset is used for foggy outdoor videos to test our object detection model in uncertain environment. Results of object detection over Road dataset are given in Figure 3 in uncertain environment. The activity recognition part of our framework is evaluated using UCF50 and YouTube 11 action datasets. UCF50 contains assorted human action videos with high-level shuddering due to camera motion, difference in viewpoint, and scalability of objects. The total number of classes are 50 with some actions closely related to each other, effecting the overall accuracy of classifiers. YouTube dataset, in contrast to UCF50 is less challenging and includes motion of camera, cluttered , light variation, and changes in viewpoint. A total of 11 different action videos are available in this dataset. 1.00 0. Table 1 shows the MVS of our implemented framework compared to other state-of-the-art techniques. We provide objective evaluation for two benchmark datasets in MVS literature i.e., Office and Bl-7f. The time complexity of our framework is given in the last column, indicating the execution time for a RPi device. The better performance of our method compared to existing methods can be seen from Table 1. On Office dataset, our method achieved the highest F-measure score compared to all methods under consideration. The most recent methods for MVS on Office dataset scores 0.90 and 0.91 with higher computational complexity. The method presented in uses heavy-weight CNN model for spatio-temporal point processing to select video skims. A gap in computational complexity from a recent MVS method over resource constrained devices is indicated in Table I where our method generates MVS 0.78 secs faster than (a). In case of Bl-7f dataset, we outperformed (a) and have similar F1-score with , but our framework is functional over resource constrained devices. The activity recognition module of our system is evaluated using two benchmark action datasets including UCF-50 and YouTube 11. The comparison using an overall accuracy metric with state-ofthe-art techniques is given in Table 2. We conducted different experiments for activity recognition using original VGG-19 features, encoded features with S1, and encoded features with S2 along with variants of SVM including linear, quadratic, and cubic SVM. S1 in Table 2 represents the first setting, where the sequential features are encoded into 8000, followed by 1000 from 15000 features of a single sequence. S2 presents the second setting of directly encoding 15000 features to a 1000 feature vector. These different settings can be alternatively used for different scenarios, considering both accuracy and complexity. On UCF50 dataset, trajectories analysis , hierarchical clustering , ML-LSTM , and deep auto-encoder with CNN achieved 92.3%, 93.2%, 94.9%, and 96.4% accuracy, repetitively. On the other hand, our S1 and S2 with cubic SVM achieved 96.2% and 93.8% overall accuracy, respectively. On YouTube 11 action dataset, the single stream CNN , hierarchical clustering , DB-LSTM , ML-LSTM , and.6% on quadratic and cubic SVM, respectively. From the above comparisons, it can be observed that the accuracy of our different settings are either higher or similar to the state-of-the-art. However, in terms of computational complexity, the existing methods are designed for high processing GPUs but our system is functional over resource constrained devices. The performance of our settings with linear SVM is under 80% but with quadric SVM, it touches the milestone of 90%. Therefore, it can be considered as state-of-the-art for processing over embedded devices with low cost VPU stick. Finally, the experiments performed over MobileNetV2 give lower accuracy on both the datasets. Thus, light-weight MobileNetV2 is not a suitable option for consideration in real-time surveillance for activity recognition. Wireless networks have constrained resources of bandwidth and communication cost. It is therefore not a feasible solution to transmit huge-sized surveillance videos to cloud or local servers due to limited transmission and storage of data. Also, searching for an event in long videos manually is a tiresome task. This section provides statistical details about saving storage capacity, bandwidth, and transmission time as illustrated in Figure 5. To the best of our knowledge, there is no specific dataset available to confirm the efficiency and effectiveness of our framework in terms of bandwidth, transmission time, and storage capacity. Thus, we considered an example video to generate keyframes, encode and transmit them to the master device for recognizing activities. We then compared all the mentioned parameters as given in Figure 5. The frames in the example video are considered as non-compressed with ideal situation of transmission. First, we calculate the total number of pixels by multiplying height (h) of the frame with width (w), no of pixels = hw. In office video, h = 480 and w = 640, ing in 307,200, which is multiplied by the depth of the frame i.e., 8. Finally, we divided the obtained number by 8 to get the size of the frame in bytes and is further converted to MB. The size of a single frame from this video is 0.29 MB. Similarly, to calculate the size of the office video-0, multiply the size of each frame by the number of frames inside it. This video has 26,940 frames, and multiplying it with size of each frame, we get 7892.57. It can be observed that there is a huge difference of storage, indicating the effectiveness of our system in terms of saving storage capacity. Saving storage directly refers to preservation of communication bandwidth because the bandwidth consumed by data is the same as their size. The transmission time in our framework is saved in two ways: 1) in low-bandwidth networks: encode the keyframes with PNG compression, and 2) transmit only important frames over IoT network saving huge time. Suppose an ideal situation in the network to transmit a frame in the WSN from slave to master device. Assume the distance as 0.3 km and speed as 200,000 km/s with data rate of 32 Mbps. To receive the data from slave (source), the data will take 3 types of times: 1) getting frame on WSN, 2) transmitting the data through the network, 3) loading data from router/hub to the master (destination) device. The time denoted as t1 in Eq. 2 is transmission time from source to network router. Transmission time t 2 in Eq. 3 is considered as the time taken by network to transmit data from source router to destination. The transmission time from destination router to the master device is the same as t1. The overall transmission time is the sum of t 1, t 2, and t 3, where t 1 and t 3 are same. The calculated transmission time for overall frames, keyframes, and encoded keyframes is 4137.98, 4.45, 1.13 seconds, respectively, as given in Figure 5. t 2 = distance f rom slave to master device transmission speed Figure 5: Transmission time and storage size comparison of transmitting and storing overall video (Office-0 video) versus keyframes and encoded keyframes. A huge difference of saving transmission time and storage capacity can be observed among all these possible settings. In this paper, we integrated MVS and activity recognition under an umbrella of a unified framework. A complete setup including slaves and a master resource constrained device working independently in an IoT is presented. The hardware requirements include slave devices equipped with camera and wireless sensors, a master device with Intel Movidius NCS for running optimized deep learning models on the edge. The slave devices capture multi-view video data, detect objects, extract features, compute mutual information, and finally generate summary. The generated summary is received at master device with optimized trained model for activity recognition. The MVS algorithm as well activity recognition models presented in this paper outperform state-of-the-art. In future, we have intention to extend this work by deeply investigating multi-view action recognition algorithms with different parameters and configurations in resource constrained environments. Further, we want to explore spiking neural networks used for various tasks in our framework for spatio-temporal features extraction advanced to activity recognition.

An efficient multi-view video summarization scheme advanced to activity recognition in IoT environments.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper aims to leverage good properties of robust visual features like SIFT to renovate CNN architectures towards better accuracy and robustness.


Question: Represent the scientific article to find a short TLDR

Question: A central goal in the study of the primate visual cortex and hierarchical models for object recognition is understanding how and why single units trade off invariance versus sensitivity to image transformations. For example, in both deep networks and visual cortex there is substantial variation from layer-to-layer and unit-to-unit in the degree of translation invariance. Here, we provide theoretical insight into this variation and its consequences for encoding in a deep network. Our critical insight comes from the fact that rectification simultaneously decreases response variance and correlation across responses to transformed stimuli, naturally inducing a positive relationship between invariance and dynamic range. Invariant input units then tend to drive the network more than those sensitive to small image transformations. We discuss consequences of this relationship for AI: deep nets naturally weight invariant units over sensitive units, and this can be strengthened with training, perhaps contributing to generalization performance. Our predict a signature relationship between invariance and dynamic range that can now be tested in future neurophysiological studies. Invariances to image transformations, such as translation and scaling, have been reported in single units in visual cortex, but just as often sensitivity to these transformations has been found (, Sharpee et al. 2013, . Similarly, in deep networks there is variation in translation invariance both within and across layers (, , , . Notionally, information about the position of the features composing objects may be important to category selectivity. For example, the detection of eyes, nose, and lips are not sufficient for face recognition, the relative positions of these parts must also be encoded. Thus it is reasonable to expect some balance between invariance and sensitivity to position. We empirically observe that in a popular deep network, in both its trained and untrained state, invariant units tend to have higher dynamic range than sensitive units (Figure 1B and C). This raises the possibility that the effective gain on invariant units into the subsequent layer is stronger than that of sensitive units. Here we provide theoretical insight into how rectification in a deep network could naturally biase networks to this difference between invariant and sensitive units. We do this by examining how co-variance of a multivariate normal distribution is influenced by rectification, and we then test these insights in a deep neural network. The response of a unit in a feed forward neural network is: r = w · g(S) where S is the response of all n input units in the previous layer, g the non-linearity of rectification g(x) = max(0, x), w is the n × 1 vector of weights, and r is the response of the unit. Randomly sampling from a distribution of input images, the response S takes on a distribution with some expectation and covariance across these images: E[S] = µ (an n × 1 vector), and Cov[S] = Σ (an n × n matrix). The application of the non-linearity transforms these moments: E[g(S)] =μ, Cov[g(S)] =Σ. Let S 1 be the responses to randomly sampled input images and S 2 the responses to a transformation of those same images. So the moments of the full distribution are: Σ 2,1Σ2,2, whereΣ 1,1 is the covariance of rectified input units responding to the original images,Σ 2,2 is the covariance of rectified input units to the transformed images andΣ 1,2 is the covariance between rectified input units responding to the reference and transformed images. We note we only define the 1st two moments above and no assumption about the distribution of the rectified responses is made. The covariance of an output unit with weights w on the n rectified input units is: T so the correlation between the response of the output unit to the reference and transformed images is: Below we investigate how theΣ i,j depend on Σ i,j and µ, which provides insight into a relationship betweenρ andσ 2. We begin by examining a model of a single rectified input unit responding to the reference and transformed images. We model the responses, S 1 and S 2, of a single input unit to the reference and transformed inputs, respectively, as a bivariate normal distribution: When these responses are acted on by rectification, both the variances of the responses and the correlation between the sets of responses is decreased. This observation is analogous to that of de la where they investigated the influence of neuronal firing threshold rectification on the pairwise correlations between neurons as a function of firing rate. We extend this observation in the next section to consider how this effect influences invariance in downstream units. It is instructive to consider a schematic (In the following section, we will write the correlation and variance after rectification explicitly as a function of the relevant parameters:σ(µ/σ) 2 andρ(µ/σ, ρ). Here we extend from the single input unit case to the multi-input unit case by examining the invarianceρ ing from taking weighted combinations of rectified input units. Our key insight is that invariance increases to the degree that directions of maximal variance in the response distribution of rectified input units are integrated. For a first order approximation of this relationship we approximate input covariance before rectification as an identity matrix scaled by the average variance: This approximation improves as off diagonal covariance shrinks andσ increases. Thus our approximate model is: =σ 2 I whereσ 2 is averaged across the diagonals of the original S 1 and T where means are approximated by averaging across the original S 1 and S 2. Cov[S 1, S 2] = Σ 1,2 =ρ σ 2 I where ρσ 2 > 0 justified by the assumption of a small transformation thus correlation is positive. For convenience sort the µ i in from high to low, then it naturally follows that the eigenvectors ofΣ 1,2 andΣ 1,1 are the same: I the identity (since the covariance matrices are diagonal) and the eigenvalues are simply the entries of the diagonal (in order from high to low sinceρ, andσ 2 are decreasing in µ; see Figure 2B) so we have: thus we have the geometric picture described in Figure 3A exactly. The denominator as a function of the direction of a unit length w (length of w does not change ρ) is an axis aligned ellipsoid with length along the ith axis ofσ 2 (µ i /σ). Notice that the numerator is the variation of the output unit thus more invariant units contribute more variance than less invariant units assuming there is not a negative correlation between w 2 i andσ 2 iρ i. The numerator is another axis aligned ellipsoid (blue) with length along the ith axis ofσ 2 (µ i /σ)ρ(µ i /σ, ρ) this numerator ellipsoid is contained within the denominator sinceρ ≤ 1. Recognizingρ as a weighted arithmetic mean with with weights c i = (note c i = 1) we see that if there is not a negative correlation of w i 2 withσ Performing simulations of a few simple input unit covariance structures shows that theρ toσ 2 relationship is maintained, though its form changes (Figure 3B). Integrating over a population of input units the form of the relationship changes from the single input unit case (Figure 3B black dashed and dotted line). Figure 3B red and cyan) overall variance increases because correlated input units are being added. Here we analyze the covariance structure of the inputs of a popular deep neural network (AlexNet) for translations of input images. We first tested the network in its untrained state by presenting a collection of 500 image patches drawn from the 2012 ImageNet validation set. References images were cropped enough to allow the original and translated images to fit within the maximal receptive field of the units being tested. We included a small and a large translation, and at each convolutional layer we measured the correlation and variance. We find a positive relationship at all layers with significant Spearman's ranked correlation for both transformations (Table 1 Untrained). The strength of the relationship tended to be stronger for the smaller translation (Figure 1C, orange). Thus in a popular deep network with no training, units which tended to have greater invariance also had higher dynamic range. We repeated the same analysis in AlexNet after it was fully trained for object recognition. Again we observed a significant positive relationship (Table 1 Trained; Figure 1B). The relationship was somewhat weaker than in the trained network. Thus training weakens but does not remove the bias of the network to associate higher dynamic range with higher translation invariance. Finally, we asked whether the network may compensate for this imbalance by placing weights of higher magnitude on low dynamic range units (a negative correlation betweenσ 2 i and w 2 i), thus effectively removing this bias. We measured whether the percent of weight magnitude on a given input unit across output units was greater for input units with higher variance. We found Conv3 (r s = 0.34) and Conv4 (r s = 0.19) tended to have higher weights on higher variance input units while there was no correlation in Conv2 and and Conv5. This indicates the network does not compensate for the imbalance in dynamic range between invariant and sensitive units but actually sometimes emphasizes it. We have documented an empirical relationship between the dynamic range of unrectified units in a deep network and their invariance. We provided a simple 1st order statistical model to explain this effect in which rectification caused the population representation to primarily vary in dimensions that were invariant to small image perturbations, whereas small perturbations were represented in directions of lower variance. Further work can investigate whether this imbalance improves generalization because of the emphasis placed on invariant over sensitive units. We note this relationship is weaker in the trained then untrained network further work can udnerstand this difference. Our approximations assumed low covariance between input units and homoegenous input variance while this may be expected in a random network it may not be true in a trained network. More crucially further theoretical work should consider the influence of co-variance between input units and invariance of output units as a function of weights. To extend insights from simplified, artificial networks to neurobiology, it will first of all be important to test whether cortical neurons showing more invariance also tend to have a higher dynamic range. If they do, this will establish a fundamental theoretical connection between computations of deep networks and the brain. We thank our reviewers for their careful and insightul comments. Above we have taken their comments into account in editing our final draft. Below we address their three main concerns. It is instructive to consider a schematic (Figure 2A) of the distribution of responses. The probability mass of the response is broken into 4 quadrants, the 1st green is unaffected by rectifications, the 2nd (purple) is projected onto the vertical axis (thick purple), the 3rd (red) is projected onto the origin (red dot), and the 4th (green) projected onto the horizontal axis (thick green). The diagonal line is the line of best fit expressing the linear relationship and the vertical line is a conditional distribution whose variance is the conditional residual which averaged gives the residual variance of the linear relationship.σ 2 decreases as µ/σ decreases because the spread of the distribution is truncated to the degree that the distribution falls beneath the threshold. For correlation it is useful 3 to consider: 2 V ar(S 1) decreases more rapidly then the average residual E[V ar(S 2 |S 1)]. Notionally we can think of V ar[E[S 2 |S 1]] as the vertical height of the diagonal line that has not been truncated (solid not truncated, dashed truncated) in Figure 2A and E[V ar(S 2 |S 1)] as the average length of vertical lines not truncated. Notice that the ratio of truncated to untruncated is lower for the diagonal then the vertical average. In the figure at µ = 0 the diagonal line is cut in half and so is the length of a vertical line drawn here. But at all other positions (µ > 0) where a vertical line would be drawn the vertical solid lines length is truncated less than half thus on average the vertical line is less truncated than the diagonals vertical length. We would like to emphasize we are not arguing that rectification explains the generalization properties of networks only that its influence on covariance may be one of many factors influencing invariance. We would like to emphasize that in this paper we pursue intuition by trying to understand a simple approximation to rectifications influence on invariance which in a simple analytic form. Our first approximation is to remove off-diagonal covariances. Since the influences of off-diagonals are additive they can be seen as modulating the effects induced by the diagonals: Thus here we analytically study the first order effect of rectification in output neurons on the basis of the variance but not covariance of their inputs. Finally we approximate the diagonal of the input variance with the average variance an approximation which minimizes squared error variation in σ 2 will hurt the strength of this approximation but not change the main effect unless this variation is negatively correlated with µ i thus canceling out the relationship between correlation and variance. We would not expect this negative correlation in an untrained network and further work can check whether this, potentially interesting, relationship exists in trained networks. We note that normalization enforces this approximation and thus these approximations may be particularly suited to networks using normalization. Distribution in quadrant I is preserved (pink), quadrant II, IV collapsed onto S1 and S2 axis respectively (thick green, blue lines), and III mapped onto origin (red). (B) Plottingσ 2 (µ/σ) againstρ(µ/σ, ρ) there is a positive relationship because both are increasing with µ/σ. ρ is transformed to Fisher's z andσ 2 plotted on log axis revealing an approximate relationship: a(σ 2) b = z(ρ).

Represent the short summary for finding the corresponding scientific article (long)

Rectification in deep neural networks naturally leads them to favor an invariant representation.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We study the use of the Wave-U-Net architecture for speech enhancement, a model introduced by Stoller et al for the separation of music vocals and accompaniment. This end-to-end learning method for audio source separation operates directly in the time domain, permitting the integrated modelling of phase information and being able to take large temporal contexts into account. Our experiments show that the proposed method improves several metrics, namely PESQ, CSIG, CBAK, COVL and SSNR, over the state-of-the-art with respect to the speech enhancement task on the Voice Bank corpus (VCTK) dataset. We find that a reduced number of hidden layers is sufficient for speech enhancement in comparison to the original system designed for singing voice separation in music. We see this initial as an encouraging signal to further explore speech enhancement in the time-domain, both as an end in itself and as a pre-processing step to speech recognition systems. The remainder of this paper is structured as follows. In section 2, we briefly review related work 27 from the literature. In section 3, we introduce briefly the Wave-U-Net architecture and its application architectures like that of BID0. Recently, the U-Net architecture on magnitude spectrograms has for each prediction and is based on the repeated application of dilated convolutions with exponentially increasing dilation factors to factor in context information. yield an estimate of the target sources, a tanh nonlinearity follows, succeeded by a final LeakyReLU. In applying the Wave-U-Net architecture to the application of speech enhancement, our objective 58 is to separate a mixture waveform DISPLAYFORM0 To evaluate and compare the quality of the enhanced speech yielded by the Wave-U-Net, we mirror filter sizes, to the task and expanding to multi-channel audio and multi-source-separation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

The Wave-U-Net architecture, recently introduced by Stoller et al for music source separation, is highly effective for speech enhancement, beating the state of the art.

Represent the short summary for finding the corresponding scientific article (long)

We present a new feed forward graph ConvNet based on generalizing the wavelet scattering transform of Mallat, and demonstrate its utility in graph classification and data exploration tasks.