Question: represent the article for finding a single-sentence tldr

Question: In weakly-supervised temporal action localization, previous works have failed to locate dense and integral regions for each entire action due to the overestimation of the most salient regions. To alleviate this issue, we propose a marginalized average attentional network (MAAN) to suppress the dominant response of the most salient regions in a principled manner. The MAAN employs a novel marginalized average aggregation (MAA) module and learns a set of latent discriminative probabilities in an end-to-end fashion. MAA samples multiple subsets from the video snippet features according to a set of latent discriminative probabilities and takes the expectation over all the averaged subset features. Theoretically, we prove that the MAA module with learned latent discriminative probabilities successfully reduces the difference in responses between the most salient regions and the others. Therefore, MAAN is able to generate better class activation sequences and identify dense and integral action regions in the videos. Moreover, we propose a fast algorithm to reduce the complexity of constructing MAA from $O(2^T)$ to $O(T^2)$. Extensive experiments on two large-scale video datasets show that our MAAN achieves a superior performance on weakly-supervised temporal action localization. Weakly-supervised temporal action localization has been of interest to the community recently. The setting is to train a model with solely video-level class labels, and to predict both the class and the temporal boundary of each action instance at the test time. The major challenge in the weakly-supervised localization problem is to find the right way to express and infer the underlying location information with only the video-level class labels. Traditionally, this is achieved by explicitly sampling several possible instances with different locations and durations BID2 BID11. The instance-level classifiers would then be trained through multiple instances learning BID4 BID40 or curriculum learning BID1 ). However, the length of actions and videos varies too much such that the number of instance proposals for each video varies a lot and it can also be huge. As a , traditional methods based on instance proposals become infeasible in many cases. Recent research, however, has pivoted to acquire the location information by generating the class activation sequence (CAS) directly BID17, which produces the classification score sequence of being each action for each snippet over time. The CAS along the 1D temporal dimension for a video is inspired by the class activation map (CAM) BID46 BID19 BID18 in weakly-supervised object detection. The CAM-based models have shown that despite being trained on image-level labels, convolutional neural networks (CNNs) have the remarkable ability to localize objects. Similar to object detection, the basic idea behind CAS-based methods for action localization in the training is to sample the non-overlapping snippets from a video, then to aggregate the snippet-level features into a video-level feature, and finally to yield a video-level class prediction. During testing, the model generates a CAS for each class that identifies the discriminative action regions, and then applies a threshold on the CAS to localize each action instance in terms of the start time and the end time. In CAS-based methods, the feature aggregator that aggregates multiple snippet-level features into a video-level feature is the critical building block of weakly-supervised neural networks. A model's ability to capture the location information of an action is primarily determined by the design of the aggregators. While using the global average pooling over a full image or across the video snippets has shown great promise in identifying the discriminative regions BID46 BID19 BID18, treating each pixel or snippet equally loses the opportunity to benefit from several more essential parts. Some recent works BID17 BID49 have tried to learn attentional weights for different snippets to compute a weighted sum as the aggregated feature. However, they suffer from the weights being easily dominated by only a few most salient snippets. In general, models trained with only video-level class labels tend to be easily responsive to small and sparse discriminative regions from the snippets of interest. This deviates from the objective of the localization task that is to locate dense and integral regions for each entire action. To mitigate this gap and reduce the effect of the domination by the most salient regions, several heuristic tricks have been proposed to apply to existing models. For example, BID35 BID44 attempt to heuristically erase the most salient regions predicted by the model which are currently being mined, and force the network to attend other salient regions in the remaining regions by forwarding the model several times. However, the heuristic multiple-run model is not end-to-end trainable. It is the ensemble of multiple-run mined regions but not the single model's own ability that learns the entire action regions. "Hide-and-seek" BID28 randomly masks out some regions of the input during training, enforcing the model to localize other salient regions when the most salient regions happen to be masked out. However, all the input regions are masked out with the same probability due to the uniform prior, and it is very likely that most of the time it is the that is being masked out. A detailed discussion about related works can be found in Appendix D.To this end, we propose the marginalized average attentional network (MAAN) to alleviate the issue raised by the domination of the most salient region in an end-to-end fashion for weakly-supervised action localization. Specifically, MAAN suppresses the action prediction response of the most salient regions by employing marginalized average aggregation (MAA) and learning the latent discriminative probability in a principled manner. Unlike the previous attentional pooling aggregator which calculates the weighted sum with attention weights, MAA first samples a subset of features according to their latent discriminative probabilities, and then calculates the average of these sampled features. Finally, MAA takes the expectation (marginalization) of the average aggregated subset features over all the possible subsets to achieve the final aggregation. As a , MAA not only alleviates the domination by the most salient regions, but also maintains the scale of the aggregated feature within a reasonable range. We theoretically prove that, with the MAA, the learned latent discriminative probability indeed reduces the difference of response between the most salient regions and the others. Therefore, MAAN can identify more dense and integral regions for each action. Moreover, since enumerating all the possible subsets is exponentially expensive, we further propose a fast iterative algorithm to reduce the complexity of the expectation calculation procedure and provide a theoretical analysis. Furthermore, MAAN is easy to train in an end-to-end fashion since all the components of the network are differentiable. Extensive experiments on two large-scale video datasets show that MAAN consistently outperforms the baseline models and achieves superior performance on weakly-supervised temporal action localization. In summary, our main contributions include: a novel end-to-end trainable marginalized average attentional network (MAAN) with a marginalized average aggregation (MAA) module in the weaklysupervised setting; theoretical analysis of the properties of MAA and an explanation of the reasons MAAN alleviates the issue raised by the domination of the most salient regions; a fast iterative algorithm that can effectively reduce the computational complexity of MAA; and a superior performance on two benchmark video datasets, THUMOS14 and ActivityNet1.3, on the weakly-supervised temporal action localization. incorporates MAA, and introduce the corresponding inference process on weakly-supervised temporal action localization in Sec. 2.4. Let {x 1, x 2, · · · x T} denote the set of snippet-level features to be aggregated, where x t ∈ R m is the m dimensional feature representation extracted from a video snippet centered at time t, and T is the total number of sampled video snippets. The conventional attentional weighted sum pooling aggregates the input snippet-level features into a video-level representation x. Denote the set of attentional weights corresponding to the snippet-level features as {λ 1, λ 2, · · · λ T}, where λ t is a scalar attentional weight for x t. Then the aggregated video-level representation is given by DISPLAYFORM0 as illustrated in FIG0 (a). Different from the conventional aggregation mechanism, the proposed MAA module aggregates the features by firstly generating a set of binary indicators to determine whether a snippet should be sampled or not. The model then computes the average aggregation of these sampled snippet-level representations. Lastly, the model computes the expectation (marginalization) of the aggregated average feature for all the possible subsets, and obtains the proposed marginalized average aggregated feature. Formally, in the proposed MAA module, we first define a set of probabilities {p 1, p 2, · · · p T}, where each p t ∈ is a scalar corresponding to x t, similar to the notation λ t mentioned previously. We then sample a set of random variables {z 1, z 2, · · · z T}, where z t ∼ Bernoulli(p t), i.e., z t ∈ {0, 1} with probability P (z t = 1) = p t. The sampled set is used to represent the subset selection of snippet-level features, in which z t = 1 indicates x t is selected, otherwise not. Therefore, the average aggregation of the sampled subset of snipped-level representations is given by s = DISPLAYFORM1 z i, and our proposed aggregated feature, defined as the expectation of all the possible subset-level average aggregated representations, is given by DISPLAYFORM2 which is illustrated in FIG0 (b). Direct learning and prediction with the attention weights λ in Eq. in weakly-supervised action localization leads to an over-response in the most salient regions. The MAA in Eq. has two properties that alleviate the domination effect of the most salient regions. First, the partial order preservation property, i.e., the latent discriminative probabilities preserve the partial order with respect to their attention weights. Second, the dominant response suppression property, i.e., the differences in the latent discriminative probabilities between the most salient items and others are smaller than the differences between their attention weights. The partial order preservation property guarantees that it does not mix up the action and non-action snippets by assigning a high latent discriminative probability to a snippet with low response. The dominant response suppression property reduces the dominant effect of the most salient regions and encourages the identification of dense and more integral action regions. Formally, we present the two properties in Proposition 1 and Proposition 2, respectively. Detailed proofs can be found in Appendix A and Appendix B respectively. Proposition 1. Let z i ∼ Bernoulli(p i) for i ∈ {1, ..., T}. Then for T ≥ 2, Eq. holds true, and DISPLAYFORM0 where DISPLAYFORM1 Proposition 1 shows that the latent discriminative probabilities {p i} preserve the partial order of the attention weights {λ i}. This means that a large attention weight corresponds to a large discriminative probability, which guarantees that the latent discriminative probabilities preserve the ranking of the action prediction response. Eq. can be seen as a factorization of the attention weight λ i into the multiplication of two components, p i and c i, for i ∈ {1, ..., T}. p i is the latent discriminative probability related to the feature of snippet i itself. The factor c i captures the contextual information of snippet i from the other snippets. This factorization can be considered to be introducing structural information into the aggregation. Factor c i can be considered as performing a structural regularization for learning the latent discriminative probabilities p i for i ∈ {1, ..., T}, as well as for learning a more informative aggregation. DISPLAYFORM2 as an index set. Then I = ∅ and for ∀i ∈ I, ∀j ∈ {1, ..., T} inequality holds true. DISPLAYFORM3 The index set I can be viewed as the most salient features set. Proposition 2 shows that the difference between the normalized latent discriminative probabilities of the most salient regions and others is smaller than the difference between their attention weights. It means that the prediction for each snippet using the latent discriminative probability can reduce the gap between the most salient featuress and the others compared to conventional methods that are based on attention weights. Thus, MAAN suppresses the dominant responses of the most salient featuress and encourages it to identify dense and more integral action regions. Directly learning the attention weights λ leans to an over response to the most salient region in weakly-supervised temporal localization. Namely, the attention weights for only a few snippets are too large and dominate the others, while attention weights for most of the other snippets that also belong to the true action are underestimated. Proposition 2 shows that latent discriminative probabilities are able to reduce the gap between the most salient features and the others compared to the attention weights. Thus, by employing the latent discriminative probabilities for prediction instead of the attention weights, our method can alleviate the dominant effect of the most salient region in weakly-supervised temporal localization. Given a video containing T snippet-level representations, there are 2 T possible configurations for the subset selection. Directly summing up all the 2 T configurations to calculate x has a complexity of O(2 T). In order to reduce the exponential complexity, we propose an iterative method to calculate x with O(T 2) complexity. Let us denote the aggregated feature of {x 1, x 2, · · · x t} with length t as h t, and denote DISPLAYFORM0 z i for simplicity, then we have a set of and the aggregated feature of {x 1, x 2, · · · x T} can be obtained as x = h T. In Eq. FORMULA8, Z t is the summation of all the z i, which indicates the number of elements selected in the subset. Although there are 2 t distinct configurations for {z 1, z 2, · · · z t}, it has only t + 1 distinct values for Z t, i.e. 0, 1, · · ·, t. Therefore, we can divide all the 2 t distinct configurations into t + 1 groups, where the configurations sharing with the same Z t fall into the same group. Then the expectation h t can be calculated as the summation of the t + 1 parts. That is, DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 where the m t i, indicating the i th part of h t for group Z t = i, is shown in Eq. FORMULA11. DISPLAYFORM4 In order to calculate DISPLAYFORM5 The key idea here is that m The latter case is also related to the probability P (Z t = i − 1). By denoting q t i−1 = P (Z t = i − 1) for simplicity, we can obtain m t+1 i as a function of several elements: DISPLAYFORM6 Similarly, the computation of q t+1 i = P (Z t+1 = i) comes from two cases: the probability of selecting i − 1 items from the first t items and selecting the (t + 1) th item, i.e., q t i−1 p t+1; and the probability of selecting i items all from the first t items and not selecting the (t + 1) th item, i.e., DISPLAYFORM7 We derive the function of m t+1 iand q t+1 i in Proposition 3. Detailed proofs can be found in Appendix C. DISPLAYFORM8 i ∈ {0, 1, · · ·, t + 1} can be obtained recurrently by Eq. FORMULA16 and Eq. FORMULA17. DISPLAYFORM9 DISPLAYFORM10 DISPLAYFORM11 Proposition 3 provides a recurrent formula to calculate m t i. With this recurrent formula, we calculate the aggregation h T by iteratively calculating m t i from i = 1 to t and t = 1 to T. Therefore, we can obtain the aggregated feature of {x 1, DISPLAYFORM12 The iterative computation procedure is summarized in Algorithm 1 in Appendix E. The time complexity is O(T 2).With the fast iterative algorithm in Algorithm 1, the MAA becomes practical for end-to-end training. A demonstration of the computation graph for q DISPLAYFORM13 Network Architecture: We now describe the network architecture that employs the MAA module described above for weakly-supervised temporal action localization. We start from a previous stateof-the-art base architecture, the sparse temporal pooling network (STPN) BID17. As shown in FIG4, it first divides the input video into several non-overlapped snippets and extracts the I3D feature for each snippet. Each snippet-level feature is then fed to an attention module to generate an attention weight between 0 and 1. STPN then uses a feature aggregator to calculate a weighted sum of the snippet-level features with these class-agnostic attention weights to create a video-level representation, as shown on the left in FIG5. The video-level representation is then passed through an FC layer followed by a sigmoid layer to obtain class scores. Our MAAN uses the attention module to generate the latent discriminative probability p t and replaces the feature aggregator from the weighted sum aggregation by the proposed marginalized average aggregation, which is demonstrated on the right in FIG5 Training with video-level class labels: Formally, the model first performs aggregation of the snippet-level features (i.e. DISPLAYFORM0 Then, it applies a logistic regression layer (FC layer + sigmoid) to output video-level classification prediction probability. Specifically, the prediction probability for class c ∈ {1, 2, · · · C} is parameterized as σ c j = σ(w c x j), where x j is the aggregated feature for video j ∈ {1, ..., N}. Suppose each video x j is i.i.d and each action class is independent from the other, the negative log-likelihood function (cross-entropy loss) is given as follows: DISPLAYFORM1 where y c j ∈ {0, 1} is the ground-truth video-level label for class c happening in video j and W = [w 1, ..., w C].Temporal Action Localization: Let s c = w c x be the video-level action prediction score, and σ(s c) = σ(w c x) be the video-level action prediction probability. In STPN, asx = T t=1 λ t x t, the s c can be rewritten as: DISPLAYFORM2 In STPN, the prediction score of snippet t for action class c in a video is defined as: DISPLAYFORM3 where σ(·) denotes the sigmoid function. In MAAN, asx = E[DISPLAYFORM4, according to Proposition 1, the s c can be rewritten as: DISPLAYFORM5 The latent discriminative probability p t corresponds to the class-agnostic attention weight for snippet t. According to Proposition 1 and Proposition 2, c t does not relate to snippet t, but captures the context of other snippets. w c corresponds to the class-specific weights for action class c for all the snippets, and w c x t indicates the relevance of snippet t to class c. To generate temporal proposals, we compute the prediction score of snippet t belonging to action class c in a video as: DISPLAYFORM6 We denote the s c = (s as the class activation sequence (CAS) for class c. Similar to STPN, the threshold is applied to the CAS for each class to extract the one-dimensional connected components to generate its temporal proposals. We then perform non-maximum suppression among temporal proposals of each class independently to remove highly overlapped detections. Compared to STPN (Eq. FORMULA0), MAAN (Eq. FORMULA0) employs the latent discriminative probability p t instead of directly using the attention weight λ t (equivalent to c t p t) for prediction. Proposition 2 suggests that MAAN can suppress the dominant response s c t compared to STPN. Thus, MAAN is more likely to achieve a better performance in weakly-supervised temporal action localization. This section discusses the experiments on the weakly-supervised temporal action localization problem, which is our main focus. We have also extended our algorithm on addressing the weakly-supervised image object detection problem and the relevant experiments are presented in Appendix F. Datasets. We evaluate MAAN on two popular action localization benchmark datasets, THU-MOS14 BID10 and ActivityNet1.3 BID8. THUMOS14 contains 20 action classes for the temporal action localization task, which consists of 200 untrimmed videos (3,027 action instances) in the validation set and 212 untrimmed videos (3,358 action instances) in the test set. Following standard practice, we train the models on the validation set without using the temporal annotations and evaluate them on the test set. ActivityNet1.3 is a large-scale video benchmark for action detection which covers a wide range of complex human activities. It provides samples from 200 activity classes with an average of 137 untrimmed videos per class and 1.41 activity instances per video, for a total of 849 video hours. This dataset contains 10,024 training videos, 4,926 validation videos and 5,044 test videos. In the experiments, we train the models on the training videos and test on the validation videos. Evaluation Metrics. We follow the standard evaluation metric by reporting mean average precision (mAP) values at several different levels of intersection over union (IoU) thresholds. We use the benchmarking code provided by ActivityNet 1 to evaluate the models. Implementation Details. We use two-stream I3D networks pre-trained on the Kinetics dataset BID12 ) to extract the snippet-level feature vectors for each video. All the videos are divided into sets of non-overlapping video snippets. Each snippet contains 16 consecutive frames or optical flow maps. We input each 16 stacked RGB frames or flow maps into the I3D RGB or flow models to extract the corresponding 1024 dimensional feature vectors. Due to the various lengths of the videos, in the training, we uniformly divide each video into T non-overlapped segments, and randomly sample one snippet from each segment. Therefore, we sample T snippets for each video as the input of the model for training. We set T to 20 in our MAAN model. The attention module in FIG4 consists of an FC layer of 1024 × 256, a LeakyReLU layer, an FC layer of 256 × 1, and a sigmoid non-linear activation, to generate the latent discriminative probability p t. We pass the aggregated video-level representation through an FC layer of 1024 × C followed by a sigmoid activation to obtain class scores. We use the ADAM optimizer BID14 with an initial learning rate of 5 × 10 −4 to optimize network parameters. At the test time, we first reject classes whose video-level probabilities are below 0.1. We then forward all the snippets of the video to generate the CAS for the remaining classes. We generate the temporal proposals by cutting the CAS with a threshold th. The combination ratio of two-stream modalities is set to 0.5 and 0.5. Our algorithm is implemented in PyTorch 2. We run all the experiments on a single NVIDIA Tesla M40 GPU with a 24 GB memory. We first compare our MAAN model on the THUMOS14 dataset with several baseline models that use different feature aggregators in FIG4 to gain some basic understanding of the behavior of our proposed MAA. The descriptions of the four baseline models are listed below. STPN. It employs the weighed sum aggregationx = T t=1 λ t x t to generate the video-level representation. Dropout. It explicitly performs dropout sampling with dropout probability p = 0.5 in STPN to obtain the video-level representation,x = T t=1 r t λ t x t, r t ∼ Bernoulli(0.5). We test all the models with the cutting threshold th as 0.2 of the max value of the CAS. We compare the detection average precision (%) at IoU = [0.1 : 0.1 : 0.9] and the video-level classification mean average precision (%) (denoted as Cls mAP) on the test set in TAB0. From TAB0, we can observe that although all the methods achieve a similar video-level classification mAP, their localization performances vary a lot. It shows that achieving a good video-level classification performance cannot guarantee obtaining a good snippet-level localization performance because the former only requires the correct prediction of the existence of an action, while the latter requires the correct prediction of both its existence and its duration and location. Moreover, TAB0 demonstrates that MAAN consistently outperforms all the baseline models at different levels of IoUs in the weakly-supervised temporal localization task. Both the "Norm" and "SoftmaxNorm" are the normalized weighted average aggregation. However, the "SoftmaxNorm" performs the worst, because the softmax function over-amplifies the weight of the most salient snippet. As a , it tends to identify very few discriminative snippets and obtains sparse and non-integral localization. The "Norm" also performs worse than our MAAN. It is the normalized weighted average over the snippet-level representation, while MAAN can be considered as the normalized weighted average (expectation) over the subsetlevel representation. Therefore, MAAN encourages the identification of dense and integral action segments as compared to "Norm" which encourages the identification of only several discriminative snippets. MAAN works better than "Dropout" because "Dropout" randomly drops out the snippets with different attention weights by uniform probabilities. At each iteration, the scale of the aggregated feature varies a lot, however, MAAN samples with the learnable latent discriminative probability and conducts the expectation of keeping the scale of the aggregated feature stable. Compared to STPN, MAAN also achieves superior . MAAN implicitly factorizes the attention weight into c t p t, where p t learns the latent discriminative probability of the current snippet, and c t captures the contextual information and regularizes the network to learn a more informative aggregation. The properties of MAA disallow the predicted class activation sequences to concentrate on the most salient regions. The quantitative show the effectiveness of the MAA feature aggregator. The temporal CAS generated by MAAN can cover large and dense regions to obtain more accurate action segments. In the example in FIG8, MAAN can discover almost all the actions that are annotated in the ground-truth; however, the STPN have missed several action segments, and also tends to only output the more salient regions in each action segment. Other methods are much sparser compared to MAAN. The first row of FIG8 shows several action segments in red and in green, corresponding to action segments that are relatively difficult and easy to be localized, respectively. We can see that all the easily-localized segments contain the whole person who is performing the "HammerThrow" action, while the difficultly-localized segments contain only a part of the person or the action. Our MAAN can successfully localize the easy segments as well as the difficult segments; however, all the other methods fail on the difficult ones. It shows that MAAN can identify several dense and integral action regions other than only the most discriminative region which is identified by the other methods. We also compare our model with the state-of-the-art action localization approaches on the THU-MOS14 dataset. The numerical are summarized in TAB1. We include both fully and weakly-supervised learning, as in BID17. As shown in TAB1, our implemented STPN performs slightly better than the reported in the original paper BID17. From TAB1, our proposed MAAN outperforms the STPN and most of the existing weakly-supervised action localization approaches. Furthermore, our model still presents competitive compared with several recent fully-supervised approaches even when trained with only video-level labels. We train the MAAN model on the ActivityNet1.3 training set and compare our performance with the recent state-of-the-art approaches on the validation set in TAB3. The action segment in ActivityNet is usually much longer than that of THUMOS14 and occupies a larger percentage of a video. We use a set of thresholds, which are [0.2, 0.15, 0.1, 0.05] of the max value of the CAS, to generate the proposals from the one-dimensional CAS. As shown in TAB3, with the set of thresholds, our implemented STPN performs slightly better than the reported in the original paper (Nguyen BID23 47.7 43.5 36.3 28.7 19.0 10.3 5.3 --Yeung et al. BID38 48.9 44.0 36.0 26.4 17.1 ----Yuan et al. BID39 51.4 42.6 33.6 26.1 18.8 ----Shou et al. BID24 --40.1 29.4 23.3 13.1 7.9 --Yuan et al. BID41 51.0 45.2 36.5 27.8 17.8 ----Xu et al. BID37 54.5 51.5 44.8 35.6 28.9 ----Zhao et al. 66.0 59.4 51.9 41.0 29.8 ---- Wang et al. 44.4 37.7 28.2 21.1 13.7 ----Singh & Lee BID28 36.4 27.8 19.5 12.7 6.8 ----STPN BID17 BID34 45.1 4.1 0.0 Shou et al. BID24 45.3 26.0 0.2 Xiong et al. 39.1 23.5 5.5Weakly-supervised STPN BID17 29.3 16.9 2.6 STPN BID17, 2018). With the same threshold and experimental setting, our proposed MAAN model outperforms the STPN approach on the large-scale ActivityNet1.3. Similar to THUMOS14, our model also achieves good that are close to some of the fully-supervised approaches. We have proposed the marginalized average attentional network (MAAN) for weakly-supervised temporal action localization. MAAN employs a novel marginalized average aggregation (MAA) operation to encourage the network to identify the dense and integral action segments and is trained in an end-to-end fashion. Theoretically, we have proved that MAA reduces the gap between the most discriminant regions in the video to the others, and thus MAAN generates better class activation sequences to infer the action locations. We have also proposed a fast algorithm to reduce the computation complexity of MAA. Our proposed MAAN achieves superior performance on both the THUMOS14 and the ActivityNet1.3 datasets on weakly-supervised temporal action localization tasks compared to current state-of-the-art methods. We thank our anonymous reviewers for their helpful feedback and suggestions. Prof. Ivor W. Tsang was supported by ARC FT130100746, ARC LP150100671, and DP180100106.A PROOF OF PROPOSITION 1 Proof. DISPLAYFORM0 In addition, DISPLAYFORM1 Thus, we achieve DISPLAYFORM2 A DISPLAYFORM3 where 1(·) denotes the indicator function. We achieve Eq. FORMULA2 by partitioning the summation into t + 1 groups. Terms belonging to group i have DISPLAYFORM4, and we achieve Eq.. We now give the proof of the recurrent formula of Eq. DISPLAYFORM0 Proof. DISPLAYFORM1 Then, we have DISPLAYFORM2 Since DISPLAYFORM3 C.3 PROOF OF RECURRENT FORMULA OF q t+1 iWe present the proof of Eq. DISPLAYFORM4 Proof. DISPLAYFORM5 = z1,z2,···zt,zt+1 DISPLAYFORM6 D RELATED WORK Video Action Analysis. Researchers have developed quite a few deep network models for video action analysis. Two-stream networks BID26 and 3D convolutional neural networks (C3D) BID29 are popular solutions to learn video representations and these techniques, including their variations, are extensively used for video action analysis. Recently, a combination of two-stream networks and 3D convolutions, referred to as I3D, was proposed as a generic video representation learning method, and served as an effective backbone network in various video analysis tasks such as recognition, localization BID23, and weakly-supervised learning.Weakly-Supervised Temporal Action Localization. There are only a few approaches based on weakly-supervised learning that rely solely on video-level class labels to localize actions in the temporal domain. Wang et al. proposed a UntrimmedNet framework, where two softmax functions are applied across class labels and proposals to perform action classification and detect important temporal segments, respectively. However, using the softmax function across proposals may not be effective for identifying multiple instances. Singh et al. BID28 designed a Hide-and-Seek model to randomly hide some regions in a video during training and force the network to seek other relevant regions. However, the randomly hiding operation, as a data augmentation, cannot guarantee whether it is the action region or the region that is hidden during training, especially when the dropout probabilities for all the regions are the same. Nguyen et al. BID17 proposed a sparse temporal pooling network (STPN) to identify a sparse set of key segments associated with the actions through attention-based temporal pooling of video segments. However, the sparse constraint may force the network to focus on very few segments and lead to incomplete detection. In order to prevent the model from focusing only on the most salient regions, we are inspired to propose the MAAN model to explicitly take the expectation with respect to the average aggregated features of all the sampled subsets from the video. Feature Aggregators. Learning discriminative localization representations with only video-level class labels requires the feature aggregation operation to turn multiple snippet-level representations into a video-level representation for classification. The feature aggregation mechanism is widely adopted in the deep learning literature and a variety of scenarios, for example, neural machine translation BID0, visual question answering BID9, and so on. However, most of these cases belong to fully-supervised learning where the goal is to learn a model that attends the most relevant features given the supervision information corresponding to the task directly. Many variant feature aggregators have been proposed, ranging from nonparametric max pooling and average pooling, to parametric hard attention BID6, soft attention BID30 BID22, second-order pooling BID5 BID15, structured attention BID13 BID16, graph aggregators BID43 BID7, and so on. Different from the fullysupervised setting where the feature aggregator is designed for the corresponding tasks, we develop a feature aggregator that is trained only with class labels, and then to be used to predict the dense action locations for test data. Different from the heuristic approaches BID35 BID44 which can be considered as a kind of hard-code attention by erasing some regions with a hand-crafted threshold, we introduce the end-to-end differentiable marginalized average aggregation which incorporates learnable latent discriminative probabilities into the learning process. E MARGINALIZED AVERAGE AGGREGATION We also evaluate the proposed model on the weakly-supervised object localization task. For weaklysupervised object localization, we are given a set of images in which each image is labeled only with its category label. The goal is to learn a model to predict both the category label as well as the bounding box for the objects in a new test image. Based on the model in BID46 (denoted as CAM model), we replace the global average pooling feature aggregator with other kinds of feature aggregator, such as the weighted sum pooling and the proposed MAA by extending the original 1D temporal version in temporal action localization into a 2D spatial version. We denote the model with weighted sum pooling as the weighted-CAM model. For the weighted-CAM model and the proposed MAAN model, we use an attention module to generate the attention weight λ in STPN or the latent discriminative probability p in MAAN. The attention module consists of a 2D convolutional layer of kernel size 1 × 1, stride 1 with 256 units, a LeakyReLU layer, a 2D convolutional layer of kernel size 1 × 1, stride 1 with 1 unit, and a sigmoid non-linear activation. We evaluate the weakly-supervised localization accuracy of the proposed model on the CUB-200-2011 dataset BID31. The CUB-200-2011 dataset has 11,788 images of 200 categories with 5,994 images for training and 5,794 for testing. We leverage the localization metric suggested by BID21 for comparison. This metric computes the percentage of images that is misclassified or with bounding boxes with less than 50% IoU with the groundtruth as the localization error. We compare our MAA aggregator (MAAN) with the weighted sum pooling (weighted-CAM) and global average pooling (CAM BID48). For MAAN and weighted-CAM, we pool the convolutional feature for aggregation into two different sizes, 4 × 4 and 7 × 7. We fix all other factors (e.g. network structure, hyper-parameters, optimizer), except for the feature aggregators to evaluate the models. The localization errors for different methods are presented in TAB5, where the GoogLeNet-GAP is the CAM model. Our method outperforms GoogLeNet-GAP by 5.06% in a Top-1 error. Meanwhile, MAAN achieves consistently lower localization error than weighted-CAM on the two learning schemes. It demonstrates that the proposed MAAN can improve the localization performance in the weakly-supervised setting. Moreover, both MAAN and weighted-CAM obtain smaller localization error when employing the 7 × 7 learning scheme than the 4 × 4 learning scheme. FIG11 visualizes the heat maps and localization bounding boxes obtained by all the compared methods. The object localization heat maps generated by the proposed MAAN can cover larger object regions and obtain more accurate bounding boxes.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A novel marginalized average attentional network for weakly-supervised temporal action localization

Represent the TLDR for finding the corresponding scientific article.

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Represent the scientific article to find a short TLDR

Question: Deep image prior (DIP), which utilizes a deep convolutional network (ConvNet) structure itself as an image prior, has attracted huge attentions in computer vision community. It empirically shows the effectiveness of ConvNet structure for various image restoration applications. However, why the DIP works so well is still unknown, and why convolution operation is essential for image reconstruction or enhancement is not very clear. In this study, we tackle these questions. The proposed approach is dividing the convolution into ``delay-embedding'' and ``transformation (\ie encoder-decoder)'', and proposing a simple, but essential, image/tensor modeling method which is closely related to dynamical systems and self-similarity. The proposed method named as manifold modeling in embedded space (MMES) is implemented by using a novel denoising-auto-encoder in combination with multi-way delay-embedding transform. In spite of its simplicity, the image/tensor completion and super-resolution of MMES are quite similar even competitive to DIP in our extensive experiments, and these would help us for reinterpreting/characterizing the DIP from a perspective of ``low-dimensional patch-manifold prior''. The most important piece of information for image/tensor restoration would be the "prior" which usually converts the optimization problems from ill-posed to well-posed, and/or gives some robustness for specific noises and outliers. Many priors were studied in computer science problems such as low-rank representation (; ; ;), smoothness (; ;), sparseness , non-negativity , statistical independence , and so on. Particularly in today's computer vision problems, total variation (TV) , low-rank representation (; ; ;), and non-local similarity priors are often used for image modeling. These priors can be obtained by analyzing basic properties of natural images, and categorized as "unsupervised image modeling". By contrast, the deep image prior (DIP) has been come from a part of "supervised" or "data-driven" image modeling framework (i.e., deep learning) although the DIP itself is one of the state-of-the-art unsupervised image restoration methods. The method of DIP can be simply explained to only optimize an untrained (i.e., randomly initialized) fully convolutional generator network (ConvNet) for minimizing squares loss between its generated image and an observed image (e.g., noisy image), and stop the optimization before the overfitting. explained the reason why a high-capacity ConvNet can be used as a prior by the following statement: Network resists "bad" solutions and descends much more quickly towards naturally-looking images, and its phenomenon of "impedance of ConvNet" was confirmed by toy experiments. However, most researchers could not be fully convinced from only above explanation because it is just a part of whole. One of the essential questions is why is it ConvNet? or in more practical perspective, to explain what is "priors in DIP" with simple and clear words (like smoothness, sparseness, low-rank etc) is very important. In this study, we tackle the question why ConvNet is essential as an image prior, and try to translate the "deep image prior" with words. For this purpose, we divide the convolution operation into "embedding" and "transformation" (see Fig. 9 in Appendix). Here, the "embedding" stands for delay/shift-embedding (i.e., Hankelization) which is a copy/duplication operation of image-patches by sliding window of patch size (τ, τ). The embedding/Hankelization is a preprocessing to capture the delay/shift-invariant feature (e.g., non-local similarity) of signals/images. This "transformation" is basically linear transformation in a simple convolution operation, and it also indicates some nonlinear transformation from the ConvNet perspective. To simplify the complicated "encoder-decoder" structure of ConvNet used in DIP, we consider the following network structure: Embedding H (linear), encoding φ r (non-linear), decoding ψ r (non-linear), and backward embedding H † (linear) (see Fig. 1). Note that its encoder-decoder part (φ r, ψ r) is just a simple multi-layer perceptron along the filter domain (i.e., manifold learning), and it is sandwitched between forward and backward embedding (H, H †). Hence, the proposed network can be characterized by Manifold Modeling in Embedded Space (MMES). The proposed MMES is designed as simple as possible while keeping a essential ConvNet structure. Some parameters τ and r in MMES are corresponded with a kernel size and a filter size in ConvNet. When we set the horizontal dimension of hidden tensor L with r, each τ 2 -dimensional fiber in H, which is a vectorization of each (τ, τ)-patch of an input image, is encoded into r-dimensional space. Note that the volume of hidden tensor L looks to be larger than that of input/output image, but representation ability of L is much lower than input/output image space since the first/last tensor (H,H) must have Hankel structure (i.e., its representation ability is equivalent to image) and the hidden tensor L is reduced to lower dimensions from H. Here, we assume r < τ 2, and its lowdimensionality indicates the existence of similar (τ, τ)-patches (i.e., self-similarity) in the image, and it would provide some "impedance" which passes self-similar patches and resist/ignore others. Each fiber of Hidden tensor L represents a coordinate on the patch-manifold of image. It should be noted that the MMES network is a special case of deep neural networks. In fact, the proposed MMES can be considered as a new kind of auto-encoder (AE) in which convolution operations have been replaced by Hankelization in pre-processing and post-processing. Compared with ConvNet, the forward and backward embedding operations can be implemented by convolution and transposed convolution with one-hot-filters (see Fig. 12 in Appendix for details). Note that the encoder-decoder part can be implemented by multiple convolution layers with kernel size and non-linear activations. In our model, we do not use convolution explicitly but just do linear transform and non-linear activation for "filter-domain" (i.e., horizontal axis of tensors in Fig. 1). The contributions in this study can be summarized as follow: A new and simple approach of image/tensor modeling is proposed which translates the ConvNet, effectiveness of the proposed method and similarity to the DIP are demonstrated in experiments, and most importantly, there is a prospect for interpreting/characterizing the DIP as "low-dimensional patch-manifold prior". Note that the idea of low-dimensional patch manifold itself has been proposed by and. Peyre had firstly formulated the patch manifold model of natural images and solve it by dictionary learning and manifold pursuit. Osher et al. formulated the regularization function to minimize dimension of patch manifold, and solved Laplace-Beltrami equation by point integral method. In comparison with these studies, we decrease the dimension of patch-manifold by utilizing AE shown in Fig. 1. A related technique, low-rank tensor modeling in embedded space, has been studied recently by. However, the modeling approaches here are different: multi-linear vs nonlinear manifold. Thus, our study would be interpreted as manifold version of in a perspective of tensor completion methods. Note that applied their model for only tensor completion task. By contrast, we investigate here tensor completion, super-resolution, and deconvolution tasks. Another related work is devoted to group sparse representation (GSR) (a). The GSR is roughly characterized as a combination of similar patch-grouping and sparse modeling which is similar to the combination of embedding and manifold-modeling. However, the computational cost of similar patch-grouping is obviously higher than embedding, and this task is naturally included in manifold learning. The main difference between above studies and our is the motivation: Essential and simple image modeling which can translate the ConvNet/DIP. The proposed MMES has many connections with ConvNet/DIP such as embedding, non-linear mapping, and the training with noise. From a perspective of DIP, there are several related works. First, the deep geometric prior utilises a good properties of a multi-layer perceptron for shape reconstruction problem which efficiently learn a smooth function from 2D space to 3D space. It helps us to understand DIP from a perspective of manifold learning. For example, it can be used for gray scale image reconstruction if an image is regarded as point could in 3D space (i, j, X ij). However, this may not provide the good image reconstruction like DIP, because it just smoothly interpolates a point cloud by surface like a Volonoi interpolation. Especially it can not provide a property of self-similarity in natural image. Second, deep decoder reconstructs natural images from noises by nonconvolutional networks which consists of linear channel/color transform, ReLU, channel/color normalization, and upsampling layers. In contrast that DIP uses over-parameterized network, deep decoder uses under-parameterized network and shows its ability of image reconstruction. Although deep decoder is a non-convolutional network, Authors emphasize the closed relationship between convolutional layers in DIP and upsampling layers in deep decoder. In this literature, Authors described "If there is no upsampling layer, then there is no notion of locality in the ant image" in deep decoder. It implies the "locality" is the essence of image model, and the convolution/upsampling layer provides it. Furthermore, the deep decoder has a close relationship with our MMES. Note that the MMES is originally/essentially has only decoder and inverse MDT (see Eq.), and the encoder is just used for satisfying Hankel structure. The decoder and inverse MDT in our MMES are respectively corresponding linear operation and upsampling layer in deep decoder. Moreover, concept of under-parameterization is also similar to our MMES. From this, we can say the essence of image model is the "locality", and its locality can be provided by "convolution", "upsampling", or "delay-embedding". This is why the image restoration from single image with deep convolutional networks has highly attentions which are called by zero-shot learning, internal learning, or self-supervised learning (; ; ; ; ; ;). Recently, two generative models: SinGAN and InGAN learned from only a single image, have been proposed. Key concept of both papers is to impose the constraint for local patches of image to be natural. From a perspective of the constraint for local patches of image, our MMES has closed relationship with these works. However, we explicitly impose a low-dimensional manifold constraint for local patches rather than adversarial training with patch discriminators. Here, on the contrary to Section 1, we start to explain the proposed method from the concept of MMES, and we systematically derive the MMES structure from it. Conceptually, the proposed tensor reconstruction method can be formulated by minimize where Y ∈ R J1×J2×···×J N is an observed corrupted tensor, X ∈ R I1×I2×···×I N is an estimated tensor, F: R I1×I2×···×I N → R J1×J2×···×J N is a linear operator which represents the observation system, H: R I1×I2×···×I N → R D×T is padding and Hankelization operator with sliding window of size (τ 1, τ 2, ..., τ N), and we impose each column of matrix H can be sampled from an r-dimensional manifold M r in D-dimensional Euclid space (see Appendix B for details). We have r ≤ D. For simplicity, we putted D:= n τ n and T:= n (I n +τ n −1). For tensor completion task, F:= P Ω is a projection operator onto support set Ω so that the missing elements are set to be zero. For superresolution task, F is a down-sampling operator of images/tensors. For deconvolution task, F is a convolution operator with some blur kernels. Fig. 2 shows the concept of proposed manifold modeling in case of image inpainting (i.e., N = 2). We minimize the distance between observation Y and reconstruction X with its support Ω, and all patches in X should be included in some restricted manifold M r. In other words, X is represented by the patch-manifold, and the property of the patch-manifold can be image priors. For example, low dimensionality of patch-manifold restricts the non-local similarity of images/tensors, and it would be related with "impedance" in DIP. We model X indirectly by designing the properties of patch-manifold M r. We consider an AE to define the r-dimensional manifold M r in (n τ n)-dimensional Euclidean space as follows: where. Note that, in general, the use of AE models is a widely accepted approach for manifold learning . The properties of the manifold M r are determined by the properties of φ r and ψ r. By employing multi-layer perceptrons (neural networks) for φ r and ψ r, encoder-decoder may provide a smooth manifold. In this section, we combine the conceptual formulation and the AE guided manifold constraint to derive a equivalent and more practical optimization problem. First, we redefine a tensor X as an output of generator: Algorithm 1 Optimization algorithm for tensor reconstruction input: where l t ∈ R r, and H † is a pseudo inverse of H. At this moment, X is a function of {l t} T t=1, however Hankel structure of matrix H can not be always guaranteed under the unconstrained condition of l t. For guaranteeing the Hankel structure of matrix H, we further transform it as follow: where we put A r: R D×T → R D×T as an operator which auto-encodes each column of a input matrix with (ψ r,φ r), and [g 1, g 2, ..., g T] as a matrix, which has Hankel structure and is transformed by Hankelization of some input tensor Z ∈ R I1×I2×···×I N. Note that Z is the most compact representation for Hankel matrix [g 1, g 2, ..., g T]. Eq. describes the MMES network shown in Fig. 1: H,φ r,ψ r and H † are respectively corresponding to forward embedding, encoding, decoding, and backward embedding, where encoder and decoder can be defined e.g. by multi-layer perceptrons (i.e., repetition of linear transformation and non-linear activation). F, where A r is an AE which defines the manifold M r. In this study, the AE/manifold is learned from an observed tensor Y itself, thus the optimization problem is finally formulated as where we refer respectively the first and second terms by a reconstruction loss and an auto-encoding loss, and λ > 0 is a trade-off parameter for balancing both losses. Optimization problem consists of two terms: a reconstruction loss, and an auto-encoding loss. Hyperparameter λ is set to balance both losses. Basically, λ should be large because auto-encoding loss should be zero. However, very large λ prohibits minimizing the reconstruction loss, and may lead to local optima. Therefore, we adjust gradually the value of λ in the optimization process. Algorithm 1 shows an optimization algorithm for tensor reconstruction and/or enhancement. For AE learning, we employs a strategy of denoising-auto-encoder (see Appendix in detail). Adaptation of λ is just an example, and it can be modified appropriately with data. Here, the trade-off parameter λ is adjusted for keeping L rec > L AE, but for no large gap between both losses. By exploiting the convolutional structure of H and H † (see Appendix B.1), the calculation flow of L rec and L AE can be easily implemented by using neural network libraries such as TensorFlow. We employed Adam optimizer for updating (Z, A r). Here, we show the selective experimental to demonstrate the close similarity and some slight differences between DIP and MMES. First, toy examples with a time-series signal and a gray-scale image were recovered by the proposed method to show its basic behaviors. Thereafter, we show the main by comparison with DIP and other selective methods on color-image inpainting, superresolution, and deconvolution tasks. Optional of optimization behavior, hyper-parameter sensitivity, and volumetric/3D image completion are shown in Appendix. In this section, we apply the proposed method into a toy example of signal recovery. Fig. 3 shows a of this experiment. A one-dimensional time-series signal is generated from Lorentz system, and corrupted by additive Gaussian noise, random missing, and three block occlusions. The corrupted signal was recovered by the subspace modeling , and the proposed manifold modeling in embedded space. Window size of delay-embedding was τ = 64, the lowest dimension of auto-encoder was r = 3, and additive noise standard deviation was set to σ = 0.05. Manifold modeling catched the structure of Lorentz attractor much better than subspace modeling. Similar patches are located near each other, and the smooth change of patterns can be observed. It implies the relationship between non-local similarity based methods (; ; ; a), and the manifold modeling (i.e., DAE) plays a key role of "patch-grouping" in the proposed method. The difference from the non-local similarity based approach is that the manifold modeling is "global" rather than "non-local" which finds similar patches of the target patch from its neighborhood area. In this section, we compare performance of the proposed method with several selected unsupervised image inpainting methods: low-rank tensor completion (HaLRTC) , parallel lowrank matrix factorization (TMac) , tubal nuclear norm regularization (tSVD) (b), Tucker decomposition with rank increment (Tucker inc.) , lowrank and total-variation (LRTV) regularization (;, smooth PARAFAC tensor completion (SPC) , GSR (a), multi-way delay embedding based Tucker modeling (MDT-Tucker) , and DIP . Implementation and detailed hyper-parameter settings are explained in Appendix. Basically, we carefully tuned the hyper-parameters for all methods to perform the best scores of peak-signal-tonoise ratio (PSNR) and structural similarity (SSIM). Fig. 5(a) shows the eight test images and averages of PSNR and SSIM for various missing ratio {50%, 70%, 90%, 95%, 99%} and for selective competitive methods. The proposed method is quite competitive with DIP. Fig. 6 shows the illustration of . The 99% of randomly selected voxels are removed from 3D-tensors, and the tensors were recovered by various methods. Basically low-rank priors (HaLRTC, TMac, tSVD, Tucker) could not recover such highly incomplete image. In piecewise smoothness prior (LRTV), over-smoothed images were reconstructed since the essential image properties could not be captured. There was a somewhat jump from them by SPC (i.e., smooth prior of basis functions in low-rank tensor decomposition). MDT-Tucker further improves it by exploiting the shift-invariant multi-linear basis. GSR nicely recovered the global pattern of images but details were insufficient. Finally, the reconstructed images by DIP and MMES recovered both global and local patterns of images. In this section, we compare the proposed method with selected unsupervised image super-resolution methods: Bicubic interpolation, GSR (a), ZSSR and DIP . Implementation and detailed hyper-parameter settings are explained in Appendix. Basically, we carefully tuned the hyper-parameters for all methods to perform the best scores of PSNR and SSIM. Fig. 5(b) shows values of PSNR and SSIM of the computer simulation . We used three color images, and six color images. Super resolution methods scaling up them from four or eight times down-scaled images of them with Lanczos2 kernels. According to this quantitative evaluation, bicubic interpolation was clearly worse than others. ZSSR worked well for up-scaling from, however the performances were substantially decreased for upscaling from. Basically, GSR, DIP, and MMES were very competitive. In detail, DIP was slightly better than GSR, and the proposed MMES was slightly better than DIP. More detailed PSNR/SSIM values are given by Table 3 in Appendix. Fig. 7 shows selected high resolution images reconstructed by four super-resolution methods. In general, bicubic method reconstructed blurred images and these were visually worse than others. GSR had smooth outlines in all images, but these were slightly blurred. ZSSR was weak for very low-resolution images. DIP reconstructed visually sharp images but these images had jagged artifacts along the diagonal lines. The proposed MMES reconstructed sharp and smooth outlines. In this section, we compare the proposed method with DIP for image deconvolution/deblurring task. Three color images are prepared and blurred by using three different Gaussian filters. For DIP we choose the best early stopping timing from {1000, 2000, ..., 10000} iterations. For MMES, we employed the fixed AE structure as [32τ 2, r, 32τ 2], and parameters as τ = 4, r = 16, and σ = 0.01 for all nine cases. Fig. 8 shows the reconstructed deblurring images by DIP and MMES. Tab. 1 shows the PSNR and SSIM values of these . We can see that the similarity of the methods qualitatively and quantitatively. It is well known that there is no mathematical definition of interpretability in machine learning and there is no one unique definition of interpretation. We understand the interpretability as a degree to which a human can consistently predict the model's or performance. The higher the interpretability of a deep learning model, the easier it is for someone to comprehend why certain performance or predictions or expected output can be achieved. We think that a model is better interpretable than another model if its performance or behaviors are easier for a human to comprehend than performance of the other models. The manifold learning and associated auto-encoder (AE) can be viewed as the generalized non-linear version of principal component analysis (PCA). In fact, manifold learning solves the key problem of dimensionality reduction very efficiently. In other words, manifold learning (modeling) is an approach to non-linear dimensionality reduction. Manifold modeling for this task are based on the idea that the dimensionality of many data sets is only artificially high. Although the patches of images (data points) consist of hundreds/thousands pixels, they may be represented as a function of only a few or quite limited number underlying parameters. That is, the patches are actually samples from a low-dimensional manifold that is embedded in a high-dimensional space. Manifold learning algorithms attempt to uncover these parameters in order to find a low dimensional representation of the images. In our MMES approach to solve the problem we applied original embedding via multi-way delay embedding transform (MDT or Hankelization). Our algorithm is based on the optimization of cost function and it works towards extracting the low-dimensional manifold that is used to describe the high-dimensional data. The manifold is described mathematically by Eq. and cost function is formulated by Eq.. As mentioned at introduction, reported an important phenomenon of noise impedance of ConvNet structures. Here, we provide a prospect for explaining the noise impedance in DIP through the MMES. Let us consider the sparse-land model, i.e. noise-free images are distributed along low-dimensional manifolds in the high-dimensional Euclidean space and images perturbed by noises thicken the manifolds (make the dimension of the manifolds higher). Under this model, the distribution of images can be assumed to be higher along the low-dimensional noise-free image manifolds. When we assume that the image patches are sampled from low-dimensional manifold like sparse-land model, it is difficult to put noisy patches on the low-dimensional manifold. Let us consider to fit the network for noisy images. In such case the fastest way for decreasing squared error (loss function) is to learn "similar patches" which often appear in a large set of image-patches. Note that finding similar image-patches for denoising is well-known problem solved, e.g., by BM3D algorithm, which find similar image patches by template matching. In contrast, our auto-encoder automatically maps similar-patches into close points on the low-dimensional manifold. When similar-patches have some noise, the low-dimensional representation tries to keep the common components of similar patches, while reducing the noise components. This has been proved by so that a (denoising) auto-encoder maps input image patches toward higher density portions in the image space. In other words, a (denoising) auto-encoder has kind of a force to reconstruct the low-dimensional patch manifold, and this is our rough explanation of noise impedance phenomenon. Although the proposed MMES and DIP are not completely equivalent, we see many analogies and similarities and we believe that our MMES model and associated learning algorithm give some new insight for DIP. A beautiful manifold representation of complicated signals in embedded space has been originally discovered in a study of dynamical system analysis (i.e., chaos analysis) for time-series signals . After this, many signal processing and computer vision applications have been studied but most methods have considered only linear approximation because of the difficulty of non-linear modeling (; ; ; ;). However nowadays, the study of non-linear/manifold modeling has been well progressed with deep learning, and it was successfully applied in this study. Interestingly, we could apply this non-linear system analysis not only for time-series signals but also natural color images and tensors (this is an extension from delay-embedding to multi-way shiftembedding). The best of our knowledge, this is the first study to apply Hankelization with AE into general tensor data reconstruction. MMES is a novel and simple image reconstruction model based on the low-dimensional patchmanifold prior which has many connections to ConvNet. We believe it helps us to understand how work ConvNet/DIP through MMES, and support to use DIP for various applications like tensor/image reconstruction or enhancement (; ; ;). Finally, we established bridges between quite different research areas such as the dynamical system analysis, the deep learning, and the tensor modeling. The proposed method is just a prototype and can be further improved by incorporating other methods such as regularizations, multi-scale extensions, and adversarial training. We can see the anti-diagonal elements of above matrix are equivalent. Such matrix is called as "Hankel matrix". For a two-dimensional array we consider unfold of it and inverse folding by unfold, and The point here is that we scan matrix elements column-wise manner. Hankelization of this twodimensional array (matrix) with τ = is given by scanning a matrix with local-window column-wise manner, and unfold and stack each local patch left-to-right. Thus, it is given as We can see that it is not a Hankel matrix. However, it is a "block Hankel matrix" in perspective of block matrix, a matrix that its elements are also matrices. We can see the block matrix itself is a Hankel matrix and all elements are Hankel matrices, too. Thus, Hankel matrix is a special case of block Hankel matrix in case of that all elements are scalar. In this paper, we say simply "Hankel structure" for block Hankel structure. Figure 9 shows an illustrative explanation of valid convolution which is decomposed into delayembedding/Hankelization and linear transformation. 1D valid convolution of f with kernel h = [h 1, h 2, h 3] can be provided by matrix-vector product of the Hankel matrix and h. In similar way, 2D valid convolution can be provided by matrix-vector product of the block Hankel matrix and unfolded kernel. Multiway-delay embedding transform (MDT) is a multi-way generalization of Hankelization proposed by. In , MDT is defined by using the multi-linear tensor product with multiple duplication matrices and tensor reshaping. Basically, we use the same operation, but a padding operation is added. Thus, the multiway-delay embedding used in this study is defined by where D×T is an unfolding operator which outputs a matrix from an input N -th order tensor, and S n ∈ R is a duplication matrix. Fig. 10 shows the duplication matrix with τ. For example, our Hankelization with reflection padding of f = [f 1, f 2, ..., f 7] with τ = 3 is given by Fig. 11 shows an example of our multiway-delay embedding in case of second order tensors. The overlapped patch grid is constructed by multi-linear tensor product with S n. Finally, all patches are splitted, lined up, and vectorized. The Moore-Penrose pseudo inverse of H is given by where, and trim τ = pad † τ is a trimming operator for removing (τ n −1) elements at start and end of each mode. Note that H † •H is an identity map, but H • H † is not, that is kind of a projection. Delay embedding and its pseudo inverse can be implemented by using convolution with all onehot-tensor windows of size (τ 1, τ 2, ..., τ N). The one-hot-tensor windows can be given by folding a D-dimensional identity matrix I D ∈ R D×D into I D ∈ R τ1×···×τ N ×D. Fig. 12 shows a calculation flow of multi-way delay embedding using convolution in a case of N = 2. Multi-linear tensor product is replaced with convolution with one-hot-tensor windows. Pseudo inverse of the convolution with padding is given by its adjoint operation, which is called as the "transposed convolution" in some neural network library, with trimming and simple scaling with Figure 10: Duplication matrix. In case that we have I columns, it consists of (I − τ + 1) identity matrices of size (τ, τ). In this section, we discuss how to design the neural network architecture of auto-encoder for restricting the manifold M r. The simplest way is controlling the value of r, and it directly restricts the dimensionality of latent space. There are many other possibilities: Tikhonov regularization , drop-out , denoising auto-encoder , variational auto-encoder (Diederik P), adversarial auto-encoder , alpha-GAN (, and so on. All methods have some perspective and promise, however the cost is not low. In this study, we select an attractive and fundamental one: "denoising auto-encoder"(DAE) . The DAE is attractive because it has a strong relationship with Tikhonov regularization , and decreases the entropy of data . Furthermore, learning with noise is also employed in the deep image prior. Finally, we designed an auto-encoder with controlling the dimension r and the standard deviation σ of additive zero-mean Gaussian noise. In case of multi-channel or color image recovery case, we use a special setting of generator network because spacial pattern of individual channels are similar and the patch-manifold can be shared. Fig. 14 shows an illustration of the auto-encoder shared version of MMES in a case of color image recovery. In this case, we put three channels of input and each channel input is embedded, independently. Then, three block Hankel matrices are concatenated, and auto-encoded simultaneously. Inverted three images are stacked as a color-image (third-order tensor), and finally color-transformed. The last color-transform can be implemented by convolution layer with kernel size, and it is also optimized as parameters. It should be noted that the input three channels are not necessary to correspond to RGB, but it would be optimized as some compact color-representation. Here, we explain detailed experimental settings in Section 4.2. In this section, we compared performance of the proposed method with several selected unsupervised image inpainting methods: low-rank tensor completion (HaLRTC) , parallel low-rank matrix factorization (TMac) , tubal nuclear norm regularization (tSVD) (b), Tucker decomposition with rank increment (Tucker inc.) , low-rank and total-variation (LRTV) regularization 2 (;, smooth PARAFAC tensor completion (SPC) 3 , GSR 4 (a), multi-way . For this experiments, hyper-parameters of all methods were tuned manually to perform the best peaksignal-to-noise ratio (PSNR) and for structural similarity (SSIM), although it would not be perfect. For DIP, we did not try the all network structures with various kernel sizes, filter sizes, and depth. We just employed "default architecture", which the details are available in supplemental material 7 of , and employed the best at the appropriate intermediate iterations in optimizations based on the value of PSNR. For the proposed MMES method, we adaptively selected the patch-size τ, and dimension r. Table 2 shows parameter settings of τ = [τ, τ] and r for MMES. Noise level of denoising auto-encoder was set as σ = 0.05 for all images. For auto-encoder, same architecture shown in Fig. 13 was employed. Initial learning rate of Adam optimizer was 0.01 and we decayed the learning rate with 0.98 every 100 iterations. The optimization was stopped after 20,000 iterations for each image. Here, we explain detailed experimental settings in Section 4.3. In this section, we compare performance of the proposed method with several selected unsupervised image super-resolution methods: bicubic interpolation, GSR 8 (a), ZSSR 9 and DIP . In this experiments, DIP was conducted with the best number of iterations from {1000, 2000, 3000, ..., 9000}. For four times (x4) up-scaling in MMES, we set τ = 6, r = 32, and σ = 0.1. For eight times (x8) up-scaling in MMES, we set τ = 6, r = 16, and σ = 0.1. For all images in MMES, the architecture of auto-encoder consists of three hidden layers with sizes of [8τ 2, r, 8τ 2]. We assumed the same Lanczos2 kernel for down-sampling system for all super-resolution methods. Tab. 3 shows values of PSNR and SSIM of the . We used three color images, and six color images. Super resolution methods scaling up them from four or eight times down-scaled images of them. According to this quantitative evaluation, bicubic interpolation was clearly worse than others. ZSSR was good for color images, however the performance were substantially decreased for color image. Basically, GSR, DIP, and MMES were very competitive. In detail, DIP was slightly better than GSR, and the proposed MMES was slightly better than DIP. For this experiment, we recovered 50% missing gray-scale image of'Lena'. We stopped the optimization algorithm after 20,000 iterations. Learning rate was set as 0.01, and we decayed the learning rate with 0.98 every 100 iterations. λ was adapted by Algorithm 1 every 10 iterations. Fig. 15 shows optimization behaviors of reconstructed image, reconstruction loss L rec, auto-encoding loss L DAE, and trade-off coefficient λ. By using trade-off adjustment, the reconstruction loss and the auto-encoding loss were intersected around 1,500 iterations, and both losses were jointly decreased after the intersection point. We evaluate the sensitivity of MMES with three hyper-parameters: r, σ, and τ. First, we fixed the patch-size as, and dimension r and noise standard deviation σ were varied. the reconstruction of a 99% missing image of'Lena' by the proposed method with different settings of (r, σ). The proposed method with very low dimension (r = 1) provided blurred , and the proposed method with very high dimension (r = 64) provided which have many peaks. Furthermore, some appropriate noise level (σ = 0.05) provides sharp and clean . For reference, Fig. 16 shows the difference of DIP optimized with and without noise. From both , the effects of learning with noise can be confirmed. Next, we fixed the noise level as σ = 0.05, and the patch-size were varied with some values of r. Fig. 18 shows the with various patch-size settings for recovering a 99% missing image. The patch sizes τ of or were appropriate for this case. Patch size is very important because it depends on the variety of patch patterns. If patch size is too large, then patch variations might expand and the structure of patch-manifold is complicated. By contrast, if patch size is too small, then the information obtained from the embedded matrix H is limited and the reconstruction becomes difficult in highly missing cases. The same problem might be occurred in all patch-based image reconstruction methods (; ; ; a). However, good patch sizes would be different for different images and types/levels of corruption, and the estimation of good patch size is an open problem. Multi-scale approach may reduce a part of this issue but the patch-size is still fixed or tuned as a hyper-parameter. In this section, we show the of MR-image/3D-tensor completion problem. The size of MR image is. We randomly remove 50%, 70%, and 90% voxels of the original MR-image and recover the missing MR-images by the proposed method and DIP. For DIP, we implemented the 3D version of default architecture in TensorFlow, but the number of filters of shallow layers were slightly reduced because of the GPU memory constraint. For the proposed method, 3D patch-size was set as τ =, the lowest dimension was r = 6, and noise level was σ = 0.05. Same architecture shown in Fig. 13 was employed. Fig. 19 shows reconstruction behavior of PSNR with final value of PSNR/SSIM in this experiment. From the values of PSNR and SSIM, the proposed MMES outperformed DIP in low-rate missing cases, and it is quite competitive in highly missing cases. The some degradation of DIP might be occurred by the insufficiency of filter sizes since much more filter sizes would be required for 3D ConvNet than 2D ConvNet. Moreover, computational times required for our MMES were significantly shorter than that of DIP in this tensor completion problem., r=4, r=8, r=16, r=32, r=48

Represent the short summary for finding the corresponding scientific article (long):

We propose a new auto-encoder incorporated with multiway delay-embedding transform toward interpreting deep image prior.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

It is shown that ResNet-type CNNs are a universal approximator and its expression ability is not worse than fully connected neural networks (FNNs) with a \textit{block-sparse} structure even if the size of each layer in the CNN is fixed.


Question: Represent the article for finding a single-sentence TLDR

Question: Federated learning is a recent advance in privacy protection. In this context, a trusted curator aggregates parameters optimized in decentralized fashion by multiple clients. The ing model is then distributed back to all clients, ultimately converging to a joint representative model without explicitly having to share the data. However, the protocol is vulnerable to differential attacks, which could originate from any party contributing during federated optimization. In such an attack, a client's contribution during training and information about their data set is revealed through analyzing the distributed model. We tackle this problem and propose an algorithm for client sided differential privacy preserving federated optimization. The aim is to hide clients' contributions during training, balancing the trade-off between privacy loss and model performance. Empirical studies suggest that given a sufficiently large number of participating clients, our proposed procedure can maintain client-level differential privacy at only a minor cost in model performance. Lately, the topic of security in machine learning is enjoying increased interest. This can be largely attributed to the success of big data in conjunction with deep learning and the urge for creating and processing ever larger data sets for data mining. However, with the emergence of more and more machine learning services becoming part of our daily lives, making use of our data, special measures must be taken to protect privacy. Unfortunately, anonymization alone often is not sufficient BID12; BID1 and standard machine learning approaches largely disregard privacy aspects and are susceptible to a variety of adversarial attacks BID11. In this regard, machine learning can be analyzed to recover private information about the participating user or employed data as well; BID16; BID3; BID6. BID2 propose a measure for to assess the memorization of privacy related data. All the aspects of privacy-preserving machine learning are aggravated when further restrictions apply such as a limited number of participating clients or restricted communication bandwidth such as mobile devices.In order to alleviate the need of explicitly sharing data for training machine learning models, decentralized approaches have been proposed, sometimes referred to as collaborative BID15 or federated learning BID9. In federated learning BID9 a model is learned by multiple clients in decentralized fashion. Learning is shifted to the clients and only learned parameters are centralized by a trusted curator. This curator then distributes an aggregated model back to the clients. However, this alone is not sufficent to preserve privacy. In BID14 it is shown that clients be identified in a federated learning setting by the model updates alone, necessitating further steps. Clients not revealing their data is an advance in privacy protection. However, when a model is learned in conventional way, its parameters reveal information about the data that was used during training. In order to solve this issue, the concept of differential privacy (dp) BID4 for learning algorithms was proposed by BID0. The aim is to ensure a learned model does not reveal whether a certain data point was used during training. We propose an algorithm that incorporates a dp-preserving mechanism into federated learning. However, opposed to BID0 we do not aim at protecting w.r.t. a single data point only. Rather, we want to ensure that a learned model does not reveal whether a client participated during decentralized training. This implies a client's whole data set is protected against differential attacks from other clients. Our main contributions: First, we show that a client's participation can be hidden while model performance is kept high in federated learning. We demonstrate that our proposed algorithm can achieve client level differential privacy at a minor loss in model performance. An independent study BID10, published at the same time, proposed a similar procedure for client level-dp. Experimental setups however differ and BID10 also includes elementlevel privacy measures. Second, we propose to dynamically adapt the dp-preserving mechanism during decentralized training. Empirical studies suggest that model performance is increased that way. This stands in contrast to latest advances in centralized training with differential privacy, were such adaptation was not beneficial. We can link this discrepancy to the fact that, compared to centralized learning, gradients in federated learning exhibit different sensibilities to noise and batch size throughout the course of training. In federated learning BID9, communication between curator and clients might be limited (e.g. mobile phones) and/or vulnerable to interception. The challenge of federated optimization is to learn a model with minimal information overhead between clients and curator. In addition, clients' data might be non-IID, unbalanced and massively distributed. The algorithm'federated averaging' recently proposed by BID9 A lot of research has been conducted in protecting differential privacy on data level when a model is learned in a centralized manner. This can be done by incorporating a dp-preserving randomized mechanism (e.g. the Gaussian mechanism) into the learning process. We use the same definition for differential privacy in randomized mechanisms as BID0: A randomized mechanism M: D → R, with domain D and range R satisfies (, δ)-differential privacy, if for any two adjacent inputs d, d ∈ D and for any subset of outputs S ⊆ R it holds that DISPLAYFORM0 In this definition, δ accounts for the probability that plain -differential privacy is broken. The Gaussian mechanism (GM) approximates a real valued function f: D → R with a differentially private mechanism. Specifically, a GM adds Gaussian noise calibrated to the functions data set sensitivity S f. This sensitivity is defined as the maximum of the absolute distance DISPLAYFORM1 In the following we assume that σ and are fixed and evaluate an inquiry to the GM about a single approximation of f (d). We can then bound the probability that -dp is broken according to: δ ≤ 5 4 exp(−(σ) 2 /2) (Theorem 3.22 in BID5). It should be noted that δ is accumulative and grows if the consecutive inquiries to the GM. Therefore, to protect privacy, an accountant keeps track of δ. Once a certain threshold for δ is reached, the GM shall not answer any new inquires. Recently, BID0 proposed a differentially private stochastic gradient descent algorithm (dp-SGD). dp-SGD works similar to mini-batch gradient descent but the gradient averaging step is approximated by a GM. In addition, the mini-batches are allocated through random sampling of the data. For being fixed, a privacy accountant keeps track of δ and stops training once a threshold is reached. Intuitively, this means training is stopped once the probability that the learned model reveals whether a certain data point is part of the training set exceeds a certain threshold. We propose to incorporate a randomized mechanism into federated learning. However, opposed to we do not aim at protecting a single data point's contribution in learning a model. Instead, we aim at protecting a whole client's data set. That is, we want to ensure that a learned model does not reveal whether a client participated during decentralized training while maintaining high model performance. In the framework of federated optimization BID9, the central curator averages client models (i.e. weight matrices) after each communication round. In our proposed algorithm, we will alter and approximate this averaging with a randomized mechanism. This is done to hide a single client's contribution within the aggregation and thus within the entire decentralized learning procedure. The randomized mechanism we use to approximate the average consists of:• Random sub-sampling (step 1 in FIG0): Let K be the total number of clients. In each communication round a random subset Z t of size m t ≤ K is sampled. The curator then distributes the central model w t to only these clients. The central model is optimized by the clients' on their data. The clients in Z t now hold distinct local models {w k} mt k=0. The difference between the optimized local model and the central model will be referred to as client k's update ∆w k = w k − w t. The updates are sent back to the central curator at the end of each communication round.• Distorting (step 3 and 4 in FIG0): A Gaussian mechanism is used to distort the sum of all updates. This requires knowledge about the set's sensitivity with respect to the summing operation. We can enforce a certain sensitivity by using scaled versions instead of the true updates: DISPLAYFORM0 ). Scaling ensures that the second norm is limited ∀k, w k 2 < S. The sensitivity of the scaled updates with respect to the summing operation is thus upper bounded by S. The GM now adds noise (scaled to sensitivity S) to the sum of all scaled updates. Dividing the GM's output by m t yields an approximation to the true average of all client's updates, while preventing leakage of crucial information about an individual. A new central model w t+1 is allocated by adding this approximation to the current central model w t. DISPLAYFORM1 Gaussian mechanism approximating sum of updatesWhen factorizing 1/m t into the Gaussian mechanism, we notice that the average's distortion is governed by the noise variance S 2 σ 2 /m. However, this distortion should not exceed a certain limit. Otherwise too much information from the sub-sampled average is destroyed by the added noise and there will not be any learning progress. GM and random sub-sampling are both randomized mechanisms. (Indeed, BID0 used exactly this kind of average approximation in dp-SGD. However, there it is used for gradient averaging, hiding a single data point's gradient at every iteration). Thus, σ and m also define the privacy loss incurred when the randomized mechanism provides an average approximation. In order to keep track of this privacy loss, we make use of the moments accountant as proposed by Abadi et al. BID0. This accounting method provides much tighter bounds on the incurred privacy loss than the standard composition theorem (3.14 in BID5). Each time the curator allocates a new model, the accountant evaluates δ given, σ and m. Training shall be Step 1: At each communication round t, m t out of total K clients are sampled uniformly at random. The central model w t is distributed to the sampled clients. Step 2: The selected clients optimize w t on their local data, leading to w k. Clients centralize their local updates: DISPLAYFORM2 Step 3: The updates are clipped such that their sensitivity can be upper bounded. The clipped updates are averaged. Step 4: The central model is updated adding the averaged, clipped updates and distorting them with Gaussian noise tuned to the sensitivity's upper bound. Having allocated the new central model, the procedure can be repeated. However, before starting step 1, a privacy accountant evaluates the privacy loss that would arise through performing another communication round. If that privacy loss is acceptable, a new round may start. stopped once δ reaches a certain threshold, i.e. the likelihood, that a clients contribution is revealed gets too high. The choice of a threshold for δ depends on the total amount of clients K. To ascertain that privacy for many is not preserved at the expense of revealing total information about a few, we have to ensure that δ 1 K, refer to chapter 2.3 for more details. Choosing S: When clipping the contributions, there is a trade-off. On the one hand, S should be chosen small such that the noise variance stays small. On the other hand, one wants to maintain as much of the original contributions as possible. Following a procedure proposed by BID0, in each communication round we calculate the median norm of all unclipped contributions and use this as the clipping bound S = median{w k} k∈Zt. We do not use a randomized mechanism for computing the median, which, strictly speaking, is a violation of privacy. However, the information leakage through the median is small (Future work will contain such a privacy measure).Choosing σ and m: for fixed S, the ratio r = σ 2 /m governs distortion and privacy loss. It follows that the higher σ and the lower m, the higher the privacy loss. The privacy accountant tells us that for fixed r = σ 2 /m, i.e. for the same level of distortion, privacy loss is smaller for σ and m both being small. An upper bound on the distortion rate r and a lower bound on the number of sub-sampled clientsm would thus lead to a choice of σ. A lower bound on m is, however, hard to estimate. That is, because data in federated settings is non-IID and contributions from clients might be very distinct. We therefore define the between clients variance V c as a measure of similarity between clients' updates. Definition. Let w i,j define the (i, j)-th parameter in an update of the form w ∈ R q×p, at some communication round t. For the sake of clarity, we will drop specific indexing of communication rounds for now. The variance of parameter (i, j) throughout all K clients is defined as, DISPLAYFORM3 where DISPLAYFORM4 We then define V c as the sum over all parameter variances in the update matrix as, DISPLAYFORM5 Further, the Update scale U s is defined as, DISPLAYFORM6 Algorithm 1 Client-side differentially private federated optimization. K is the number of participating clients; B is the local mini-batch size, E the number of local epochs, η is the learning rate, {σ} T t=0is the set of variances for the GM. {m t} T t=0 determines the number of participating clients at each communication round. defines the dp we aim for. Q is the threshold for δ, the probability that -dp is broken. T is the number of communication rounds after which δ surpasses Q. B is a set holding client's data sliced into batches of size B 1: procedure SERVER EXECUTION 2:Initialize: w 0, Accountant(, K) initialize weights and the priv. accountant 3:for each round t = 1, 2,... do δ ← Accountant(m t, σ t) Accountant returns priv. loss for current round for each client k ∈ Z t in parallel do 8: DISPLAYFORM0 DISPLAYFORM1 w ← w t for each local Epoch i = 1, 2,... E do In order to test our proposed algorithm we simulate a federated setting. For the sake of comparability, we choose a similar experimental setup as BID9 did. We divide the sorted MNIST set into shards. Consequently, each client gets two shards. This way most clients will have samples from two digits only. A single client could thus never train a model on their data such that it reaches high classification accuracy for all ten digits. We are investigating differential privacy in the federated setting for scenarios of K ∈ {100, 1000, 10000}. In each setting the clients get exactly 600 data points. For K ∈ {1000, 10000}, data points are repeated. For all three scenarios K ∈ {100, 1000, 10000} we performed a cross-validation grid search on the following parameters:• Number of batches per client B• Epochs to run on each client E• Number of clients participating in each round m• The GM parameter σ In accordance to BID0 we fixed to the value of 8. During training we keep track of privacy loss using the privacy accountant. Training is stopped once δ reaches e − 3, e − 5, e − 6 for 100, 1000 and 10000 clients, respectively. In addition, we also analyze the between clients variance over the course of training. In the cross validation grid search we look for those models that reach the highest accuracy while staying below the respective bound on δ. In addition, when multiple models reach the same accuracy, the one with fewer needed communication rounds is preferred. TAB3 holds the best models found for K ∈ {100, 1000, 10000}. We list the accuracy (ACC), the number of communication rounds (CR) needed and the arising communication costs (CC). Communication costs are defined as the number of times a model gets send by a client over the course of training, i.e. T t=0 m t. In addition, as a benchmark, TAB3 also holds the ACC, CR and CC of the best performing non-differentially private model for K = 100. In Fig. 2, the accuracy of all four best performing models is depicted over the course of training. In FIG3, the accuracy of non-differentially private federated optimization for K = 100 is depicted again together with the between clients variance and the update scale over the course of training. 100 clients, non differentially private 10000 clients, differentially private 1000 clients, differentially private 100 clients, differentially private Figure 2: Accuracy of digit classification from non-IID MNIST-data held by clients over the course of decentralized training. For differentially private federated optimization, dots at the end of accuracy curves indicate that the δ-threshold was reached and training therefore stopped. As intuitively expected, the number of participating clients has a major impact on the achieved model performance. For 100 and 1000 clients, model accuracy does not converge and stays significantly below the non-differentially private performance. However, 78% and 92% accuracy for K ∈ {100, 1000} are still substantially better than anything clients would be able to achieve when only training on their own data. In domains where K lays in this order of magnitude and differential privacy is of utmost importance, such models would still substantially benefit any client participating. An example for such a domain are hospitals. Several hundred could jointly learn a model, while information about a specific hospital stays hidden. In addition, the jointly learned model could be used as an initialization for further client-side training. For K = 10000, the differentially private model almost reaches accuracies of the non-differential private one. This suggests that for scenarios where many parties are involved, differential privacy comes at almost no cost in model performance. These scenarios include mobile phones and other consumer devices. In the cross-validation grid search we also found that raising m t over the course of training improves model performance. When looking at a single early communication round, lowering both m t and σ t in a fashion such that σ 2 t /m t stays constant, has almost no impact on the accuracy gain during that round. however, privacy loss is reduced when both parameters are lowered. This means more communication rounds can be performed later on in training, before the privacy budget is drained. In subsequent communication rounds, a large m t is unavoidable to gain accuracy, and a higher privacy cost has to be embraced in order to improve the model. This observation can be linked to recent advances of information theory in learning algorithms. As observable in FIG3, BID17 suggest, we can distinguish two different phases of training: label fitting and data fitting phase. During label fitting phase, updates by clients are similar and thus V c is low, as FIG3 shows. U c, however, is high during this initial phase, as big updates to the randomly initialized weights are performed. During data fitting phase V c rises. The individual updates w k look less alike, as each client optimizes on their data set. U c however drastically shrinks, as a local optima of the global model is approached, accuracy converges and the contributions cancel each other out to a certain extend. FIG3 shows these dependencies of V c and U c.We can conclude: i) At early communication rounds, small subsets of clients might still contribute an average update w t representative of the true data distribution ii) At later stages a balanced (and therefore bigger) fraction of clients is needed to reach a certain representativity for an update. iii) High U c makes early updates less vulnerable to noise. We were able to show through first empirical studies that differential privacy on a client level is feasible and high model accuracies can be reached when sufficiently many parties are involved. Furthermore, we showed that careful investigation of the data and update distribution can lead to optimized privacy budgeting. For future work, we plan to derive optimal bounds in terms of signal to noise ratio in dependence of communication round, data representativity and between-client variance as well as further investigate the connection to information theory. Additionally, we plan to further investigate the dataset dependency of the bounds. For assessing further applicability in bandwith-limited settings, we plan to investigate the applicability of proposed approach in context of compressed gradients such as proposed by BID8.

Ensuring that models learned in federated fashion do not reveal a client's participation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.