Question: Represent the article for finding a single-sentence TLDR

Question: Given a video and a sentence, the goal of weakly-supervised video moment retrieval is to locate the video segment which is described by the sentence without having access to temporal annotations during training. Instead, a model must learn how to identify the correct segment (i.e. moment) when only being provided with video-sentence pairs. Thus, an inherent challenge is automatically inferring the latent correspondence between visual and language representations. To facilitate this alignment, we propose our Weakly-supervised Moment Alignment Network (wMAN) which exploits a multi-level co-attention mechanism to learn richer multimodal representations. The aforementioned mechanism is comprised of a Frame-By-Word interaction module as well as a novel Word-Conditioned Visual Graph (WCVG). Our approach also incorporates a novel application of positional encodings, commonly used in Transformers, to learn visual-semantic representations that contain contextual information of their relative positions in the temporal sequence through iterative message-passing. Comprehensive experiments on the DiDeMo and Charades-STA datasets demonstrate the effectiveness of our learned representations: our combined wMAN model not only outperforms the state-of-the-art weakly-supervised method by a significant margin but also does better than strongly-supervised state-of-the-art methods on some metrics. Video understanding has been a mainstay of artificial intelligence research. Recent work has sought to better reason about videos by learning more effective spatio-temporal representations . The video moment retrieval task, also known as text-to-clip retrieval, combines language and video understanding to find activities described by a natural language sentence. The main objective of the task is to identify the video segment within a longer video that is most relevant to a sentence. This requires a model to learn the mapping of correspondences (alignment) between the visual and natural language modalities. In the strongly-supervised setting, existing methods generally learn joint visual-semantic representations by projecting video and language representations into a common embedding space and leverage provided temporal annotations to learn regressive functions for localization. However, such temporal annotations are often ambiguous and expensive to collect. seeks to circumvent these problems by proposing to address this task in the weakly-supervised setting where only full video-sentence pairs are provided as weak labels. However, the lack of temporal annotations renders the aforementioned approaches infeasible. In their approach (Figure 1a), proposes a Text-Guided Attention (TGA) mechanism to attend on segment-level features w.r.t. the sentence-level representations. However, such an approach treats the segment-level visual representations as independent inputs and ignores the contextual information derived from other segments in the video. More importantly, it does not exploit the fine-grained semantics of each word in the sentence. Consequently, existing methods are not able to reason about the latent alignment between the visual and language representations comprehensively. Figure 1: Given a video and a sentence, our aim is to retrieve the most relevant segment (the red bounding box in this example). Existing methods consider video frames as independent inputs and ignore the contextual information derived from other frames in the video. They compute a similarity score between the segment and the entire sentence to determine their relevance to each other. In contrast, our proposed approach aggregates contextual information from all the frames using graph propagation and leverages fine-grained frame-by-word interactions for more accurate retrieval. (Only some interactions are shown to prevent overcrowding the figure.) In this paper, we take another step towards addressing the limitations of current weakly-supervised video moment retrieval methods by exploiting the fine-grained temporal and visual relevance of each video frame to each word (Figure 1b). Our approach is built on two core insights: 1) The temporal occurrence of frames or segments in a video provides vital visual information required to reason about the presence of an event; 2) The semantics of the query are integral to reasoning about the relationships between entities in the video. With this in mind, we propose our Weakly-Supervised Moment Alignment Network (wMAN). An illustrative overview of our model is shown in Figure 2. The key component of wMAN is a multi-level co-attention mechanism that is encapsulated by a Frame-by-Word (FBW) interaction module as well as a Word-Conditioned Visual Graph (WCVG). To begin, we exploit the similarity scores of all possible pairs of visual frame and word features to create frame-specific sentence representations and word-specific video representations. The intuition is that frames relevant to a word should have a higher measure of similarity as compared to the rest. The word representations are updated by their word-specific video representations to create visual-semantic representations. Then a graph (WCVG) is built upon the frame and visualsemantic representations as nodes and introduces another level of attention between them. During the message-passing process, the frame nodes are iteratively updated with relational information from the visual-semantic nodes to create the final temporally-aware multimodal representations. The contribution of each visual-semantic node to a frame node is dynamically weighted based on their similarity. To learn such representations, wMAN also incorporates positional encodings into the visual representations to integrate contextual information about their relative positions. Such contextual information encourages the learning of temporally-aware multimodal representations. To learn these representations, we use a Multiple Instance Learning (MIL) framework that is similar in nature to the Stacked Cross Attention Network (SCAN) model. The SCAN model leverages image region-by-word interactions to learn better representations for image-text matching. In addition, the WCVG module draws inspiration from the Language-Conditioned Graph Network (LCGN) by which seeks to create context-aware object features in an image. However, the LCGN model works with sentence-level representations, which does not account for the semantics of each word to each visual node comprehensively. wMAN also distinguishes itself from the above-mentioned models by extracting temporally-aware multimodal representations from videos and their corresponding descriptions, whereas SCAN and LCGN only work on images. The contributions of our paper are summarized below: • We propose a simple yet intuitive MIL approach for weakly-supervised video moment retrieval from language queries by exploiting fine-grained frame-by-word alignment. • Our novel Word-Conditioned Visual Graph learns richer visual-semantic context through a multi-level co-attention mechanism. • We introduce a novel application of positional embeddings in video representations to learn temporally-aware multimodal representations. To demonstrate the effectiveness of our learned temporally-aware multimodal representations, we perform extensive experiments over two datasets, Didemo and Charades-STA , where we outperform the state-of-the-art weakly supervised model by a significant margin and strongly-supervised state-of-the-art models on some metrics. Most of the recent works in video moment retrieval based on natural language queries (; ; ; ; ;) are in the strongly-supervised setting, where the provided temporal annotations can be used to improve the alignment between the visual and language modalities. Among them, the Moment Alignment Network (MAN) introduced by utilizes a structured graph network to model temporal relationships between candidate moments, but one of the distinguishing factors with our wMAN is that our iterative message-passing process is conditioned on the multimodal interactions between frame and word representations. The TGN model bears some resemblance to ours by leveraging frame-by-word interactions to improve performance. However, it only uses a single level of attention which is not able to infer the correspondence between the visual and language modalities comprehensively. In addition, we reiterate that all these methods train their models using strong supervision, whereas we address the weakly supervised setting of this task. There are also a number of closely-related tasks to video moment retrieval such as temporal activity detection in videos. A general pipeline of proposal and classification is adopted by various temporal activity detection models (; ;) with the temporal proposals learnt by temporal coordinate regression. However, these approaches assume you are provided with a predefined list of activities, rather than an open-ended list provided via natural language queries at test time. Methods for visual phrase grounding also tend to be provided with natural language queries as input;;;; ), but the task is performed over image regions to locate a related bounding box rather than video segments to locate the correct moment. In the video moment retrieval task, given a ground truth video-sentence pair, the goal is to retrieve the most relevant video moment related to the description. The weakly-supervised version of this task we address can be formulated under the multiple instance learning (MIL) paradigm. When training using MIL, one receives a bag of items, where the bag is labeled as a positive if at least one item in the bag is a positive, and is labeled as a negative otherwise. In weakly-supervised moment retrieval, we are provided with a video-sentence pair (i.e., a bag) and the video segments are the items that we must learn to correctly label as relevant to the sentence (i.e., positive) or not. , we assume sentences are only associated with their ground truth video, and any other videos are negative examples. To build a good video-sentence representation, we introduce our Weakly-Supervised Moment Alignment Network (wMAN), which learns context-aware visualsemantic representations from fine-grained frame-by-word interactions. As seen in Figure 2, our network has two major components - representation learning constructed from the Frame-ByWord attention and Positional Embeddings , described in Section 3.1, and a Word-Conditioned Visual Graph where we update video segment representations based on context from the rest of the video, described in Section 3.2. These learned video segment representations are used to determine their relevance to their corresponding attended sentence representations using a LogSumExp (LSE) pooling similarity metric, described in Section 3.3. In this section we discuss our initial video and sentence representations which are updated with contextual information in Section 3.2. Each word in an input sentence is encoded using GloVe embeddings and then fed into a Gated Recurrent Unit (GRU) . The output of this GRU is denoted as W = {w 1, w 2, ···, w Q} where Q is the number of words in the sentence. Each frame in the input video is encoded using a pretrained Convolutional Neural Network (CNN). In the case of a 3D CNN this actually corresponds to a small chunk of sequential frames, but we shall refer to this as a frame representation throughout this paper for simplicity. To capture long-range dependencies, we feed the frame features into a Long Short-Term Memory (LSTM) . The latent hidden state output from the LSTM are concatenated with positional encodings (described below) to form the initial video representations, denoted as V = {v 1, v 2, ..., v N} where N is the number of frame features for video V. To provide some notion of the relative position of each frame we include the PE features which have been used in language tasks like learning language representations using BERT . These PE features can be thought of similar to the temporal endpoint features (TEF) used in prior work for strongly supervised moment retrieval task (e.g.,), but the PE features provide information about the temporal position of each frame rather than the rough position at the segment level. For the desired PE features of dimension d, let pos indicates the temporal position of each frame, i is the index of the feature being encoded, and M is a scalar constant, then the PE features are defined as: Through experiments, we found the hyper-parameter M = 10,000 works well for all videos. These PE features are concatenated with the LSTM encoded frame features at corresponding frame position before going to the cross-modal interaction layers. Rather than relating a sentence-level representation with each frame as done in prior work , we aggregate similarity scores between all frame and word combinations from the input video and sentence. These Frame-By-Word (FBW) similarity scores are used to compute attention weights to identify which frame and word combinations are important for retrieving the correct video segment. More formally, for N video frames and Q words in the input, we compute: Note that v now represents the concatenation of the video frame features and the PE features. Frame-Specific Sentence Representations. We obtain the normalized relevance of each word w.r.t. to each frame from the FBW similarity matrix, and use it to compute attention for each word: Using the above-mentioned attention weights, a weighted combination of all the words are created, with correlated words to the frame gaining high attention. Intuitively, a word-frame pair should have a high similarity score if the frame contains a reference to the word. Then the frame-specific sentence representation emphasizes words relevant to the frame and is defined as: Note that these frame-specific sentence representations don't participate in the iterative messagepassing process (Section 3.2). Instead, they are used to infer the final similarity score between a video segment and the query (Section 3.3). Word-Specific Video Representations. To determine the normalized relevance of each frame w.r.t. to each word, we compute the attention weights of each frame: Similarly, we attend to the visual frame features with respect to each word by creating a weighted combination of visual frame features determined by the relevance of each frame to the word. The formulation of each word-specific video-representation is defined as: These word-specific video representations are used in our Word-Conditioned Visual Graph, which we will discuss in the next section. Given the sets of visual representations, word representations and their corresponding word-specific video representations, the WCVG aims to learn temporally-aware multimodal representations by integrating visual-semantic and contextual information into the visual features. To begin, the word representations are updated with their corresponding video representations to create a new visualsemantic representation w vis j by concatenating each word w j and the word-specific video representation f j. Intuitively, the visual-semantic representations not only contain the semantic context of each word but also a summary of the video with respect to each word. A fully connected graph is then constructed with the visual features v i and the embedded attention of visual-semantic representations w vis j as nodes. Iterative Word-Conditioned Message-Passing The iterative message-passing process introduces a second round of FBW interaction similar to that in Section 3.1.1 to infer the latent temporal correspondence between each frame v i and visual-semantic representation w. To realize this, we first learn a projection W 1 followed by a ReLU of w vis j to obtain a new word representation to compute a new similarity matrix s ij on every message-passing iteration, namely, we obtain a replacement for w j in Eq. via w j = ReLU (W 1 (w vis j)). Updates of Visual Representations During the update process, each visual-semantic node sends its message (represented by its representation) to each visual node weighted by their edge weights. The representations of the visual nodes at the t-th iteration are updated by summing up the incoming messages as follows: where a ij is obtained by applying Eq. to the newly computed FBW similarity matrix s ij, and W 2 is a learned projection to make v t i the same dimensions as the frame-specific sentence representation l i (refer to Eq. ) which are finally used to compute a sentence-segment similarity score. The final updated visual representations are used to compute the relevance of each frame to its attended sentence-representations. A segment is defined as any arbitrary continuous sequence of visual features. We denote a segment as S = {v where K is the number of frame features contained within the segment S. We adopt the LogSumExp (LSE) pooling similarity metric used in SCAN, to determine the relevance each proposal segment has to the query: λ is a hyperparameter that weighs the relevance of the most salient parts of the video segment to the corresponding frame-specific sentence representations. Finally, following , given a triplet (X +, Y +, Y −), where (X +, Y +) is a positive pair and (X +, Y −) a negative pair, we use a margin-based ranking loss L T to train our model which ensures the positive pair's similarity score is better than the negative pair's by at least a margin. Our model's loss is then defined as: Sim LSE is used as the similarity metric between all pairs. At test time, Sim LSE is also used to rank the candidate temporal segments generated by sliding windows, and the top scoring segments will the localized segments corresponding to the input query sentence. We evaluate the capability of wMAN to accurately localize video moments based on natural language queries without temporal annotations on two datasets -DiDeMo and Charades-STA. On the DiDeMo dataset, we adopt the mean Intersection-Over-Union (IOU) and Recall@N at IOU threshold = θ. Recall@N represents the percentage of the test sliding window samples which have a overlap of at least θ with the ground-truth segments. mIOU is the average IOU with the groundtruth segments for the highest ranking segment to each query input. On the Charades-STA dataset, only the Recall@N metric is used for evaluation. Charades-STA The Charades-STA dataset is built upon the original Charades [] dataset which contains video-level paragraph descriptions and temporal annotations for activities. Charades-STA is created by breaking down the paragraphs to generate sentence-level annotations and aligning the sentences with corresponding video segments. In total, it contains 12,408 and 3,720 query-moment pairs in the training and test sets respectively. For fair comparison with the weakly model TGA , we use the same non-overlapping sliding windows of sizes 128 and 256 frames to generate candidate temporal segments. These 21 segments will used to compute the similarities with the input query and the top scored segment will be returned as the localization . For fair comparison, we utilize the same input features as the state-of-the-art method . Specifically, the word representations are initialized with GloVe embeddings and fine-tuned during the training process. For the experiments on DiDeMo, we use the provided mean-pooled visual frame and optical flow features. The visual frame features are extracted from the fc7 layer of VGG-16 [] pretrained on ImageNet []. The input visual features for our experiments on Charades-STA are C3D [] features. We adopt an initial learning rate of 1e − 5 and a margin= 0.5 used in our model's triplet loss (Eq. 9). In addition, we use three iterations for the message-passing process. Our model is trained end-to-end using the ADAM optimizer. The in Table 1 show that our full model outperforms the TGA model by a significant margin on all metrics. In particular, the Recall@1 accuracy when IOU = 0.7 obtained by our model is almost doubled that of TGA. It is notable that we observe a consistent trend of the Recall@1 accuracies improving the most across all IOU values. This not only demonstrates the importance of richer joint visual-semantic representations for accurate localization but also the superior capability of our model to learn them. Our model also performs comparably to the strongly-supervised MAN model on several metrics. To better understand the contributions of each component of our model, we present a comprehensive set of ablation experiments in Table 2. Note that our combined wMAN model is comprised of the FBW and WCVG components as well as the incorporation of PEs. The obtained by our FBW variant demonstrate that capturing fine-grained frame-by-word interactions is essential to inferring the latent temporal alignment between these two modalities. More importantly, the in the second row (FBW-WCVG) show that the second stage of multimodal attention, introduced by the WCVG module, encourages the augmented learning of intermodal relationships. Finally, we also observe that incorporating positional encodings into the visual representations (FBW-WCVG + PE) are especially helpful in improving Recall@1 accuracies for all IOU values. We provide for a model variant that include TEFs which encode the location of each video segment. In Table 2, our experiments show that TEFs actually hurt performance slightly. Our model variant with PEs (FBW- To gain insights into the fine-grained interactions between frames and words, we provide visualizations in Figure 3 . Our model is able to determine the most salient frames with respect to each word relatively well. In both examples, we observe that the top three salient frames with respect to each word are generally distributed over the same subset of frames. This seems to be indicative of the fact that our model leverages contextual information from all video frames as well as words in determining the salience of each frame to a specific word. Table 3 reports the on the DiDeMo dataset. In addition to reporting the state-of-the-art weakly supervised , we also include the obtained by strongly-supervised methods. It can be observed that our model outperforms the TGA model by a significant margin, even tripling the Recall@1 accuracy achieved by them. This demonstrates the effect of learning richer joint visualsemantic representations on the accurate localization of video moments. In fact, our full model outperforms the strongly-supervised TGN and MCN models on the Recall@1 metric by approximately 10%. We observe a consistent trend in the ablation studies (Table 4) as with those of Charades-STA. In particular, through comparing the ablation models FBW and FBW-WCVG, we demonstrate the effectiveness of our multi-level co-attention mechanism in WCVG where it improves the Recall@1 accuracy by a significant margin. Similar to our observations in Table 2, PEs help to encourage accurate latent alignment between the visual and language modalities, while TEFs fail in this aspect. In this work, we propose our weakly-supervised Moment Alignment Network with WordConditioned Visual Graph which exploits a multi-level co-attention mechanism to infer the latent alignment between visual and language representations at fine-grained word and frame level. Learning context-aware visual-semantic representations helps our model to reason about the temporal occurrence of an event as well as the relationships of entities described in the natural language query. (b) Figure 3: Visualization of the final relevance weights of each word in the query with respect to each frame. Here, we display the top three weights assigned to the frames for each phrase. The colors of the three numbers indicate the correspondence to the words in the query sentence. We also show the ground truth (GT) temporal annotation as well as our predicted weakly localized temporal segments in seconds. The highly correlated frames to each query word generally fall into the GT temporal segment in both examples. In Table 5, we show the comparisons of the different methods with different number of model parameters. While wMAN has 18M parameters as compared to 3M parameters in TGA, the performance gains are not simply attributed to the number of model parameters. We increase the dimensions of visual and semantic representations as well as corresponding fully-connected layers in the TGA model which leads to a total of 19M parameters. Despite having more parameters than wMAN, it still does significantly worse on all metrics. We also provide obtained by a direct adaptation of the Language-Conditioned Graph Network (LCGN), which is designed to work on the image level for VQA as well. While LCGN leverages attention over the words in the natural language query, the computed attention is only conditioned on the entire sentence without contextual information derived from the objects' visual representations. In contrast, the co-attention mechanism in our combined wMAN model is conditioned on both semantic and contextual visual information derived from words and video frames respectively. LCGN is also a lot more complicated and requires significantly more computing resources than wMAN. Despite possessing much more parameters than wMAN, it is still not able to achieve comparable to ours. In this section, we include ablation on the number of message-passing rounds required to learn effective visual-semantic representations. In our experiments, we have found that three rounds work best on both Charades-Sta and DiDeMo. C ABLATION ON TEST SETS The in Tables 8 and 9 emphasize the importance of both PEs and WCVG in our approach. We observe the best performances when PEs are integrated into the visual-semantic representations learned by the WCVG. The consistency of these performance gains across both datasets seems indicative of the increased capability of our model to learn temporally-aware visual-semantic representations.

Represent the short summary for finding the corresponding scientific article (long)

Weakly-Supervised Text-Based Video Moment Retrieval

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In machine learning tasks, overtting frequently crops up when the number of samples of target domain is insufﬁcient, for the generalization ability of the classiﬁer is poor in this circumstance. To solve this problem, transfer learning utilizes the knowledge of similar domains to improve the robustness of the learner. The main idea of existing transfer learning algorithms is to reduce the dierence between domains by sample selection or domain adaptation. However, no matter what transfer learning algorithm we use, the difference always exists and the hybrid training of source and target data leads to reducing ﬁtting capability of the learner on target domain. Moreover, when the relatedness between domains is too low, negative transfer is more likely to occur. To tackle the problem, we proposed a two-phase transfer learning architecture based on ensemble learning, which uses the existing transfer learning algorithms to train the weak learners in the ﬁrst stage, and uses the predictions of target data to train the ﬁnal learner in the second stage. Under this architecture, the ﬁtting capability and generalization capability can be guaranteed at the same time. We evaluated the proposed method on public datasets, which demonstrates the effectiveness and robustness of our proposed method. Transfer learning has attracted more and more attention since it was first proposed in 1995 BID11 and is becoming an important field of machine learning. The main purpose of transfer learning is to solve the problem that the same distributed data is hard to get in practical applications by using different distributed data of similar domains. Several different kinds of transfer stratagies are proposed in recent years, transfer learning can be devided into 4 categories BID17, including instance-based transfer learning, feature-based transfer learning, parameter-based transfer learning and relation-based transfer learning. In this paper, we focus on how to enhance the performance of instance-based transfer learning and feature-based transfer learning when limited labeled data from target domain can be obtained. In transfer learning tasks, when diff-distribution data is obtained to improve the generalization ability of learners, the fitting ability on target data set will be affected more or less, especially when the domains are not relative enough, negative transfer might occur BID11, it's hard to trade off between generalization and fitting. Most of the existing methods to prevent negative transfer learning are based on similarity measure(e.g., maximum mean distance(MMD), KL divergence), which is used for choosing useful knowledge on source domains. However, similarity and transferability are not equivalent concepts. To solve those problems, we proposed a novel transfer learning architecture to improve the fitting capability of final learner on target domain and the generalization capability is provided by weak learners. As shown in FIG0, to decrease the learning error on target training set when limited labeled data on target domain can be obtained, ensemble learning is introduced and the performances of transfer learning algorithms are significantly improved as a . In the first stage, traditional transfer learning algorithms are applied to diversify training data(e.g., Adaptive weight adjustment of boosting-based transfer learning or different parameter settings of domain adaptation). Then diversified training data is fed to several weak classifiers to improve the generalization ability on target data. To guarantee the fitting capability on target data, the predictions of target data is vectorized to be fed to the final estimator. This architecture brings the following advantages:• When the similarity between domains is low, the final estimator can still achieve good performance on target training set. Firstly, source data and target data are hybrid together to train the weak learners, then super learner is used to fit the predictions of target data.• Parameter setting is simplified and performance is better than individual estimators under normal conditions. To test the effectiveness of the method, we respectively modified TrAdaboost BID1 and BDA BID16 as the base algorithms for data diversification and desired is achieved.1.1 RELATED WORK TrAdaboost proposed by BID1 is a typical instance-based transfer learning algorithm, which transfer knowledge by reweighting samples of target domain and source domain according to the training error. In this method, source samples are used directly for hybrid training. As the earliest boosting based transfer learning method, there are many inherent defects in TrAdaboost (e.g., high requirements for similarity between domains, negative transfer can easily happen). Moreover, TrAdaboost is extended from Adaboost and use WMA(Weighted Majority Algorithm) BID6 to update the weights, and the source instance that is not correctly classified on a consistent basis would converge to zero by ⌈N/2⌉ and would not be used in the final classifier's output since that classifier only uses boosting iterations⌈N/2⌉ → N. Two weakness caused by disregarding first half of ensembles analysised in BID0 are in the list below:• As the source weights convergence rapidly, after ⌈N/2⌉ iterations, the source weights will be too low to make full use of source knowledge.• Later classifiers merely focus on the harder instances. To deal with rapid convergence of TrAdaboost, BID2 proposed TransferBoost, which apply a 2-phase training process at each iteration to test whether negative transfer has occurred and adjust the weights according to the . BID0 introduces an adaptive factor in weights update to slow down the convergence. BID19 proposed multisource TrAdaboost, aimed at utilize instances from multiple source domains to improve the transfer performance. In this paper, we still use the WMA to achieve data diversification in experiment of instance-based transfer learning, but stacking rather than boosting is used in final predictions. Feature based transfer learning mainly realizes transfer learning by reducing the distribution difference(e.g., MMD) between source domain and target domain by feature mapping, which is the most studied method for transfer knowledge in recent years. BID12 proposed transfer components analysis(TCA) as early as 2011, TCA achieve knowledge transfer by mapping the feature of source domain and target domain to a new feature space where the MMD between domains can be minimized. However, not using labels brings a defect that only marginal distribution can be matched. To address the problem, BID8 proposed joint distribution adaptation(JDA) which fit the marginal distribution and conditional distribution at the same time, for unlabeled target data, it utilizes pseudo-labels provided by classifier trained on source data. After that, BID16 extended JDA for imbalanced data. In neural networks, it's easy to transfer knowledge by pre-train and fine-tune because feature extracted by lower layers are mostly common for different tasks BID20, to transfer knowledge in higher layers which extract task-specific features, BID15, BID9 and BID10 add MMD to the optimization target in higher layers. The learning performance in the target domain using the source data could be poorer than that without using the source data, This phenomenon is called negative transfer BID11. To avoid negative transfer, BID7 point out that one of the most important research issues in transfer learning is to determine whether a given source domain is effective in transferring knowledge to a target domain, and then to determine how much of the knowledge should be transferred from a source domain to a target domain. Researchers have proposed to evaluate the relatedness between the source and target domains. When limited labeled target data can be obtained, two of the methods are listed below:• Introduce the predened parameters to qualify the relevance between the source and target domains. However, it is very labor consuming and time costing to manually select their proper values.• Examine the relatedness between domains directly to guide transfer learning. The notion of positive transferability was first introduced in BID13 for the assessment of synergy between the source and target domains in their prediction models, and a criterion to measure the positive transferability between sample pairs of different domains in terms of their prediction distributions is proposed in that research. BID14 proposed a kernel method to evaluate the task relatedness and the instance similarities to avoid negative transfer BID4 proposed a method to detection the occurance of negative transfer which can also delay the point of negative transfer in the process of transfer learning. BID3 remind that most previous work treats knowledge from every source domain as a valuable contribution to the task on the target domain could increase the risk of negative transfer. A two-phase multiple source transfer framework is proposed, which can effectively downgrade the contributions of irrelevant source domains and properly evaluate the importance of source domains even when the class distributions are imbalanced. Stacking is one of the ensemble learning methods that fuses multiple weak classifiers to get a better performance than any single one BID5. When using stacking, diversification of weak learners has an important impact on the performance of ensemble BID18. Here are some common ways to achieve diversification:• Diversifying input data: using different subset of samples or features.• Diversifying outputs: classifiers are only for certain categories.• Diversifying models: using different classification models. In this paper, we can also regard the proposed architecture as a stacking model which uses transfer learning algorithms to achieve input diversification. In this section, we introduce how the instance-based transfer learning is applied to the proposed architecture. we use TrAdaboost as an example and make a simple modification to turn it to stacking. In TrAdaboost, we need a large number of labeled data on source domain and limited labeled data on the target domain. We use X = X S ∪ X T to represent the feature space. Source space(X S) and target space(X T) are defined as DISPLAYFORM0 respectively. Then the hybrid training data set is defined in equation 1. DISPLAYFORM1 Weight vector is initialized firstly by w 1 = (w DISPLAYFORM2 In t th iteration, the weights are updated by Equation 3. DISPLAYFORM3 Here, β t = ϵ t 1−ϵt and β = 1/(1+ √ 2 ln n/N). It is noteworthy that original TrAdaboost is for binary classification. In order to facilitate experimental analysis and comparison, we extend the traditional TrAdaboost to a multi-classification algorithm according to Multi-class Adaboost proposed in BID21, then β t and beta defined as: DISPLAYFORM4 K is the class number. Equation 5 defines the final output for each class. DISPLAYFORM5 Moreover, for single-label problem, we use softmax to transfer P k (x) to probabilities. To address the rapid convergence problem, BID2 proposed TransferBoost, which utilizes all the weak classifiers for ensemble, but in the experiment, early stop can improve the final performance, so which classifiers should be chosen is still a problem. BID0 proposed dynamic TrAdaboost, In this algorithm, an adaptive factor is introduced to limit the convergence rate of source sample weight. However, it's not always effective in practical use. Theoretical upper bound of training error in target domain is not changed in dynamic TrAdaboost, which is related to the training error on target domain, we have: DISPLAYFORM6 Algorithm 1 stacked generalization for instance-based transfer learning. Call Learner, providing it labeled target data set with the distribution w t. Then get back a hypothesis of S. DISPLAYFORM7 Calculate the error on S:6: DISPLAYFORM0 Get a subset of source task DISPLAYFORM1 Call learner, providing it S t ∪ T. Then get back a hypothesis of T. Calculate the error on T using equation. 2.10:updata β t using equation. 4. Update the new weight vector using equation. 3. 12: end for 13: Construct probability vectors by concatenating DISPLAYFORM0 Although dynamic TrAdaboost can improve the weights of source samples after iteration ⌈N/2⌉ and the generalization capability is improved, it's very likely that the error rate on source domain ϵ t increases, sometimes it even aggravates the occurrence of negative transfer when the domains are not similar enough. We use stacking to address the problems above in this section, in the data diversification stage, TrAdaboost is used to reweight samples for each weak classifier. Meanwhile, because we make use of all the weak classifiers, to avoid the high source weights of irrelative source samples negatively effects on the task in early iterations, a two-phase sampling mechanism is introduced in our algorithm. A formal description of stacking for instance-based transfer learning is given in Algorithm 1. The main difference between stacking for instance-based transfer learning and TrAdaboost are listed as follows:• A two-phase sampling process and an extra parameter λ is introduced. Firstly, target data is fed to weak learner and the weighted error rate of source samples are used to decide which samples can be used for hybrid learning by comparing with the threshold λ. As the source weights reduces with the number of iterations increasing, more and more source samples will be utilized.• Stacking rather than TrAdaboost is used to get the final output. We construct a feature matrix by the outputs of weak learners on target data, then use a super learner(e.g., LogitRegression in our experiment) to fit the labels. In this way, training error on target set can be minimized. When compared with TrAdaboost, stacking is insensitive to the performance of each weak classifier because the training error on target data can be minimized in stacking, which means it's more robust in most cases and brings some benefits:• When using stacking, all of the weak classifiers could be used.• When source domain is not related enough, stacking performs better. One of a popular methods for feature-based transfer learning to achieve knowledge transfer is domain adaptation, which minimizes the distribution difference between domains by mapping features to a new space, where we could measure the distribution difference by MMD. Generally speaking, we use P (X S), P (X T) and P (Y S |X S), P (Y T |X T) to represent the marginal distribution and conditional distribution of source domain and target domain respectively. In Pan et al. FORMULA2, transfer component analysis(TCA) was proposed to find a mapping which makes P (ϕ(X S)) ≈ P (ϕ(X T)), the MMD between domains in TCA is defined as: DISPLAYFORM0 Long et al. FORMULA2 proposed joint distibution adaptation(JDA) to minimize the differences of marginal distribution and conditinal distribution at the same time, the MMD of conditional distribution is defined as: DISPLAYFORM1 In BID16, balanced distribution adaptation was proposed, in which algorithm, an extra parameter is introduced to adjust the importance of the distributions, the optimization target is defined by Equation FORMULA13: DISPLAYFORM2 To solve the nonlinear problem, we could use a kernel matrix defined by: DISPLAYFORM3, then the optimization proplem can be formalized as: DISPLAYFORM4 Where H is a centering matrix, M 0 and M c represent the MMD matrix of marginal distribution and conditional distribution respectively, A is the mapping matrix. In domain adaptation, performace is sensitive to the selection of parameters(e.g., kernel type, kernel param or regularization param). For instance, if we use rbf kernel, as defined in Equation 11, to construct the kernel matrix. selection of kernel param σ has an influence on the mapping. In this paper, we use BDA as a base algorithm in the proposed architecture to achieve data diversification by using different kernel types and parameter settings. DISPLAYFORM5 By taking adavantage of stacking, we could get a better transfer performance than any single algorithm. Here, we introduce how we choose the kernel parameter in our experiments. In ensemble learning, it's significant to use unrelated weak classifiers for a better performance (i.e., learners should have different kinds of classification capabilities). moreover, performances of learners shouldn't be too different or the poor learners will have an negative effect on ensemble. In another word, we choose the kernel parameter in a largest range where the performance is acceptable. We take the following steps to select parameters. Firstly, search a best value of kernel parameter σ for weak classifier, where the accuracy on validation set is Accuracy max. secondly, set a threshold parameter λ and find an interval (σ min, σ max) around σ where the accuracy on validation set satisfy Accuracy max − λ ≤ Accuracy when σ ∈ (σ min, σ max). Finally, select N parameters in (σ min, σ max) by uniformly-spaced sampling. When multiple type of kernels are utilized, we choose parameter sets for each seperately by repeating the above steps. In our method, the settings of λ and N should be taken into consideration, if λ is too large, the performance of each learner can't be guaranteed, if λ is too small, training data can't be diversified enough. Set N to a large number would help to get a better performance in most cases, while the complexity could be high. Algorithm 2 stacked generalization for feature-based transfer learning. DISPLAYFORM6 Contruct kernel matrix K t using κ t. Solve the eigendecomposition problem and use d smallest eigenvectors to build A t. Train t th leaner on {A DISPLAYFORM0 Call learner t providing DISPLAYFORM1 Construct probability vectors by concatenating DISPLAYFORM2, where K is the number of classes. 10: end for 11: Construct target feature matrix DISPLAYFORM3 Algorithm 2 presents the detail of our method. In our algorithm, kernel function κ t can be differentiated by kernel types or kernel params. In t th iteration, we choose κ t to mapping the feature space, then get the matrix A by BDA, A ⊤ K tar and A ⊤ K src are feed to weak learner. After that, we concatenate predictions of A ⊤ K tar as features of the super leaner. In this paper, we assume that there are limited labeled data on target set, so we use modified BDA, which uses real label rather than pseudo label, to adapt conditional distribution. To evaluate the effectiveness of our method, 6 public datasets and 11 domains as shown in TAB3 are used in our experiment of instance-based transfer learning and feature-based transfer learning. The detail is described as follow: Figure 3 : Accuracy of instance-based transfer on mnist vs. usps changed with the iterations and ratio of #target. Figure 3 shows the experiment of transfer between mnist and USPS under different iterations and ratios of #source and #target. We observe that stacking achieves much better performance than TrAdaboost. Firstly, acuuracy of stacking method is higher when ratio changing from 1% to 10%. Especially, the fewer the labeled target samples are, the more improvement stacking method could achieve. Secondly, few iterations are required for stacking method to achieve a relatively good performance, when the curve is close to convergence, there's still about 5% improvement compared with TrAdaboost. Moreover, in both transfer tasks USPS→ mnist and mnist → USPS, stacking method performs significantly better than TrAdaboost. The reason why stacking performs better is analyzed in section 3.1, we assume that the introduction of source leads to under fitting on target data when hybrid training, to confirm our hypothesis, we made a comparision of training error on source data and target data between TrAdaboost and stacking method, FIG2 shows the of four of the transfer tasks. TAB4 shows the of all the transfer tasks under 20 iterations. BDA is chosen as the base algorithm to achieve data diversification in our experiment, we mainly test the infulence of different kernel functions has on the perfomance and the effectiveness of the method proposed in section 3.2. The ratio of #source and #target is set to 5% and we use rbf kernel, poly kernel and sam kernel to conduct our experiment, for the sake of simplicity, kernel function is defined in TAB5, where γ is variable for different weak learners. DISPLAYFORM0 To observe how the selection of kernels affects the feature distribution, we visualize the feature representations learned by BDA under different kernel parameters and kernel types when adapting domains A and C. As shown in FIG5 (a), data distribution and similarity between domains change with the kernel parameters, when compared with FIG5 (b), which presents feature distribution of rbf kernel, it's obvious that using different kernel types can provide more diversity. In this paper, to construct the kernel set by sampling parameters in a range where the performance is not too worse than the best one, we followed the method given in section 3.2 and set threshold λ varies from 5% to 10% for different tasks. For each kernel type, we select 10 different parameters(i.e., 10 weak classifiers) for stacking. Tabel 4 shows the comparison between single algorithm and ensemble learning. For each kernel type, we give the best accuracy, average accuracy of weak learners and accuracy of ensemble learning, Randomforest and LogitRegression as the weak learner and super learner respectively. Accuracy of integrating all the kernel types is shown in the last column and the best performance of each task is in bold. We can learn from the table that ensemble learning outperfoms the best single learner in all the tasks, and in most cases, using both of rbf and another type of kernels are able to improve the performance. However, when should we use multiple kernel types in stacking needs to be further studied. In summary, the reason why the proposed method can improve the performance of feature-based transfer learning is listed as follows: Firstly, we use super learner to fit the target domain, so the bias of weak learners introduced by hybrid training with source data is reduced. Secondly, multiple kernels are utilized to achieve data diversification, so we could integrate the classification ability of weak learners trained on diff-distribution data. In this paper, we proposed a 2-phase transfer learning architecture, which uses the traditional transfer learning algorithm to achieve data diversification in the first stage and the target data is fitted in the second stage by stacking method, so the generalization ability and fitting ability on target data could be satisfied at the same time. The experiment of instance-based transfer learning and feature-based transfer learning on 11 domains proves the validity of our method. In summary, this framework has the following advantages:• No matter if source domain and target domain are similar, the training error on target data set can be minimized theoretically.• We reduce the risk of negative transfer in a simple and effective way without a similarity measure.• Introduction of ensemble learning gives a better performance than any single learner.• Most existing transfer learning algorithm can be integrated into this framework. Moreover, there're still some problems require our further study, some other data diversification method for transfer learning might be useful in our model, such as changing the parameter µ in BDA, integrating multiple kinds of transfer learning algorithms, or even applying this framework for multi-source transfer learning.

How to use stacked generalization to improve the performance of existing transfer learning algorithms when limited labeled data is available.

Represent the TLDR for finding the corresponding scientific article:

Considering neural network optimization process as a model selection problem, we introduce a biological plausible normalization method that extracts statistical regularity under MDL principle to tackle imbalanced and limited data issue.


Question: Represent the article for finding a single-sentence TLDR

Question: Deep learning has achieved astonishing on many tasks with large amounts of data and generalization within the proximity of training data. For many important real-world applications, these requirements are unfeasible and additional prior knowledge on the task domain is required to overcome the ing problems. In particular, learning physics models for model-based control requires robust extrapolation from fewer samples – often collected online in real-time – and model errors may lead to drastic damages of the system. Directly incorporating physical insight has enabled us to obtain a novel deep model learning approach that extrapolates well while requiring fewer samples. As a first example, we propose Deep Lagrangian Networks (DeLaN) as a deep network structure upon which Lagrangian Mechanics have been imposed. DeLaN can learn the equations of motion of a mechanical system (i.e., system dynamics) with a deep network efficiently while ensuring physical plausibility. The ing DeLaN network performs very well at robot tracking control. The proposed method did not only outperform previous model learning approaches at learning speed but exhibits substantially improved and more robust extrapolation to novel trajectories and learns online in real-time. A Deep learning has achieved astonishing on many tasks with large amounts of data and generalization within the proximity of training data. For many important real-world applications, these requirements are unfeasible and additional prior knowledge on the task domain is required to overcome the ing problems. In particular, learning physics models for model-based control requires robust extrapolation from fewer samples -often collected online in real-time -and model errors may lead to drastic damages of the system. Directly incorporating physical insight has enabled us to obtain a novel deep model learning approach that extrapolates well while requiring fewer samples. As a first example, we propose Deep Lagrangian Networks (DeLaN) as a deep network structure upon which Lagrangian Mechanics have been imposed. DeLaN can learn the equations of motion of a mechanical system (i.e., system dynamics) with a deep network efficiently while ensuring physical plausibility. The ing DeLaN network performs very well at robot tracking control. The proposed method did not only outperform previous model learning approaches at learning speed but exhibits substantially improved and more robust extrapolation to novel trajectories and learns online in real-time. In the last five years, deep learning has propelled most areas of learning forward at an impressive pace BID22 BID31 BID41 -with the exception of physically embodied systems. This lag in comparison to other application areas is somewhat surprising as learning physical models is critical for applications that control embodied systems, reason about prior actions or plan future actions (e.g., service robotics, industrial automation). Instead, most engineers prefer classical off-the-shelf modeling as it ensures physical plausibility -at a high cost of precise measurements1 and engineering effort. These plausible representations are preferred as these models guarantee to extrapolate to new samples, while learned models only achieve good performance in the vicinity of the training data. To learn a model that obtains physically plausible representations, we propose to use the insights from physics as a model prior for deep learning. In particular, the combination of deep learning and physics seems natural as the compositional structure of deep networks enables the efficient computation of the derivatives at machine precision BID35 and, thus, can encode a differential equation describing physical processes. Therefore, we suggest to encode the physics prior in the form of a differential in the network topology. This adapted topology amplifies the information content of the training samples, regularizes the end-to-end training, and emphasizes robust models capable of extrapolating to new samples while simultaneously ensuring physical plausibility. Hereby, we concentrate on learning models of mechanical systems using the Euler-Lagrange-Equation, a second order ordinary differential equation (ODE) originating from Lagrangian Mechanics, as physics prior. We focus on learning models of mechanical systems as this problem is one of the fundamental challenges of robotics BID7 BID39. The contribution of this work is twofold. First, we derive a network topology called Deep Lagrangian Networks (DeLaN) encoding the Euler-Lagrange equation originating from Lagrangian Mechanics. This topology can be trained using standard end-to-end optimization techniques while maintaining physical plausibility. Therefore, the obtained model must comply with physics. Unlike previous approaches to learning physics BID1 BID25, which engineered fixed features from physical assumptions requiring knowledge of the specific physical embodiment, we are'only' enforcing physics upon a generic deep network. For DeLaN only the system state and the control signal are specific to the physical system but neither the proposed network structure nor the training procedure. Second, we extensively evaluate the proposed approach by using the model to control a simulated 2 degrees of freedom (dof) robot and the physical 7-dof robot Barrett WAM in real time. We demonstrate DeLaN's control performance where DeLaN learns the dynamics model online starting from random initialization. In comparison to analytic-and other learned models, DeLaN yields a better control performance while at the same time extrapolates to new desired trajectories. In the following we provide an overview about related work (Section 2) and briefly summarize Lagrangian Mechanics (Section 3). Subsequently, we derive our proposed approach DeLaN and the necessary characteristics for end-to-end training are shown (Section 4). Finally, the experiments in Section 5 evaluate the model learning performance for both simulated and physical robots. Here, DeLaN outperforms existing approaches. Models describing system dynamics, i.e. the coupling of control input τ τ τ and system state q, are essential for model-based control approaches BID17. Depending on the control approach, the control law relies either on the forward model f, mapping from control input to the change of system state, or on the inverse model f −1, mapping from system change to control input, i.e., DISPLAYFORM0 Examples for application of these models are inverse dynamics control BID7, which uses the inverse model to compensate system dynamics, while model-predictive control BID4 and optimal control BID46 use the forward model to plan the control input. These models can be either derived from physics or learned from data. The physics models must be derived for the individual system embodiment and requires precise knowledge of the physical properties BID0. When learning the model2, mostly standard machine learning techniques are applied to fit either the forward-or inverse-model to the training data. E.g., authors used Linear Regression BID39 BID14, Gaussian Mixture Regression BID3 BID20, Gaussian Process Regression BID21 BID34 BID32, Support Vector Regression BID5 BID10, feedforward- BID18 BID26 BID25 BID38 or recurrent neural networks BID37 to fit the model to the observed measurements. Only few approaches incorporate prior knowledge into the learning problem. BID38 use the graph representation of the kinematic structure as input. While the work of BID1, commonly referenced as the standard system identification technique for robot manipulators BID40, uses the Newton-Euler formalism to derive physics features using the kinematic structure and the joint measurements such that the learning of the dynamics model simplifies to linear regression. Similarly, BID25 hard-code these physics features within a neural network and learn the dynamics parameters using gradient descent rather than linear regression. Even though these physics features are derived from physics, the 2Further information can be found in the model learning survey by BID33 learned parameters for mass, center of gravity and inertia must not necessarily comply with physics as the learned parameters may violate the positive definiteness of the inertia matrix or the parallel axis theorem BID44. Furthermore, the linear regression is commonly underdetermined and only allows to infer linear combinations of the dynamics parameters and cannot be applied to close-loop kinematics BID40.DeLaN follows the line of structured learning problems but in contrast to previous approaches guarantees physical plausibility and provides a more general formulation. This general formulation enables DeLaN to learn the dynamics for any kinematic structure, including kinematic trees and closed-loop kinematics, and in addition does not require any knowledge about the kinematic structure. Therefore, DeLaN is identical for all mechanical systems, which is in strong contrast to the NewtonEuler approaches, where the features are specific to the kinematic structure. Only the system state and input is specific to the system but neither the network topology nor the optimization procedure. The combination of differential equations and Neural Networks has previously been investigated in literature. Early on BID23 BID24 proposed to learn the solution of partial differential equations (PDE) using neural networks and currently this topic is being rediscovered by BID35; BID42; BID28. Most research focuses on using machine learning to overcome the limitations of PDE solvers. E.g., BID42 proposed the Deep Galerkin method to solve a high-dimensional PDE from scattered data. Only the work of BID36 took the opposite standpoint of using the knowledge of the specific differential equation to structure the learning problem and achieve lower sample complexity. In this paper, we follow the same motivation as BID36 but take a different approach. Rather than explicitly solving the differential equation, DeLaN only uses the structure of the differential equation to guide the learning problem of inferring the equations of motion. Thereby the differential equation is only implicitly solved. In addition, the proposed approach uses different encoding of the partial derivatives, which achieves the efficient computation within a single feed-forward pass, enabling the application within control loops. Describing the equations of motion for mechanical systems has been extensively studied and various formalisms to derive these equations exist. The most prominent are Newtonian-, Hamiltonianand Lagrangian-Mechanics. Within this work Lagrangian Mechanics is used, more specifically the Euler-Lagrange formulation with non-conservative forces and generalized coordinates.3 Generalized coordinates are coordinates that uniquely define the system configuration. This formalism defines the Lagrangian L as a function of generalized coordinates q describing the complete dynamics of a given system. The Lagrangian is not unique and every L which yields the correct equations of motion is valid. The Lagrangian is generally chosen to be DISPLAYFORM0 where T is the kinetic energy and V is the potential energy. The kinetic energy T can be computed for all choices of generalized coordinates using T = 1 2 q T H(q) q, whereas H(q) is the symmetric and positive definite inertia matrix BID7. The positive definiteness ensures that all non-zero velocities lead to positive kinetic energy. Applying the calculus of variations yields the Euler-Lagrange equation with non-conservative forces described by DISPLAYFORM1 where τ τ τ are generalized forces. Substituting L and dV/dq = g(q) into Equation 3 yields the second order ordinary differential equation (ODE) described by DISPLAYFORM2 3More information can be found in the textbooks BID13 BID7 BID9 where c describes the forces generated by the Centripetal and Coriolis forces BID9. Using this ODE any multi-particle mechanical system with holonomic constraints can be described. For example various authors used this ODE to manually derived the equations of motion for coupled pendulums BID13, robotic manipulators with flexible joints BID2 BID43, parallel robots BID30 BID11 BID27 or legged robots BID15 BID12. DISPLAYFORM3 Starting from the Euler-Lagrange equation FORMULA4 ), traditional engineering approaches would estimate H(q) and g(q) from the approximated or measured masses, lengths and moments of inertia. On the contrary most traditional model learning approaches would ignore the structure and learn the inverse dynamics model directly from data. DeLaN bridges this gap by incorporating the structure introduced by the ODE into the learning problem and learns the parameters in an end-to-end fashion. More concretely, DeLaN approximates the inverse model by representing the unknown functions g(q) and H(q) as a feed-forward networks. Rather than representing H(q) directly, the lower-triangular matrix L(q) is represented as deep network. Therefore, g(q) and H(q) are described bŷ DISPLAYFORM4 where. refers to an approximation and θ and ψ are the respective network parameters. The parameters θ and ψ can be obtained by minimizing the violation of the physical law described by Lagrangian Mechanics. Therefore, the optimization problem is described by DISPLAYFORM5 withf DISPLAYFORM6 wheref −1 is the inverse model and can be any differentiable loss function. The computational graph off −1 is shown in FIG0.Using this formulation one can conclude further properties of the learned model. NeitherL nor g are functions of q or q and, hence, the obtained parameters should, within limits, generalize to arbitrary velocities and accelerations. In addition, the obtained model can be reformulated and used as a forward model. Solving Equation 6 for q yields the forward model described bŷ DISPLAYFORM7 whereLL T is guaranteed to be invertible due to the positive definite constraint FORMULA8 ). However, solving the optimization problem of Equation 5 directly is not possible due to the ill-posedness of the Lagrangian L not being unique. The Euler-Lagrange equation is invariant to linear transformation and, hence, the Lagrangian L = αL + β solves the Euler-Lagrange equation if α is non-zero and L is a valid Lagrangian. This problem can be mitigated by adding an additional penalty term to Equation 5 described by DISPLAYFORM8 where Ω is the L 2 -norm of the network weights. Solving the optimization problem of Equation 9 with a gradient based end-to-end learning approach is non-trivial due to the positive definite constraint FORMULA8 ) and the derivatives contained inf −1. In particular, d(LL T)/dt and ∂ q T LL T q /∂q i cannot be computed using automatic differentiation as t is not an input of the network and most implementations of automatic differentiation do not allow the backpropagation of the gradient through the computed derivatives. Therefore, the derivatives contained inf −1 must be computed analytically to exploit the full gradient information for training of the parameters. In the following we introduce a network structure that fulfills the positive-definite constraint for all parameters (Section 4.1), prove that the derivatives d(LL T)/dt and ∂ q T LL T q /∂q i can be computed analytically (Section 4.2) and show an efficient implementation for computing the derivatives using a single feed-forward pass (Section 4.3). Using these three properties the ing network architecture can be used within a real-time control loop and trained using standard end-to-end optimization techniques. The derivatives d LL T /dt and ∂ q T LL T q /∂q i are required for computing the control signal τ τ τ using the inverse model and, hence, must be available within the forward pass. In addition, the second order derivatives, used within the backpropagation of the gradients, must exist to train the network using end-to-end training. To enable the computation of the second order derivatives using automatic differentiation the forward computation must be performed analytically. Both derivatives, d LL T /dt and ∂ q T LL T q /∂q i, have closed form solutions and can be derived by first computing the respective derivative of L and second substituting the reshaped derivative of the vectorized form l. For the temporal derivative d LL T /dt this yields DISPLAYFORM0 whereas dL/dt can be substituted with the reshaped form of DISPLAYFORM1 where i refers to the i-th network layer consisting of an affine transformation and the non-linearity g, i.e., DISPLAYFORM2 can be simplified as the network weights W i and biases b i are time-invariant, i.e., dW i /dt = 0 and db i /dt = 0. Therefore, dl/dt is described by DISPLAYFORM3 Due to the compositional structure of the network and the differentiability of the non-linearity, the derivative with respect to the network input dl/dq can be computed by recursively applying the chain rule, i.e., DISPLAYFORM4 where g is the derivative of the non-linearity. Similarly to the previous derivation, the partial derivative of the quadratic term can be computed using the chain rule, which yields DISPLAYFORM5 whereas ∂L/∂q i can be constructed using the columns of previously derived ∂l/∂q. Therefore, all derivatives included withinf can be computed in closed form. The derivatives of Section 4.2 must be computed within a real-time control loop and only add minimal computational complexity in order to not break the real-time constraint. l and ∂l/∂q, required within Equation 10 and Equation 14, can be simultaneously computed using an extended standard layer. Extending the affine transformation and non-linearity of the standard layer with an additional sub-graph for computing ∂h i /∂h i−1 yields the Lagrangian layer described by DISPLAYFORM0 The computational graph of the Lagrangian layer is shown in FIG1. Chaining the Lagrangian layer yields the compositional structure of ∂l/∂q (Equation 13) and enables the efficient computation of ∂l/∂q. Additional reshaping operations compute dL/dt and ∂L/∂q i. To demonstrate the applicability and extrapolation of DeLaN, the proposed network topology is applied to model-based control for a simulated 2-dof robot FIG2 ) and the physical 7-dof robot Barrett WAM FIG2 ). The performance of DeLaN is evaluated using the tracking error on train and test trajectories and compared to a learned and analytic model. This evaluation scheme follows existing work BID34 BID38 as the tracking error is the relevant performance indicator while the mean squared error (MSE)4 obtained using sample based optimization exaggerates model performance BID16. In addition to most previous work, we strictly limit all model predictions to real-time and perform the learning online, i.e., the models are randomly initialized and must learn the model during the experiment. Within the experiment the robot executes multiple desired trajectories with specified joint positions, velocities and accelerations. The control signal, consisting of motor torques, is generated using a non-linear feedforward controller, i.e., a low gain PD-Controller augmented with a feed-forward torque τ τ τ f f to compensate system dynamics. The control law is described by DISPLAYFORM0 where K p, K d are the controller gains and q d, q d, q d the desired joint positions, velocities and accelerations. The control-loop is shown in FIG2. For all experiments the control frequency is set to 500Hz while the desired joint state and respectively τ τ τ f f is updated with a frequency of f d = 200Hz. All feed-forward torques are computed online and, hence, the computation time is strictly limited to T ≤ 1/200s. The tracking performance is defined as the sum of the MSE evaluated at the sampling points of the reference trajectory. For the desired trajectories two different data sets are used. The first data set contains all single stroke characters5 while the second data set uses cosine curves in joint space FIG2 ). The 20 characters are spatially and temporally re-scaled to comply with the robot kinematics. The joint references are computed using the inverse kinematics. Due to the different characters, the desired trajectories contain smooth and sharp turns and cover a wide variety of different shapes but are limited to a small task space region. In contrast, the cosine trajectories are smooth but cover a large task space region. The performance of DeLaN is compared to an analytic inverse dynamics model, a standard feedforward neural network (FF-NN) and a PD-Controller. For the analytic models the torque is computed using the Recursive Newton-Euler algorithm (RNE) BID29, which computes the feedforward torque using estimated physical properties of the system, i.e. the link dimensions, masses and moments of inertia. For implementations the open-source library PyBullet is used. Both deep networks use the same dimensionality, ReLu nonlinearities and must learn the system dynamics online starting from random initialization. The training samples containing joint states and applied torques (q, q, q, τ τ τ) 0,...T are directly read from the control loop as shown in FIG2.4An offline comparisons evaluating the MSE on datasets can be found in the Appendix A. 5The data set was created by BID45 and is available at ) The training runs in a separate process on the same machine and solves the optimization problem online. Once the training process computed a new model, the inverse modelf −1 of the control loop is updated. The 2-dof robot shown in FIG2 is simulated using PyBullet and executes the character and cosine trajectories. FIG3 shows the ground truth torques of the characters'a','d','e', the torque ground truth components and the learned decomposition using DeLaN FIG3. Even though DeLaN is trained on the super-imposed torques, DeLaN learns to disambiguate the inertial force H q, the Coriolis and Centrifugal force c(q, q) and the gravitational force g(q) as the respective curves overlap closely. Hence, DeLaN is capable of learning the underlying physical model using the proposed network topology trained with standard end-to-end optimization. FIG3 shows the offline MSE on the test set averaged over multiple seeds for the FF-NN and DeLaN w.r.t. to different training set sizes. The different training set sizes correspond to the combination of n random characters, i.e., a training set size of 1 corresponds to training the model on a single character and evaluating the performance on the remaining 19 characters. DeLaN clearly obtains a lower test MSE compared to the FF-NN. Especially the difference in performance increases when the training set is reduced. This increasing difference on the test MSE highlights the reduced sample complexity and the good extrapolation to unseen samples. This difference in performance is amplified on the real-time control-task where the models are learned online starting from random initialization. FIG4 and b shows the accumulated tracking error per testing character and the testing error averaged over all test characters while FIG4 shows the qualitative comparison of the control performance6. It is important to point out that all shown are averaged over multiple seeds and only incorporate characters not used for training and, hence, focus the evaluation on the extrapolation to new trajectories. The qualitative comparison shows that DeLaN is able to execute all 20 characters when trained on 8 random characters. The obtained tracking error is comparable to the analytic model, which in this case contains the simulation parameters and is optimal. In contrast, the FF-NN shows significant deviation from the desired trajectories when trained on 8 random characters. The quantitative comparison of the accumulated tracking error over seeds FIG4 shows that DeLaN obtains lower tracking error on all training set sizes compared to the FF-NN. This good performance using only few training characters shows that DeLaN has a lower sample complexity and better extrapolation to unseen trajectories compared to the FF-NN. comparable. When the velocities are increased the performance of FF-NN deteriorates because the new trajectories do not lie within the vicinity of the training distribution as the domain of the FF-NN is defined as (q, q, q). Therefore, FF-NN cannot extrapolate to the testing data. In contrast, the domain of the networksL andĝ composing DeLaN only consist of q, rather than (q, q, q). This reduced domain enables DeLaN, within limit, to extrapolate to the test trajectories. The increase in tracking error is caused by the structure off −1, where model errors to scale quadratic with velocities. However, the obtained tracking error on the testing trajectories is significantly lower compared to FF-NN. For physical experiments the desired trajectories are executed on the Barrett WAM, a robot with direct cable drives. The direct cable drives produce high torques generating fast and dexterous movements but yield complex dynamics, which cannot be modelled using rigid-body dynamics due to the variable stiffness and lengths of the cables7. Therefore, the Barrett WAM is ideal for testing the applicability of model learning and analytic models8 on complex dynamics. For the physical experiments we focus on the cosine trajectories as these trajectories produce dynamic movements while character trajectories are mainly dominated by the gravitational forces. In addition, only the dynamics of the four lower joints are learned because these joints dominate the dynamics and the upper joints cannot be sufficiently excited to retrieve the dynamics parameters. FIG5 and d show the tracking error on the cosine trajectories using the the simulated Barrett WAM while FIG5 and f show the tracking error of the physical Barrett WAM. It is important to note, that the simulation only simulates the rigid-body dynamics not including the direct cables drives and the simulation parameters are inconsistent with the parameters of the analytic model. Therefore, the analytic model is not optimal. On the training trajectories executed on the physical system the FF-NN performs better compared to DeLaN and the analytic model. DeLaN achieves slightly better tracking error than the analytic model, which uses the same rigid-body assumptions as DeLaN. That shows DeLaN can learn a dynamics model of the WAM but is limited by the model assumptions of Lagrangian Mechanics. These assumptions cannot represent the dynamics of the 7The cable drives and cables could be modelled simplistically using two joints connected by massless spring. 8The analytic model of the Barrett WAM is obtained using a publicly available URDF We introduced the concept of incorporating a physics prior within the deep learning framework to achieve lower sample complexity and better extrapolation. In particular, we proposed Deep Lagrangian Networks (DeLaN), a deep network on which Lagrangian Mechanics is imposed. This specific network topology enabled us to learn the system dynamics using end-to-end training while maintaining physical plausibility. We showed that DeLaN is able to learn the underlying physics from a super-imposed signal, as DeLaN can recover the contribution of the inertial-, gravitational and centripetal forces from sensor data. The quantitative evaluation within a real-time control loop assessing the tracking error showed that DeLaN can learn the system dynamics online, obtains lower sample complexity and better generalization compared to a feed-forward neural network. DeLaN can extrapolate to new trajectories as well as to increased velocities, where the performance of the feedforward network deteriorates due to the overfitting to the training data. When applied to a physical systems with complex dynamics the bounded representational power of the physics prior can be limiting. However, this limited representational power enforces the physical plausibility and obtains the lower sample complexity and substantially better generalization. In future work the physics prior should be extended to represent a wider system class by introducing additional non-conservative forces within the Lagrangian. The mean squared error averaged of 20 seeds on the training-(a) and test-set (b) of the character trajectories for the two joint robot. The models are trained offline using n characters and tested using the remaining 20 − n characters. The training samples are corrupted with white noise, while the performance is tested on noise-free trajectories. To evaluate the performance of DeLaN without the control task, DeLaN was trained offline on previously collected data and evaluated using the mean squared error (MSE) on the test and training set. For comparison, DeLaN is compared to the system identification approach (SI) described by BID1, a feed-forward neural network (FF-NN) and the Recursive Newton Euler algorithm (RNE) using an analytic model. For this comparison, one must point out that the system identification approach relies on the availability of the kinematics, as the Jacobians and transformations w.r.t. to every link must be known to compute the necessary features. In contrast, neither DeLaN nor the FF-NN require this knowledge and must implicitly also learn the kinematics. FIG6 shows the MSE averaged over 20 seeds on the character data set executed on the two-joint robot. For this data set, the models are trained using noisy samples and evaluated on the noise-free and previously unseen characters. The FF-NN performs the best on the training set, but overfits to the training data. Therefore, the FF-NN does not generalize to unseen characters. In contrast, the SI approach does not overfit to the noise and extrapolates to previously unseen characters. In comparison, the structure of DeLaN regularizes the training and prevents the overfitting to the corrupted training data. Therefore, DeLaN extrapolates better than the FF-NN but not as good as the SI approach. Similar can be observed on the cosine data set using the Barrett WAM simulated in SL FIG7. The FF-NN performs best on the training trajectory but the performance deteriorates when this network extrapolates to higher velocities. SI performs worse on the training trajectory but extrapolates to higher velocities. In comparison, DeLaN performs comparable to the SI approach on the training trajectory, extrapolates significantly better than the FF-NN but does not extrapolate as good as the SI approach. For the physical system FIG7, the differ from the in simulation. On the physical system the SI approach only achieves the same performance as RNE, which is significantly worse compared to the performance of DeLaN and the FF-NN. When evaluating the extrapolation to higher velocities, the analytic model and the SI approach extrapolate to higher velocities, while the MSE for the FF-NN significantly increases. In comparison, DeLaN extrapolates better compared to the FF-NN but not as good as the analytic model or the SI approach. This performance difference between the simulation and physical system can be explained by the underlying model assumptions and the robustness to noise. While DeLaN only assumes rigidbody dynamics, the SI approach also assumes the exact knowledge of the kinematic structure. For simulation both assumptions are valid. However, for the physical system, the exact kinematics are unknown due to production imperfections and the direct cable drives applying torques to flexible joints violate the rigid-body assumption. Therefore, the SI approach performs significantly worse on the physical system. Furthermore, the noise robustness becomes more important for the physical system due to the inherent sensor noise. While the linear regression of the SI approach is easily corrupted by noise or outliers, the gradient based optimization of the networks is more robust to noise. This robustness can be observed in Figure 9, which shows the correlation between the variance of Gaussian noise corrupting the training data and the MSE of the simulated and noise-free cosine trajectories. With increasing noise levels, the MSE of the SI approach increases significantly faster compared to the models learned using gradient descent. Concluding, the extrapolation of DeLaN to unseen trajectories and higher velocities is not as good as the SI approach but significantly better than the generic FF-NN. This increased extrapolation compared to the generic network is achieved by the Lagrangian Mechanics prior of DeLaN. Even though this prior promotes extrapolation, the prior also hinders the performance on the physical robot, because the prior cannot represent the dynamics of the direct cable drives. Therefore, DeLaN performs worse than the FF-NN, which does not assume any model structure. However, DeLaN outperforms the SI approach on the physical system, which also assumes rigid-body dynamics and requires the exact knowledge of the kinematics.

This paper introduces a physics prior for Deep Learning and applies the resulting network topology for model-based control.

Represent the short summary for finding the corresponding scientific article (long):

This paper proposes a new approach to incorporating desired invariance to representations learning, based on the observations that the current state-of-the-art AFL has practical issues.