Question:
represent the scientific article to find a short tldr

Given Classification systems typically act in isolation, meaning they are required to implicitly memorize the characteristics of all candidate classes in order to classify. The cost of this is increased memory usage and poor sample efficiency. We propose a model which instead verifies using reference images during the classification process, reducing the burden of memorization. The model uses iterative non-differentiable queries in order to classify an image. We demonstrate that such a model is feasible to train and can match baseline accuracy while being more parameter efficient. However, we show that finding the correct balance between image recognition and verification is essential to pushing the model towards desired behavior, suggesting that a pipeline of recognition followed by verification is a more promising approach towards designing more powerful networks with simpler architectures. FIG3: Overview of our hybrid model in contrast with two opposing approaches. (a) Recognition network directly predicts the class given the input. (b) Verification network predicts binary output indicating the amount of similarity or likelihood that they belong in the same class. The verification network can be used to compare to all reference images from each class to produce the final class prediction. (c) Our approach, RVNN, queries for reference image from a particular class at each time step, and makes a class prediction at the last time step. BID9. Then to classify, the image is pair-wise compared with a support image from every class and 37 image with the maximum similarity score is chosen. Matching networks extends the verification-38 based approach by outputting a prediction based on a weighted-sum of similarity across classes. Additionally the work introduces an episodic-training regime which encourages the model to better 40 learn for the one-shot learning scenario. Prototypical Networks uses Euclidean Distance in embedding 41 space as a verification metric rather than a learned metric, while maintaining the same training regime 42 as Matching Networks to encourage different classes to have distant means in embedding space. One recent work outside of few-shot learning domain is the Retrieval-Augmented Convolutional Neural Networks (RaCNN), which combines CNN recognition network with a retrieval engine 45 for support images to help increase adversarial robustness. For all the above few shot learning approaches, verification with support images from all classes are required before a classification decision is made. Hence the classification decision is solely derived 48 from verifications. RaCNN is closer to our approach, which uses a hybrid between verification 49 and recognition. However, RaCNN simply retrieves the K closest support image neighbours in the 50 embedding space, whereas our model is required to form a hypothesis as to which class to compare 51 with. In cases in which there are a large number of classes, we expect our approach to excel. As well, this introduces a non-differentiable component in our model not present in previous work. Prior work has also looked at the concept of incorporating external knowledge into the decision 55 making process of neural networks, often with non-differentiable components. Buck order to produce a prediction C for the class. FIG2 illustrates the full process. The subsequent sections detail the implementations of the three components as well as training 84 considerations. The recurrent querying model f rnn was implemented using a Gated Recurrent Unit (GRU). We 105 also considered passing in additional information to f rnn, such as the query that was used in the 106 previous time steps. We implemented f q as sampling a class based on the categorical probability given by the softmax of 109 the logits from the f rnn. Therefore, f q can be written as: DISPLAYFORM0 DISPLAYFORM1 To sample S n+1, we can use the Gumbel-Max trick: DISPLAYFORM2 where g i...g k are i.i.d samples drawn from Gumbel distribution. However, the arg max operator 112 is not differentiable, so instead we explored two approaches during training. The first is to use the Gumbel-Softmax trick, also known as the Concrete estimator. We 114 relaxed the arg max operator to a differentiable softmax function with temperature parameter τ: DISPLAYFORM0 The τ parameter is annealed exponentially from τ = 1 to τ = 0.5 as the training iterations progresses. The second approach is to use simple Straight-Through estimator BID0. In the forward pass, we apply 117 the Gumbel-Max trick to take discrete query choices. Then on the backward pass, we set the derivative 118 of the query with respect to the softmax probabilities to be identity so that the out-going gradient 119 from the arg max operator is equal to the incoming gradient during backpropagation: DISPLAYFORM0 We highlight that our model differ from several existing networks in various aspects. The performance of the model is assessed by both reduced parameter usage and sample efficiency. Reduced parameter usage is measured relative to a baseline model, in this case the CNN architecture The performance of the model alone does not indicate whether our approach is functioning as intended. We also experiment with several small modifications to our architecture as well as a few hyper- parameters that are unique to our model. We list them here below. • Architectural Considerations -Query Memory (QM): The query from the past time step is passed to the RNN. the RNN learns to adapt its policy to varying levels of accuracy, and see whether a better can 184 be achieved when they are used in tandem. We also experiment with fixing query policy to assess 185 whether the RNN actually learns intelligent query behavior. From our overall performance metrics we observe that at both smaller and larger sizes of model, our usage. This is in agreement with our hypotheses that our model would be more parameter efficient. RNN's learned policy is compared against a random query policy and the optimal query policy (never 218 repeating a query). From the figure we see that the model is able to conduct a better than random 219 query policy, but is not able to achieve optimal performance. We also observed that performance 220 increases with a higher RNN size up to 200. This suggests that the model is in some part able to track 221 previously unsuccessful queries and remember if there was a match. However, its memory is not 222 perfect and it cannot achieve optimal performance. and a real (neural network) comparator. Of the query-based models, informed queries performed the best over random and no query models. appropriate pipeline for our model is to perform a recognition operation which is then followed by 241 verification, rather than perform them simultaneously. This new pipeline would also imply that our model may be less well suited for the one-shot learning 243 task than initially believed, as a reasonably-well trained recognition module is required as the first tasks of this form are pre-requisite if neural-network models are considered to be as such BID5., a positive would be Image classification via iteratively querying for reference image from a candidate class with a RNN and use CNN to compare to the input image & a negative would be theoretically understand the regularization effect of distillation. We show that early stopping is essential in this process. From this perspective, we developed a distillation method for learning with corrupted Label with theoretical guarantees.

Question: A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model. To address this issue, in this paper, we develop knowledge flow which moves ‘knowledge’ from multiple deep nets, referred to as teachers, to a new deep net model, called the student. The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too. Upon training with knowledge flow the student is independent of the teachers. We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming fine-tuning and other ‘knowledge exchange’ methods. Research communities have amassed a sizable number of deep net architectures for different tasks, and new ones are added almost daily. Some of those architectures are trained from scratch while others are fine-tuned, i.e., before training, their weights are initialized using a structurally similar deep net which was trained on different data. Beyond fine-tuning, particularly in reinforcement learning, teachers have also been considered in one way or another by BID23; BID6; BID30; BID13; BID0 BID21; BID2; BID26; BID20. For instance, progressive neural net BID23 keeps multiple teachers during both training and inference, and learns to extract useful features from the teachers for a new target task. PathNet BID6 uses genetic algorithms to choose pathways from a giant network for learning new tasks.' Growing a Brain' BID30 fine-tunes a neural network while growing the network's capacity (wider or deeper layers). Actor-mimic BID20 pre-trains a big model on multiple source tasks, then the big model is used as a weight initialization for a new model which will be trained on a new target task. Knowledge distillation BID9 distills knowledge from a large ensemble of models to a smaller student model. However, all the aforementioned techniques have limitations. For example, progressive neural net models BID23 grow with the number of teachers. This large number of parameters limits the number of teachers a progressive neural net can handle, and largely increases the training and testing time. In PathNet BID6, searching over a big network for pathways is computationally intensive. For fine-tuning based methods such as'Growing a Brain' BID30 and actor-mimic BID20, only one pretrained model can be used at a time. Hence, their performance heavily relies on the chosen pretrained model. To address these shortcomings, we develop knowledge flow which moves'knowledge' of multiple teachers when training a student. Irrespective of how many teachers we use, the student is guaranteed to become independent at the final stage of training and the size of the ing student net remains constant. In addition, our framework makes no restrictions on the deep net size of the teacher and student, which provides flexibility in choosing teacher models. Importantly, our approach is applicable to a variety of tasks from reinforcement learning to fully-supervised training. We evaluate knowledge flow on a variety of tasks from reinforcement learning to fully-supervised learning. In particular, we follow BID23; BID6 and compare on the same ∞ k=0 γ k r t+k, where γ is the discount factor. The expected future reward when observing state x and when following policy π θπ is defined as V π θπ (x t) = E τ ∼π θπ [R t |x t], where τ = {(x t, a t, r t), (x t+1, a t+1, r t+1),...} is a trajectory generated by following π θπ from state x t.The goal of reinforcement learning is to find a policy that maximizes the expected future reward from each state x t. Without loss of generality, in this paper, we follow the asynchronous advantage actor-critic (A3C) formulation BID17. In A3C, the policy mapping π θπ (x) = arg max a∈Aπθπ (a|x) is obtained from a probability distribution over states, wherê π θπ (a|x) is modeled by a deep net with parameters θ π. The value function is also approximated by a deep net V θv (x), having parameters θ v.To optimize the policy parameters θ π given a state x t, a loss function based on a scaled negative log-likelihood and a negative entropy regularizer is common: DISPLAYFORM0 [− logπ θπ (a t |x t)(R t − V θv (x t)) − βH(π θπ (·|x t))].Hereby, R t = k−1 i=0 γ i r t+i + γ k V θv (x t+k) is the empirical k-step return obtained when starting in state x t, and |τ | is the length of the trajectory τ generated by following π θπ. The scalar β ≥ 0 is a user-specified constant, and H(π θπ (·|x t)) is the entropy function, which encourages exploration by favoring a uniform probability distributionπ θπ (a|x). To optimize the value function V θv, it is common to use the squared loss DISPLAYFORM1 By minimizing the empirical expectation of τ π (θ π) and τ v (θ v), i.e., by addressing DISPLAYFORM2 alternatingly, we learn a policy and a value function that maximize expected return. Instead of optimizing the programs given in Eq. and Eq. from scratch, the aforementioned warm-start techniques (see Sec. 5 for more) are applicable. To address their mentioned shortcomings, we propose knowledge flow, a framework that moves'knowledge' from an arbitrary number of deep nets, henceforth referred to as'teachers' to a deep net under training, called the'student.' (c) Average normalized weights for teachers' and the student's layers. At the beginning of training, the student heavily relies on teacher one. As training progresses, teacher one's weight decreases, and the student's weight increases until the student is eventually independent. DISPLAYFORM0 Knowledge flow is outlined on example deep nets in FIG0. We train the parameters of the student net which are randomly initialized. To this end we take advantage of teachers, whose parameters are fixed and obtained from pre-trained models on different source tasks by different algorithms. For example, for reinforcement learning, we may consider teachers trained by A3C BID17, A2C or DQN BID16.'Knowledge' of multiple teachers is transferred to a student by adding transformed and scaled intermediate representations from the teacher deep nets to the student net. To achieve this, we modify the student net, i.e., f θ in the supervised setting and π θπ (a|x), V θv (x) in the reinforcement learning case. We add teacher representations which are transformed by multiplication with a trainable matrix Q and scaled via a weight p w that is normalized to sum to one for each student layer and parameterized via trainable parameters w. The normalized weights encode which of the teachers' or the student's representation to trust at every layer of the student net. Note that a teacher can help the student at different levels of abstraction with input from different levels of its net. Importantly, after training, the student model should perform well on the target task without relying on teachers. To achieve this, as training progresses, we increasingly encourage a high normalized weight on the student representation, which forces the student to eventually capture all the'knowledge.' Due to the trainable scaling, at an early stage of training, we observe the student to rely heavily on the'knowledge' of the teacher to quickly obtain better performance. However, as training proceeds, the student is encouraged to become more and more independent. During final stages of training, the student will no longer be able to rely on teachers, which ensures that the student has learned to master the desired task on its own. This is observed in FIG0 To formally encourage this successive transfer we introduce two additional loss functions. The first, referred to as the dependency loss dep (w), captures how much a student relies on teachers. It depends on the weight vector w which encodes the strength of the coupling. The second one ensures that a student's behavior doesn't change rapidly when the teachers' influence decreases. We use loss KL (·, ·) to capture the change. By combining student net modifications and additional loss terms, for the supervised task we obtain DISPLAYFORM0 and for reinforcement learning the transformed program reads as follows: DISPLAYFORM1 Loss˜ · · (θ, w, Q) originates from the original loss · · (θ) (Eqs. FORMULA3 - FORMULA2) by transforming the deep net to include cross-connections, hence its dependence on w, Q. The tilde ('·') denotes this dependence, also for probability distributionf and policy distributionπ. Parameters from the current and a previous iteration are referred to via θ and θ old respectively. For both supervised and reinforcement learning, λ 1 and λ 2 control the strength which is used to decrease the influence of the teacher. A low λ 1 allows the student to rely on teachers. Close to the end of training, the student should be independent. Therefore, we set λ 1 to a small value at the beginning, and gradually increase its value as training progresses. Note that we don't make any assumptions about teachers and student's objective. If a teacher's and student's objective differ, negative transfer may occur initially. However, the proposed method quickly decreases the weight for teacher layers to reduce this effect. Despite differences, students could potentially still benefit from the low level representation of the teachers. We do observe this low level knowledge transfer in our experiments. In the following we first describe how to modify the deep nets, before we detail the loss functions dep and KL, which are used to successively decrease the influence of the teachers. For each layer j in the student model, we define a candidate set L j, which contains l j 0 and all the teachers' layers to be considered. For example, in FIG0, layer one of the student model is combined with layer one of teacher one and layer two of teacher two. Therefore, the candidate set of layer one of the student model is given by DISPLAYFORM2 To decide which teachers' or the student's representation to trust at every layer of the student net, we introduce a normalized weight p j w (l) for all j ∈ {1, . . ., L 0}, where l ∈ L j, summing to one for each layer j in the student deep net, i.e., DISPLAYFORM3 To obtain the combined intermediate representation of layer j for the student model, we use DISPLAYFORM4 DISPLAYFORM5 The maximal number of introduced matrices Q in our framework is DISPLAYFORM6 In practice, we don't link a student's layer to every layer of a teacher network. Intuitively, a teachers' bottom layer BID6 and progressive neural network (PNN) BID23. Since PathNet and PNN don't report exact scores we obtain their numbers from their plots and indicate that with a ∼ symbol. The of the state-of-the-art methods: A3C BID17, PPO, and ACKTR BID31 features are very likely irrelevant to a student's top layer features. Indeed, we observed that linking a teachers' bottom layer to a student's top layer generally doesn't yield improvements. Therefore, in practice, we recommend to link one teacher layer to one or two student layers, in which case we introduce on the order of M L 0 matrices Q. Also note that while additional trainable parameters Q and w are introduced in our framework, Q and w are not part of the ing student network since we ensure p DISPLAYFORM7 at the end of training as discussed next. Hence, the additional parameters function as auxiliary knobs that help the student learn faster. In the final stage of training, the student will be independent (see FIG0 (c)) and does no longer rely on Q, w, or any transformed representations from teachers. Decreasing Teachers' Influence: We successively decrease the influence of the teachers during training by gradually encouraging the normalized weight p j w (l j 0) to increase to a value of 1 ∀j ∈ {1, 2, . . ., L 0}. To capture how much the student relies on teachers, we introduce the dependence cost as the negative log probability: DISPLAYFORM8 By minimizing dep (w), we encourage weights for the layers of the student to increase. Hence we encourage the student to become more and more independent. During the final stage of training, p j w (l j 0) approaches one for all j ∈ {1, . . ., L 0}, making the student independent of the transformed representation obtained from teachers. Empirically, we found that a fast decrease of the influence of the teacher can degrade the performance. This is intuitive as it requires some time to find good transformations Q. Moreover, decreasing the influence of a teacher too fast may change the output distribution over labels or actions of the student model too much, and thus lead to performance loss. To prevent changing a student's output distribution too fast, we found a Kullback-Leibler (KL) regularizer to yield good . More specifically, in the case of supervised learning we use DISPLAYFORM9 Hereby, θ is the set of current parameters, and θ old are the previous ones. In the reinforcement learning case we use DISPLAYFORM10 In the following we evaluate knowledge flow on reinforcement and supervised learning tasks. Results are reported by using only the student model to avoid even the smallest influence from any teacher nets. We evaluate knowledge flow on reinforcement learning using Atari games that were used by BID23 BID6. Following existing work, the input to our agent are raw images from the environment. The agent learns to predict actions only based on the rewards and the input images from the environment. The agent chooses an action every four frames, and the last action is repeated on the skipped four frames. For all teacher models and the student model, we use the fully forward architecture of A3C BID17. The model has three hidden layers. The first layer is a convolutional layer with 16 filters of size 8x8 and stride 4. The second layer is a convolutional layer with 32 filters of size 4x4 and stride 2. The third layer is a fully connected layer with 256 hidden units. Following the third hidden layer are two sets of output. One is a softmax output that provides a probability distribution over all valid actions. The other one is a scalar output that provides the estimated value function. We use the same hyper-parameter settings as BID17 except for the learning rate. BID17 use RMSProp with shared statistics while we use Adam with shared statistics, which we found to give better when training the baselines. The learning rate is set to 10 −4 and gradually decreased to zero for all experiments. To select λ 1 and λ 2 in our framework, we follow progressive neural net BID23: randomly sample λ 1 ∈ {0.05, 0.1, 0.5} and λ 2 ∈ {0.001, 0.01, 0.05}. Note that λ 1 is set to zero at the beginning of training, and linearly increased to the sampled value at the end of training. Following BID23, we repeat each experiment 25 times with different random seeds and randomly sampled λ 1 and λ 2. The of the top three out of 25 runs are reported. As A3C, we run 16 agents on 16 CPU cores in parallel. Evaluation Metrics: We follow the evaluation procedure of BID16. The trained student models are evaluated by playing each game for 30 episodes. We also follow the'no-op' procedure: at the beginning of each testing episode, the agents perform up to 30'no-op' actions. Results: We first compare our framework with PathNet BID6 and progressive neural net (PNN) BID23, which are state-of-the-art transfer reinforcement learning frameworks, using their experimental settings. The comparison is summarized in TAB0. The state-of-the-art BID17 BID31 on Atari games are also included in TAB0 for reference. Compared to PathNet, a student model trained using our transfer framework with one teacher achieves higher scores in 11 out of 14 experiments. Compared with PNN, for a two-teacher framework, our trained student model has only 0.7M parameters and PNN has 16M parameters. Nonetheless we observe higher scores in five out of the seven experiments. The demonstrate that knowledge flow effectively transfers knowledge from teachers to the student. TAB0 also indicates that, in our framework, when the number of teachers increases from one to two, the student's performance improves significantly across all experiments. The training curves for the experiments are shown in FIG1. The curve is the average of the top three out of 25 runs. We observe our approach to generally perform very well. To further evaluate knowledge flow, we experiment with different combinations of environment/teacher settings. These settings are not used by PathNet and progressive neural network. The are summarized in TAB2, where "ours w/ expert" represents that one teacher is expert for the target game; "ours w/ non-expert" represents that both teachers are not experts for the target game; "Fine-tune" represents fine-tuning from a non-expert on a new target game; "A3C baseline" represents our implementation of the A3C baseline; "A3C" represents the scores reported originally BID17. Note that our A3C implementation achieves better scores than those reported by BID17 for most of the games. As shown in TAB2, knowledge flow with expert teacher performs better than the baseline across all experiments, which we interpret as evidence that knowledge flow successfully transfers'knowledge' from an expert teacher to the student. In addition, knowledge flow with non-expert teachers also outperforms fine-tuning on a non-expert teacher. The reasons are twofold: First, a student model in knowledge flow can learn from multiple teachers while the fine-tuning method can only start from one setting. Second, in knowledge flow, the student can avoid the negative impact from insufficiently pretrained teachers, while fine-tuning from an insufficiently pretrained model slows down the training process and may degrade the overall performance. The training curves for the experiments are shown in FIG5. More training curves are in the Appendix (Fig. 6). Note that in knowledge flow, the student can benefit from the intermediate representations of the teacher, even if input space, output space and objectives differ. For example, in FIG5, the two teachers are Chopper Command and Space Invaders, which are quite different from the target game Seaquest. The student model still benefits from learning from the teachers and achieves scores ten times larger than learning without teacher and fine-tuning from a teacher. For supervised learning, we use a variety of image classification benchmarks, including CIFAR-10 , CIFAR-100 BID12, STL-10 , and EM-NIST BID4. The parameters λ 1 for the dependent cost and λ 2 for the KL cost are determined using the validation set of each dataset. Evaluation Metrics: To evaluate the trained student model we report top-1 error rate on the test set of each dataset. All plots and reported numbers are the average of three runs obtained using different random seeds. (b) CIFAR-10/CIFAR-100: CIFAR-10 and CIFAR-100 datasets consist of colored images of size 32 × 32. CIFAR-10 (C10) has 10 classes and CIFAR-100 (C100) has 100 classes. For both dataset, the training and test sets contain 50,000 and 10,000 images respectively. We perform all experiments on CIFAR-10 and CIFAR-100 with standard data augmentation BID10.We use Densenet BID10 ) (depth 100, growth rate 24) as a baseline and follow their hyper-parameter settings to train our baseline, teacher and student models. For our approach, we first train teachers on CIFAR-10, CIFAR-100, and SVHN BID18. We then train the student model using a different combination of teachers. We compare our to fine-tuning and the baseline model. As shown in TAB3 (a), for the CIFAR-10 target task, fine-tuning from the CIFAR-100 expert improves 4% over the baseline. Fine-tuning from the SVHN expert performs worse than the baseline model. Intuitively, for the CIFAR-10 target task, the CIFAR-100 deep net is a good teacher while a deep net trained with SVHN isn't. Presented with both good and inadequate teachers, knowledge flow improves by 13% over the baseline. This demonstrates that knowledge flow can not only leverage a good teacher's'knowledge,' but it can also avoid misleading influence. As detailed in TAB3 (b), the are similar on the CIFAR-100 dataset. To further demonstrate the properties of knowledge flow, additional are in the appendix. As mentioned before,'knowledge' transfer has been considered using a variety of techniques. We briefly discuss related work in contrast to our approach in the following and defer details to Sec. 8. PathNet BID6 enables multiple agents to train the same deep net while reusing parameters and avoiding catastrophic forgetting. In contrast to this formulation we consider availability of multiple pre-trained teacher nets. Progressive Net BID23 ) leverages transfer and avoids catastrophic forgetting by introducing lateral connections to previously learned features. Our discussed method uses similar lateral connections. However, in contrast to BID23, our method ensures independence of the student upon training, addressing a limitation in BID23 where only a fraction of the capacity of the student is eventually utilized. Distral a neologism combining'distill & transfer learning' BID26 considers joint training of multiple tasks. Multiple tasks share a'distilled' policy which encodes common behavior between different tasks. While each worker addresses its own task, a shared policy encourages consistency between the policies. Different from Distral, which is a multi-task learning framework, knowledge flow addresses a single task, while in multi-task learning, multiple tasks are addressed at the same time. Hence, common for multi-task learning and knowledge flow is a transfer of information. However, in multi-task learning, information extracted from different tasks are shared to boost performance, while, in knowledge flow, the information of multiple teachers is leveraged to help a student learn better a single, new, previously unseen task. Other related work includes actor-mimic BID20, learning without forgetting BID13, growing a brain BID30, policy distillation, domain adaptation BID19, knowledge distillation BID9 or lifelong learning BID2. A more detailed discussion on related work is provided in Sec. 8 of the supplementary material. We developed a general knowledge flow approach that permits to train a deep net from any number of teachers. We showed for reinforcement learning and supervised learning, demonstrating improvements compared to training from scratch and to fine-tuning. In the future we plan to learn when to use which teacher and how to actively swap teachers during training of a student. BID9 to distill knowledge from a larger model (teacher) to a smaller model (student). The student models have 50% -5% parameters of the teacher models. Following their setup, we conduct experiments on MNIST, MNIST with digit'3' missing in the training set, CIFAR-100, and ImageNet. For MNIST and MNIST with digit'3' missing, following KD, the teacher model is an MLP with two hidden layers of 1200 hidden units, and the student model is an MLP with two hidden layers of 800 hidden units. For CIFAR-100, we use the model from Chen FORMULA2 as teacher model. The student model follows the structure of the teacher, but the number of output channels of each convolutional layer is halved. For ImageNet, the teacher model is a 50-layer ResNet BID8, and the student model is a 18-layer ResNet. The test error of the distilled student model are summarize in TAB4. Our framework has consistently better performance than KD, because the student model in our framework benefits not only from the output layer behavior of the teacher but also from intermediate layer representations of the teacher. The'EMNIST Letters' dataset consists of images of size 28 × 28 pixels showing handwritten letters. It has 26 balanced classes. Each class contains lower and upper case letters. The training and test sets contain 124,800 and 20,800 images respectively. The'EMNIST Digits' dataset consists of images of size 28 × 28 pixels showing handwritten digits. It has 10 balanced classes. The training and test sets contain 240,000 and 40,000 images respectively. In this case we use the MNIST model from as a baseline, teacher and student model. We trained teachers on EMNIST Digits, EMNIST Letters, and EMNIST Letters with only 13 classes. Our target task is EMNIST Letters. The student model is trained with different teachers and the are compared to fine-tuning, the baseline model, and the state-of-the-art on EMNIST.The are summarized in Table 5. Compared to the baseline and fine-tuning, student learning in our framework with expert teacher (EMNIST Letters), semi-expert teacher (Half EMNIST Letters), and non-expert teacher (EMNIST Digits) all have better performance. In FIG6 we illustrate the accuracy over epochs for training of different models. The STL-10 dataset consist of colored images of size 96 × 96 pixels. It has 10 balanced classes. The training set contains 5,000 labeled images and 100,000 unlabeled images. The test set contains 8,000 images. In our experiment, we only use the 5,000 labeled images for training. We use the STL-10 model from as our baseline, teacher and student model. We trained teachers on CIFAR-10 and CIFAR-100. We compare our to fine-tuning and the baseline in Table 6. Note that STL-10 is very similar to CIFAR-10 and CIFAR-100. Therefore, both CIFAR-10 and CIFAR-100 are very good teachers. As shown in Table 6, compared to the baseline, fine-tuning a Table 6: Our approach on the STL-10 dataset (fully supervised). Zhao et al. FORMULA2 25.20 BID27 21.34Baseline 25.50 Fine-tune from C1014.32 Fine-tune from C100 14.38 Ours (C100) 12.35 Ours (C10, C100) 11.09 model using weights pretrained on CIFAR-10 and CIFAR-100 reduce test errors by more than 10%.Compared with fine-tuning, student model training in our framework further reduces the test error by 3%. Note that we only train on the labeled data while other approaches use this data for testing of semi-supervised approaches. Hence our are obtained using fewer data and may not be directly comparable. We still list their in Table 6 for reference. In Fig. 5 we illustrate the accuracy over the epochs of training. We also compare to Distral BID26, which is the state-of-the-art multi-task reinforcement learning framework. We used'KL + ent 1 col', which has a central model (m 0), and a task model (m i) for each task. We perform the experiments on Atari games. In the experiments, we have three tasks (task 1, task 2, task 3). The teachers of task 2 (m 2) and task 3 (m 3) are provided for our framework. Distral is trained for 120M steps (40M steps/task), and our model is trained for 40M steps. For fair comparison, we report of Distral's task 1 model (m 1), which is better than its center model (m 0). The are summarized in TAB5. Distral is suboptimal, because it aims to learn a multi-task agent. In addition, identical action and state space is assumed. When the target task is very different from the source tasks, Distral cannot decrease the teacher influence. In contrast, our framework can decrease a teacher's influence, and thus reduce negative transfer. Following the reviewer's suggestion, we plot the averaged normalized weight (p w) for teachers and the student in the C10 experiment, where C100 and SVHN experts are teachers. Intuitively, the C100 teacher should have a higher p w value than the SVHN teacher, because C100 is more relevant to C10. The plot verifies this intuition. As shown in FIG8, p w of the C100 teacher is higher than that of the SVHN teacher over the entire training. Note, both teachers' normalized weights approach zero at the end of training. To verify that the student really benefits from the knowledge of teachers, we conduct an ablation study suggested by a reviewer. We use teacher models that haven't been trained at all. Intuitively, learning with untrained teachers should have worse performance than learning with knowledgeable teachers. Our experiments verify this intuition. In Fig. 8 (a), where the target task is hero, learning with untrained teachers ('w/ untrained teachers') achieves an average reward of 15934. Learning with knowledgeable teachers ('Ours with seaquest and riverraid teacher') achieves an average reward of 30928. More are presented in Figs. 8 (b, c). The show that knowledge flow achieves higher rewards than training with untrained teachers in different environments and teacher-student settings. The KL term prevents the student's output distribution over actions or labels from drastic changes when the teachers' influence is decreasing. To investigate the importance of the KL term, we conduct an ablation study where the KL coefficient (λ 2) is set to zero. The is summarized in Fig. 9. Considering Fig. 9 (a), where the target task is MsPacman and the teachers are Riverraid and Seaquest experts. Without the KL term, when a teacher's influence decreases, the rewards drop drastically. In contrast, with a KL term, we don't observe performance drops. At the end of training, learning with the KL term achieves an average reward of 2907 and learning without the KL term achieves an average reward of 1215. More are presented in Fig. 9 (b, c), which shows that training with the KL term achieves higher reward than training without the KL term. In additional experiments, following the suggestion of a reviewer, we use architectures for the teacher which differ from the student model. More specifically, we use the model of BID16 as a teacher model. The teacher model consists of 3 convolutional layers, which have 32, 64, and 64 filters, followed by a hidden fully connected layer which has 512 ReLUs. We use the model of BID17 as the student model. The student model consists of 2 convolutional layers, which have 16 and 32 filters respectively, followed by a hidden fully connected layer which has 256 ReLUs. Both models' fully connected layers are followed by two output layers for actions and values. In the experiments, we link each teacher's first convolutional layer to the student's first convolutional layer. Moreover, we link each teacher's third convolutional layer to the student's second convolutional layer, and each teacher's fully connected layer to the student's fully connected layer. In the experiment, the target task is KungFu Master, and the teachers are experts for Seaquest and Riverraid. The are summarized in FIG0. We observed that learning with teachers, whose architecture differs from the student, to have similar performance as learning with teachers which have the same architecture. Consider as an example FIG0, where the target task is KungFu Master, and the teachers are experts for Seaquest and Riverraid. At the end of training, learning with teachers of different architectures achieves an average reward of 37520, and learning with teachers of the same architecture achieves an average reward of 35012. More are shown in FIG0. The show that knowledge flow can enable higher rewards, even if the teachers and the student architectures differ. 7.6 AVERAGE NETWORK AS θ old For the parameters θ old an average network can be used. To investigate how usage of an average network to obtain the parameters θ old affects the performance, we conduct an experiment where θ old is computed using the exponential running average of the model weight. More specifically, θ old is updated as follows: θ old ← α · θ old + (1 − α) · θ, where α = 0.9. The are summarized in FIG0. We observe that using an exponential average to compute θ old in very similar performance as using a single model. Consider FIG0, where the target task is Boxing and the teacher is a Riverraid expert. At the end of training, using an average network to obtain θ old achieves an average reward of 96.2 and using a single network to obtain θ old achieves an average reward of 96.0. More on using an average network are shown in FIG0 (b, c). As mentioned before, variants of'knowledge' transfer have been considered using a variety of techniques, for instance, fine-tuning, progressive neural nets BID23, PathNet BID6,'Growing a Brain' BID30, actor-mimic BID20, learning without forgetting BID13. Also related are techniques on transfer learning and lifelong learning. We discuss those methods and contrast them to our approach in the following. PathNet BID6 enables multiple agents to train the same giant deep net while reusing parameters and avoiding catastrophic forgetting. To this end, agents embedded in the neural net discover which weights can be reused for new tasks and restrict application of gradients to those parameters. In contrast to this formulation we consider availability of multiple teacher nets, which are trained. Progressive Net BID23 ) leverages transfer and avoids catastrophic forgetting by introducing lateral connections to previously learned features. Our discussed method uses similar lateral connections. However, in contrast to BID23, we introduce scaling with normalized weights. This ensures independence of the student upon training, addressing a limitation in BID23 where only a fraction of the capacity of the student is eventually utilized. Distral a neologism combining'distill & transfer learning' BID26 considers joint training of multiple tasks. Multiple tasks share a'distilled' policy which encodes common behavior between different tasks. While each worker addresses its own task, a shared policy encourages consistency between the policies. Different from Distral, which is a multi-task learning framework, knowledge flow addresses a single task, while in multi-task learning, multiple tasks are addressed at the same time. Hence, common for multi-task learning and knowledge flow is a transfer of information. However, in multi-task learning, information extracted from different tasks are shared to boost performance, while, in knowledge flow, the information of multiple teachers is leveraged to help a student learn better a single, new, previously unseen task. Knowledge distillation BID9 distills information form a larger deep net into a smaller one. It assumes both nets are trained on the same dataset. In contrast, our technique allows knowledge transfer between different source and target domains. Actor-mimic BID20 enables an agent to learn how to address multiple tasks simultaneously and generalize the extracted knowledge to new domains. A single policy net learns how to act in a set of tasks following the guidance of several expert teachers. A combination of feature regression and cross entropy loss is used to encourage the student to produce similar actions and representations. Our proposed technique differs in that we take advantage of a teachers representation at the beginning of training, Learning without forgetting BID13 permits to add a new task to a deep net without forgetting the original capabilities. Importantly, only data from the new task is used and the old capabilities are retained by first recording the old networks output on the new data. Similar techniques have been developed by BID7 BID11. In contrast, we transfer'knowledge' from teacher networks more explicitly.

represent the tldr for finding the corresponding scientific article
For example, 'We apply canonical forms of gradient complexes (barcodes) to explore neural networks loss surfaces.' should be close to 'We apply canonical forms of gradient complexes (barcodes) to explore neural networks loss surfaces. We present an algorithm for calculations of the objective function's barcodes of minima. Our experiments confirm two principal observations: the barcodes of minima are located in a small lower part of the range of values of objective function and increase of the neural network's depth brings down the minima's barcodes. This has natural implications for the neural network learning and the ability to generalize. The learning via finding minima of objective functions is the principal strategy underlying majority of learning algorithms. For example, in Neural Network training, the objective function's input is model parameters (weights) and the objective function's output is the loss on training dataset. The graph of the loss function, often called loss surface, typically has complex structure (e.g. see loss surface visualisations by): non-convexity, many local minima, flat regions, steep slopes. These obstacles harm exploration of the loss surface and complicate searching for optimal network weights. The optimization of modern neural networks is based on the gradient descent algorithm. The global topological characteristics of the gradient vector field trajectories are captured by the Morse complex via decomposing the parameter space into cells of uniform flow, see; and references therein. The invariants of Morse complex called "canonical forms"(or barcodes) constitute the fundamental summary of the topology of the gradient vector field flow. The "canonical forms", or barcodes, in this context are decompositions of the change of topology of the sublevel sets of objective function into simple "birth-death" phenomena of topological feautures of different dimensions. The calculation of the barcodes for different functions constitutes the essence of the topological data analysis. The currently available software packages for the calculation of barcodes of functions, also called "sublevel persistence", are GUDHI, Dionysus, PHAT, and TDA package which incorporates all three previous packages B.T.. They are based on the algorithm, described in , see also appendix and e.g. and references therein. This algorithm which has complexity of O(n 3). These packages can currently handle calculations of barcodes for functions defined on a grid of up to 10 6 points, and in dimensions two and three. Thus all current packages have the scalability issues. We describe a new algorithm for computations of the barcodes of functions in lowest degree. Our algorithm works with functions defined on randomly sampled or specifically chosen point clouds. Point cloud based methods are known to work better than grid based methods in optimization related problems . We also use the fact that the definition of the barcode of lowest degree can be reformulated in geometrical terms (see definition 1 in section 2). The previously known algorithms were based on the more algebraic approach as in definition 3. Our algorithm has complexity of O(n log(n)). It was tested in dimensions up to 16 and with number of points of up to 10 8. In this work, we develop a methodology to describe the properties of the loss surface of the neural network via topological features of local minima. We emphasize that the value of the objective function at the minimum can be viewed as only a part of its topological characteristic from the "canonical form" (barcode). The second half can be described as the value of objective function at the index-one saddle, which can be naturally associated with each local minimum. The difference between the values of objective function at the associated index-one saddle and at the local minimum is a topological invariant of the minimum. For optimization algorithms this quantity measures, in particular, the obligatory penalty for moving from the given local minimum to a lower minimum. The main contributions of the paper are as follows: Applying the one-to-one correspondence between local minima and 1-saddles to exploration of loss surfaces. For each local minimum p there is canonically defined 1-saddle q (see Section 2). The 1-saddle associated with p can be described as follows. The 1-saddle q is precisely the point where the connected component of the sublevel set Θ f ≤c = {θ ∈ Θ | f (θ) ≤ c} containing the minimum p merges with another connected component of the sublevel set whose minimum is lower. This correspondence between the local minima and the 1-saddles, killing a connected component of Θ f ≤c, is one-to-one. The segment [f (p), f (q)] is then the "canonical form" invariant attached to the minimum p. The set of all such segments is the barcode ("canonical form") of minima invariant of f. It is a robust topological invariant of objective function. It is invariant in particular under the action of homeomorphisms of Θ. Full "canonical form" invariants give a concise summary of the topology of objective function and of the global structure of its gradient flow. Algorithm for calculations of the barcodes (canonical invariants) of minima. We describe an algorithm for calculation of the canonical invariants of minima. The algorithm works with function's values on a a randomly sampled or specifically chosen set of points. The local minima give birth to clusters of points in sublevel sets. The algorithm works by looking at neighbors of each point with lower value of the function and deciding if this point belongs to the existing clusters, gives birth to a new cluster (minimum), or merges two or more clusters (index one saddle). A variant of the algorithm has complexity of O(n log(n)), where n is the cardinality of the set of points. Calculations confirming observations on behaviour of neural networks loss functions barcodes. We calculate the canonical invariants (barcodes) of minima for small fully-connected neural networks of up to three hidden layers and verify that all segments of minima's barcode belong to a small lower part of the total range of loss function's values and that with the increase in the neural network depth the minima's barcodes descend lower. The usefulness of our approach and algorithms is clearly not limited to the optimization problems. Our algorithm permits really fast computation of the canonical form invariants (persistence barcodes) of many functions which were not accessible until now. These sublevel persistence barcodes have been successfully applied in different discipline, to mention just a few: cognitive science (M. K.), cosmology , see e.g. and references therein. Our viewpoint should also have applications in chemistry and material science where 1-saddle points on potential energy landscapes correspond to transition states and minima are stable states corresponding to different materials or protein foldings (see e.g. ,). The article is structured as follows. First we describe three definitions of barcodes of minima. After that our algorithm for their calculation is described. In the last part we give examples of calculations, including the loss functions of simple neural nets. The "canonical form" invariants (barcodes) give a concise summary of topological features of functions (see , and references therein). These invariants describe a decomposition of the change of topology of the function into the finite sum of "birth"-"death" of elementary features. We propose to apply these invariants as a tool for exploring topology of loss surfaces. In this work we concentrate on the part of these canonical form invariants, describing the "birth"-"death" phenomena of connected components of sublevel sets of the function. However it should be stressed that this approach works similarly also for "almost minima", i.e. for the critical points (manifolds) of small indexes, which are often the terminal points of the optimization algorithms in very high dimensions. We give three definitions of the "canonical form" invariants of minima. The values of parameter c at which the topology of sublevel set Let p be one of minima of f. When c increases from f (p)− to f (p)+, a new connected component of the set Θ f ≤c is born (see fig 1a, the connected components S 1, S 2, S 3 of sublevel set are born at the blue, green and red minima correspondingly. If p is a minimum, which is not global, then, when c is increased, the connected component of Θ f ≤c born at p merges with a connected component born at a lower minimum. Let q is the merging point where this happens. The intersection of the set Θ f <f (q) with any small neighborhood of q has two connected components. This is the index-one saddle q associated with p. (a) "Death" of the connected component S3. The connected component S3 of sublevel set merges with connected component S2 at red saddle, red saddle is associated with the red minimum. (b) "Death" of the connected component S4. The connected component S4 of sublevel set merges with connected component S1 at violet saddle, violet saddle is associated with the violet minimum (c) "Death" of the connected component S2. The connected component S2 of sublevel set merges with connected component S1 at green saddle, green saddle is associated with the green minimum. Figure 1: Merging of connected components of sublevel sets at saddles. Note that the green saddle is associated with the green minimum which is separated by another minimum from the green saddle. Also these two subsets of small neighborhood of q belong to two different connected components of the whole set Θ f <f (q). The 1-saddles of this type are called "+" ("plus") or "death" type. The described correspondence between local minima and 1-saddles of this type is one-to-one. In a similar way, the 1-saddle q associated with p can be described also as follows. Proposition 2.1. Consider various paths γ starting from the local minimum p and going to a lower minimum. Let m γ ∈ Θ is the maximum of the restriction of f to such path γ. Then 1-saddle q which is paired with the local minimum p is the minimum over the set of all such paths γ of the maxima m γ: The correspondence in the opposite direction can be described analogously. Let q is a 1-saddle point of such type that the two branches of the set Θ f ≤f (q)− near q belong to two different connected components of Θ f ≤f (q)−. A new connected component of the set Θ f ≤c is formed when c decreases from f (q) + to f (q) −. The restriction of f to each of the two connected components has its global minimum. Proposition 2.2. Given a 1-saddle q, the minimum p which is paired with q is the new minimum of f on the connected component of the set Θ f ≤c which is formed when c decreases from f (q) + to f (q) −. The two branches of the set Θ f ≤f (q)− near q can also belong to the same connected components of this set. Then such saddle is of "birth" type and it is naturally paired with index-two saddle of "death" type (see theorem 2.3). Chain complex is the algebraic counterpart of intuitive idea representing complicated geometric objects as a decomposition into simple pieces. It converts such a decomposition into a collection of vector spaces and linear maps. A chain complex (C *, ∂ *) is a sequence of finite-dimensional k-vector spaces and linear operators The j−th homology of the chain complex (C *, ∂ *) is the quotient A chain complex C * is called R−filtered if C * is equipped with an increasing sequence of sub- by a finite set of real numbers s 1 < s 2 <... < s max. Theorem 2.3. Any R−filtered chain complex C * can be brought by a linear transformation preserving the filtration to "canonical form", a canonically defined direct sum of R−filtered complexes of two types: one-dimensional complexes with trivial differential ∂ j (e i) = 0 and two-dimensional complexes with trivial homology ∂ j (e i2) = e i1. The ing canonical form is uniquely determined. The full barcode is a visualization of the decomposition of an R−filtered complexes according to the theorem 2.3. Each filtered 2-dimensional complex with trivial homology ∂ j (e i2) = e i1, e i1 = F ≤s1, e i1, e i2 = F ≤s2 describes a topological feature in dimension j which is "born" at s 1 and which "dies" at s 2. It is represented by segment [s 1, s 2] in the degree-j barcode. And each filtered 1-dimensional complex with trivial differential, ∂ j e i = 0, e i = F ≤r describes a topological feature in dimension j which is "born" at r and never "dies". It is represented by the half-line [r, +∞[ in the degree-j barcode. The proof of the theorem is given in Appendix. Essentially, one can bring an R−filtered complex to the required canonical form by induction, starting from the lowest basis elements of degree one, in such a way that the manipulation of degree j basis elements does not destroy the canonical form in degree j − 1 and in lower filtration pieces in degree j. Let f : Θ → R is smooth, or more generally, piece-wise smooth continuous function such that the sublevel sets Θ f ≤c = {θ ∈ Θ | f (θ) ≤ c} are compact. One filtered complex naturally associated with function f and such that the subcomplexes F s C * compute the homology of sublevel sets Θ f ≤s is the gradient (Morse) complex, see e.g.; and references therein. Without loss of generality the function f can be assumed smooth here, otherwise one can always replace f by its smoothing. By adding a small perturbation such as a regularization term we can also assume that critical points of f are non-degenerate. The generators of the gradient (Morse) complex correspond to the critical points of f. The differential is defined by counting gradient trajectories between critical points when their number is finite. The canonical form of the gradient (Morse) complex describes a decomposition of the gradient flow associated with f into standard simple pieces. Let p be a minimum, which is not a global minimum. Then the generator corresponding to p represents trivial homology class in the canonical form, since the homology class of its connected component is already represented by the global minimum. Then p is the lower generator of a two-dimensional complex with trivial homology in the canonical form. I.e. p is paired with an index-one saddle q in the canonical form. The segment [f (p), f (q)] is then the canonical invariant (barcode) corresponding to the minimum p. The full canonical form of the gradient (Morse) complex of all indexes is a summary of global structure of the objective function's gradient flow. The total number of different topological features in sublevel sets Θ f ≤c of the objective function can be read immediately from the barcode. Namely the number of intersections of horizontal line at level c with segments in the index j barcode gives the number of independent topological features of dimension j in Θ f ≤c. The description of the barcode of minima on manifold Θ with nonempty boundary ∂Θ is modified in the following way. A connected component can be also born at a local minimum of restriction of f to the boundary f | ∂Θ, if gradf is pointed inside manifold Θ. The merging of two connected components can also happen at an index-one critical point of f | ∂Θ, if gradf is pointed inside Θ. In this section we describe the developed algorithm for calculation of the canonical form invariants of local minima. The computation exploits the first definition of barcodes (see Section 2), which is based on the evolution on the connected components of the sublevel sets. To analyse the surface of the given function f: Θ → R, we first build its approximation by finite graph-based construction. To do this, we consider a random subset of points {θ 1, . . ., θ N} ∈ Θ and build a graph with these points as vertices. The edges connect close points. Thus, for every vertex θ n, by comparing f (θ n) with f (θ n) for neighbors θ n of θ n, we are able to understand the local topology near the point θ n. At the same time, connected componenets of sublevel sets Θ f ≤c = {θ ∈ Θ | f (θ) ≤ c} will naturally correspond to connected components of the subgraph on point θ n, such that f (θ n) ≤ c. Two technical details here are the choice of points θ n and the definition of closeness, i.e. when to connect points by an edge. In our experiments, we sample points uniformly from some rectangular box of interest. To add edges, we compute the oriented k-Nearest Neighbor Graph on the given points and then drop the orientation of edges. Thus, every node in the obtained graph has a degree in [k, 2k]. In all our experiments we use k = 2D, where D is the dimension of f's input. Next we describe our algorithm, which computes barcodes of a function from its graph-based approximation described above. The key idea is to monitor the evolution of the connected components of the sublevel sets of the graph, i.e. sets Θ c = {θ n | f (θ n) ≤ c} for increasing c. For simplicity we assume that points θ are ordered w.r.t. the value of function f, i.e. for n < n we have f (θ n) < f (θ n). In this case we are interested in the evolution of connected components throughout the process sequential adding of vertices θ 1, θ 2,..., θ N to graph, starting from an empty graph. We denote the subgraph on vertices θ 1,..., θ n by Θ n. When we add new vertex θ n+1 to θ n, there are three possibilities for connected componenets to evolve: 1. Vertex θ n+1 has zero degree in Θ n+1. This means that θ n+1 is a local minimum of f and it forms a new connected component in the sublevel set. 2. All the neighbors of θ n+1 in Θ n+1 belong to one connected component in Θ n. 3. All the neighbors of θ n+1 in Θ n+1 belong to ≥ 2 connected components s 1, s 2,..., s K ⊂ Θ n. Thus, all these components will form a single connected component in Θ n+1. Algorithm 1: Barcodes of minima computation for function on a graph. Input: Connected undirected graph G = (V, E); function f on graph vertices. Output: Barcodes: a list of "birth"-"death" pairs. In the third case, according to definition 1 of Section 2 the point θ n+1 is a 1-saddle point. Thus, one of the components s k swallows the rest. This is the component which has the lowest minimal value. For other components, 2 this gives their barcodes: for s k the birth-death pair is min We summarize the procedure in the following algorithm 1. Note that we assume that the input graph is connected (otherwise the algorithm can be run on separate connected components). In the practical implementation of the algorithm, we precompute the values of function f at all the vertices of G. Besides that, we use the disjoint set data structure to store and merge connected components during the process. We also keep and update the global minima in each component. We did not include these tricks into the algorithm's pseuso-code in order to keep it simple. The ing complexity of the algorithm is O(N log N) in the number of points. Here it is important to note that the procedure of graph creation may be itself time-consuming. In our case, the most time consuming operation is nearest neighbor search. In our code, we used efficient HNSW Algorithm for aproximate NN search by. In this section we apply our algorithm to describing the surfaces of functions. In Subsection 4.1 we apply the algorithm to toy visual examples. In Subsection 4.2 we apply the algorithm to analyse the loss surfaces of small neural networks. In this subsection we demonstrate the application of the algorithm to simple toy functions f: R D → R. For D ∈ {1, 2} we consider three following functions: 1. Polynomial of a single variable of degree 4 with 2 local minima (see Fig. 2a): 2. Camel function with 3 humps, i.e. 3 local minima (see Fig. 2b): 3. Camel function with 6 humps, i.e. 6 local minima (see Fig. 2c): Function plots with their corresponding barcodes of minima are given in Figure 2. The barcode of the global minimum is represented by the dashed half-line which goes to infinity. In this section we compute and analyse barcodes of small fully connected neural networks with up to three hidden layers. For several architectures of the neural networks many on the loss surface and its local minima are known (see e.g. and references therein). Different geometrical and topological properties of loss surfaces were studied in;;;. There is no ground truth on how should the best loss surface of a neural network looks like. Nevertheless, there exists many common opinions on this topic. First of all, from practical optimization point of view, the desired local (or global) minima should be easily reached via basic training methods such as Stochastic Gradient Descent, see. Usually this requires more-or-less stable slopes of the surface to prevent instabilities such as gradient explosions or vanishing gradients. Secondly, the value of obtained minimum is typically desired to be close to global, i.e. attain smallest training error. Thirdly, from the generalization point of view, such minima are required to provide small loss on the testing set. Although in general it is assumed that the good local optimum is the one that is flat, some recent development provide completely contrary arguments and examples, e.g. sharp minima that generalize well. Besides the optimization of the weights for a given architecture, neural network training implies also a choice of the architecture of the network, as well as the loss function to be used for training. In fact, it is the choice of the architecture and the loss function that determines the shape of the loss surface. Thus, proper selection of the network's architecture may simplify the loss surface and lead to potential improvements in the weight optimization procedure. We have analyzed very tiny neural networks. However our method permits full exploration of the loss surface as opposed to stochastical exploration of higher-dimensional loss surfaces. Let us emphasize that even from practical point of view it is important to understand first the behavior of barcodes in simplest examples where all hyper-parameters optimization schemes can be easily turned off. For every analysed neural network the objective function is its mean squared error for predicting (randomly selected) function g: [−π, π] → R given by plus l 2 −regularization. The error is computed for prediction on uniformly distributed inputs x ∈ {−π + 2π 100 k | k = 0, 1, . . ., 100}. The neural networks considered were fully connected one-hidden layer with 2, 3 and 4 neurons, two-hidden layers with 2x2, 3x2 and 3x3 neurons, and three hidden layers with 2x2x2 and 3x2x2 neurons. We have calculated the barcodes of the loss functions on the hyper-cubical sets Θ which were chosen based on the typical range of parameters of minima. The are as shown in Figure 3. We summarize our findings into two main observations: In this work we have introduced a methodology for analysing the plots of functions, in particular, loss surfaces of neural networks. The methodology is based on computing topological invariants called canonical forms or barcodes. To compute barcodes we used a graph-based construction which approximates the function plot. Then we apply the algorithm we developed to compute the barcodes of minima on the graph. Our experimental of computing barcodes for small neural networks lead to two principal observations. First all barcodes sit in a tiny lower part of the total function's range. Secondly, with increase of the depth of neural network the barcodes descend lower. From the practical point of view, this means that gradient descent optimization cannot stuck in high local minima, and it is also not difficult to get from one local minimum to another (with smaller value) during learning. The method we developed has several further research directions. Although we tested the method on small neural networks, it is possible to apply it to large-scale modern neural networks such as convolutional networks (i.e. ResNet, VGG, AlexNet, U-Net, see) for imageprocessing based tasks. However, in this case the graph-based approximation we use requires wise choice of representative graph vertices, which is a hardcore in high-dimensional spaces (dense filling of area by points is computationally intractable). Another direction is to study the connections between the barcode of local minima and the generalization properties of given minimum and of neural network. There are clearly also connections, deserving further investigation, between the barcodes of minima and concerning the rate of convergency during learning of neural networks.'

‘Knowledge Flow’ trains a deep net (student) by injecting information from multiple nets (teachers). The student is independent upon training and performs very well on learned tasks irrespective of the setting (reinforcement or supervised learning).

represent the short summary for finding the corresponding scientific article (long):

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.