Semi-Supervised Learning Enabled by Multiscale Deep Neural Network Inversion
Randall Balestriero Herve Glotin Richard Baraniuk







Semi-Supervised Learning Enabled by
Multiscale Deep Neural Network Inversion



    Randall Balestriero

ECE, Rice University USA

randallbalestriero@gmail.com





Hervé Glotin

LSIS, UTLN, France

Richard G. Baraniuk

ECE, Rice UNiversity, USA


    
=======================================================================================================================================================================


Deep Neural Networks (DNNs) provide state-of-the-art solutions in several difficult machine perceptual tasks. 
However, their performance relies on the availability of a large set of labeled training data, which limits the breadth of their applicability.
Hence, there is a need for new semi-supervised learning methods for DNNs that can leverage both (a small amount of) labeled and unlabeled training data.
In this paper, we develop a general loss function enabling DNNs of any topology to be trained in a semi-supervised manner without extra hyper-parameters.
As opposed to current semi-supervised techniques based on topology-specific or unstable approaches, ours is both robust and general.
We demonstrate that our approach reaches state-of-the-art performance on the SVHN (9.82% test error, with 500 labels and wide Resnet) and CIFAR10 (16.38% test error, with 8000 labels and sigmoid convolutional neural network) data sets.






§ INTRODUCTION


A deep neural network (DNNs) processes a signal x to produce an inference by composing L parametric mappings  f^(L)_θ^(ℓ),…, f^(1)_θ^(1) called layers, each with respective internal parameters θ^(ℓ). 
Each layer f^(ℓ)_θ^(ℓ) takes an input volume z^(ℓ-1)(x) to create an output volume z^(ℓ)(x),ℓ=1,…,L with z^(0)(x)=x



    z^(ℓ)(x)= (f^(ℓ)_θ^(ℓ)∘…∘ f^(1)_θ^(1))(x),   ℓ = 1,…,L.


Composing L layers thus generates a collection of volumes until the final output z^(L)(x) is reached.











For the case of classification with C classes, on which we focus in this paper, the final DNN output z^(L)(x) is transformed into a probability distribution via the softmax nonlinearity S:ℝ^C→ℝ^C to create the final prediction y(x)=S(z^(L)(x)), where

    ŷ(x)_c=e^z^(L)(x)_c/∑_c=1^Ce^z^(L)(x)_c,c=1,…,C.

The prediction corresponds to a class membership probability of x belonging to class c.

Given a (large) set of input/output pairs 
𝒟_s={(x_n,y_n)_n=1^N_s} the parameters  Θ={θ^(1),…θ^(L)} of the DNN are learned by comparing the prediction y(x_n) to the target y_n via a loss function such as cross-entropy ℒ_ CE <cit.>. 
The induced error is then minimized by updating the parameters Θ via first-order techniques such as gradient descent <cit.> leveraging backpropagation  <cit.>. 

While powerful, the application of fully supervised learning framework is limited by the often high cost of obtaining the required (very) large labeled labeled training dataset.


Consequently, there is growing interest in leveraging unlabeled data 𝒟_ u={(x_n,∅)_n=1^N_u}, which are often abundant.   
The focus of this paper is on fusing the information present in both 𝒟_s and 𝒟_ u to effect semi-supervised learning with DNNs.


Deriving an semi-supervised learning framework that is robust and most importantly architecture agnostic allowing the use of resnet, and with few hyper-parameters remains an open problem in deep learning.

In this paper, we tackle this challenge by introducing a new semi-supervised learning framework for DNNs.
The framework proposes first a renormalization of the original semi-supervised loss presented in <cit.> as well as a multiscale reconstruction loss that contributes stability during learning by reducing the impact of noisy or corrupted inputs. The term inversion is used loosely as the inversion problem in a nonlinear DNN is an ill posed problem in general. Thus in our case f^(-1) is used in place of the ill-defined inverse.
We summarize our major contributions as:



    
  * A thorough analysis of the loss function presented in <cit.> and removal of all hyper-parameters via a loss-dependent renormalization that obviates fine hyper-parameter cross-validation (see Section <ref>).
    
    
  * Introduction of a new multiscale loss for semi-supervised learning that is robust to initialization, the sampling of the labeled dataset 𝒟, and the presence of noise in the input (see Section <ref>).
    
    
  * A series of exhaustive experiments with the SVHN and CIFAR10 datasets and multiple DNN topologies that demonstrate that our approach achieves state-of-the-art results (see Section <ref>).
    






Related Work:

The problem of semi-supervised learning with DNN has been attempted by several groups.
The improved generative adversarial network (GAN) technique <cit.> couples two deep networks: a generative model creating new signal samples, and a discriminative model performing supervised learning. The discriminator simultaneously performs two tasks: discriminating between the true sample distribution and the generated one, and classifying the labeled samples from 𝒟_s.
Triple Generative Adversarial Nets <cit.> propose a extension of the GAN framework for the particular task of semi-supervised by introduction of a third player. The task thus becomes simpler as there exist one discriminator labeling images (fake or real) and another predicting if the couples (image,label) are fake or not. Through this, better stability is reached. Finally, Good Semi-supervised Learning That Requires a Bad GAN <cit.> currently hold SOTA method. This work lessen the same problem of GAN for semi-supervised of <cit.> by deriving analytical conditions and better formulation for the GAN objective hence providing a finer loss function as opposed to a third network.


The probabilistic formulation of deep convolutional networks presented in <cit.> supports semi-supervised learning. However, due to the need to have tractable probabilistic graphical model (PGMs), many simplifications led to this approach being applicable exclusively with Deep Convolutional Networks (DCN) topologies with Relu and max-pooling. Also, it requires the inputs x and inner representations z^(ℓ),ℓ=1,…,L to be non-negative, making most general tasks out of reach.
Temporal Ensembling for Semi-Supervised Learning <cit.> propose to constrain the representations of a same input stimuli to be identical in the latent space despite the presence of dropout noise. This search of stability in the representation is analogous to the one of a siamese network <cit.> but instead of presenting two different inputs, the same is used through two different models (induced by dropout). This technique provides an explicit loss for the unsupervised examples leading to the Π model just described and a more efficient method denoted as temporal ensembling.
Distributional Smoothing with Virtual Adversarial Training <cit.> proposes also a regularization term constraining the regularity of the DNN mapping for a given sample. Based on this a semi-supervised setting is derived by imposing for the unlabeled samples to maintain a stable DNN. Those two last described methods are the closest one of the proposed approach in this paper for which, the DNN stability will be replaced by a reconstruction ability, closely related to the DNN stability.

Classical approaches when considering the options for DNN inversion  was provided in <cit.> and in general relate to flavors of autoencoders <cit.>, such as the stacked convolutional autoencoder <cit.>.
As such, the semi-supervised with ladder network approach <cit.> can be seen as a particular autoencoder. It employs a per-layer reconstruction loss defined as ℒ_R^(ℓ)(x) = ||z^(ℓ)(x)-  d z^(ℓ+1)(x)/d z^(ℓ)(x)^T z^(ℓ+1)(x) ||^2,ℓ = L-1,…, 0. In the latter equation, d z^(ℓ+1)(x)/d z^(ℓ)(x) represents the derivative of the representation of the ℓ+1 layer w.r.t. the previous layer representation.
By forcing the inner layer to output an encoding describing the class distribution of the input via softmax nonlinearity, this deep unsupervised model is turned into a semi-supervised model. There remains a lack of a path to generalize this approach to other network topologies, such as recurrent or residual networks. Also, the per-layer ”greedy” reconstruction loss might be sub-optimal unless correctly weighted pushing the need for a precise and large cross-validation of hyper-parameters.



Other attempts based on back-propagation such as in <cit.> provides working solutions and efficient implementations, yet, did not leverage the approach for semi-supervised learning. In addition, generalization those layer specific technique to any architecture is not clear.
 However, recent work on DNN inversion <cit.> has developed a general approach applicable to any topology. In particular, they presented  semi-supervised state-of-the-art results on MNIST via the use of a Resnet topology. To do so, they introduce a generic way to invert a given DNN and  define a global reconstruction loss ℒ_R^(ℓ)(x) = ||x-  d z^(L)(x)/dx^Tz^(L)(x) ||^2, as well as an entropy loss ℒ_E(ŷ(x))=-∑_c=1^Cŷ(x)_c log(ŷ(x)_c) for the unlabeled examples. One can notice the difference from the ladder network by defining a global reconstruction loss as opposed to per-layer. 


For all the presented method, two main drawbacks arise. The first one is the presence of hyper-parameters to combine the different losses. The second, comes from the reconstruction loss. In the presence of noise, or corrupted inputs, the reconstruction objective will lead to noisy weights updates for all parameters θ^(ℓ),ℓ=1,…,L slowing convergence and hurting final performances. 




§ A UNIVERSAL & ROBUST SEMI-SUPERVISED LOSS


In order to overcome the input sensitivity of the reconstruction loss as well as the need for fine cross-validation by hyper-parameter removal. 
We first introduce notations and review the original scheme of <cit.>.
Afterwards, we will develop a simple loss-dependent renormalization that makes the loss's behavior invariant to the task and topology at hand. 
We will robustify our method by modifying the reconstruction loss leading to greater stability for real world datasets (as we demonstrate below in Sec. <ref>). 




 §.§ Multi-Objective Loss Renormalization


The work on semi-supervised learning for DNNs proposed in <cit.> leverages the inverse DNN formula defined as f^-1(x):=df(x)/dx^Tf(x). Based on this, a reconstruction loss has been defined as 

    ℒ_R(x) = x-  d z^(L)(x)/dx^Tz^(L)(x) ^2.

This loss acts as a data-driven network regularizer such that information of unlabeled samples is taken into account in the way DNNs model their input <cit.>. This is opposed to the standard structural regularization such as Tikhonov penalty <cit.>. Additionally, an entropy loss was defined for the unlabeled samples as 

    ℒ_E(ŷ(x))=-∑_c=1^Cŷ(x)_c log(ŷ(x)_c).

The presence of the entropy loss applied on the unlabeled data is natural. In fact, for supervised labels, the optimal output distribution is the one of minimum entropy (i.e., Dirac) constrained such that the position of this energy impulse is at the right index (class) position. For unsupervised examples, while this index position is unknown, the optimal remains a distribution of minimal entropy. Hence, ℒ_E acts as a guide, or attention model, on the internal parameters pushing unsupervised examples towards a known labeled cluster learned via the cross-entropy ℒ_CE and 𝒟_s. As a result, the final semi-supervised loss is a convex combination of the three losses defined as 


    ℒ(x,y)=   α 1_{y≠∅}ℒ_CE(x,y)+(1-α)β1_{y = ∅}ℒ_E(x)
       +(1-α)(1-β) ℒ_R(x).


The coefficients α,β∈ [0,1]^2 represent the weighting of the supervised versus unsupervised losses as well as regularization versus clustering. However, cross-validation of those parameters is cumbersome and heavy on computational power.
This, we renormalize each of the losses to ensure that their impact is equally distributed w.r.t. the overall loss. The following renormalized global loss function is proposed

    ℒ(x,y   )=1/log(C)(1_{y≠∅}ℒ_CE(x,y)_Supervised Cluster Labeling
       +1_{y = ∅}ℒ_E(x)_Unsupervised Clustering)+1/Dℒ_R(x)_Input Reconstruction,

with D the dimensionality of the input x.
The two losses ℒ_E and ℒ_CE are of same amplitude order. In both cases, we have at initialization ℒ_CE(x,y)≈log(C) and ℒ_E(x)≈log(C) as ŷ(x) ∼π(C) with π(C) Dirichlet distribution with uniform parameters. For the reconstruction loss ℒ_R, the range depends on the infinite norm of the considered input x. As we set in the experiments ||x||_∞=1, we ensure that this loss lies in the same range of values as the cross-entropy and entropy one. Also, due to the standard weight initialization of the layers, at initialization, reconstruction should not reach high amplitude values per pixel. This makes the three losses behaving with the same regime. We now propose further extension of this loss and specifically the reconstruction loss to provide stable and robust performances when dealing with real world datasets.






 §.§ Robust Semi-Supervised Learning via Multiscale  Reconstruction Loss


By introducing a finer reconstruction loss we aim to make performances robust to input noise and corruption as well as providing stable gradient updates. To do so we replace the global reconstruction loss based only on the input and its reconstruction by a convex combination of all the inner layers reconstructions. We do so for all the inner representations z^(ℓ),ℓ=0,…,L-1 including the input and excluding the final output. Let first define the per layer reconstruction loss as

    ℒ_R^(ℓ)(x) =   z^(ℓ)(x)-  d z^(L)(x)/dz^(ℓ)(x)^Tz^(L)(x) ^2,

for ℓ = L-1,…, 0. In order to provide renormalization of each of those local losses we first remind briefly standard notations. Each of the generated DNN volumes z^(ℓ),ℓ=1,…,L is of shape  (C^(ℓ),I^(ℓ),J^(ℓ)). We denote by D^(ℓ) the total size of the ℓ^th volume defined as D^(ℓ)=C^(ℓ)I^(ℓ)J^(ℓ).
Hence the local loss is defined as ℒ(x,y) by replacing the normalized reconstruction term 1/Dℒ_R(x) with introduced re-normalized per layer reconstruction as

    1/Dℒ_R(x)_Global→1/L∑_ℓ=0^L-11/D^(ℓ)ℒ^(ℓ)_R(x)_Local/Hierarchical

Doing so, we have the following property making this reconstruction loss robust and stable for general tasks. For clarity we will now denote by Γ and  λ the global and local reconstruction losses as

    Γ(x) =   1/Dℒ_R(x)
    λ(x) =   1/L∑_ℓ=0^L-11/D^(ℓ)ℒ^(ℓ)_R(x)



Given the local reconstruction loss λ, the impact of corrupted or noisy inputs is inversely proportional to the number of layers in the DNN.

This result is direct since we have 1/Lℒ_R^(0)(x) → 0 as L increases. 
We present in Fig. <ref> a depiction of the process with the blue arrow representing the forward pass, the red the reconstruction and this for all the layers including the input considered as layer 0.

Hence, the impact of incorrect input normalization or presence of noise will only induce noisy gradients for the updates of θ^(ℓ),ℓ=1,…,L via the erroneous term 1/Lℒ_R^(0)(x). This induced noisy gradient will then be overcome by the induced ones from the inner layers reconstruction loss. Doing so, a DNN will maintain inner layer stability even if this implies an incorrect input reconstruction. We observe this exact behavior and the explosion of inner layer regularity in the experiment section where we provide and analysis the evolution of the losses for the λ, and Γ settings.
The way we defined the per layer loss might seem arbitrary as opposed to the other possibility being ℒ^(ℓ)_R(x)=|| x-dz^(ℓ)/dxz^(ℓ) ||,ℓ=1,…,L. In this latter case, there is a per layer loss. Yet, in the presence of noise, corruption or simply class independent information in x,  this loss will provide noisy updates to all inner layers with the same impact disregarding of the number of layers. Hence, as it is the case for most application, with for example background, measurement noise and so on, our proposition is the one that should be chosen to ensure that these perturbations do not impact negatively the learning. Yet information of unlabeled examples are taken into account. Heuristically, it is observed that inner representation, by being the result of succession of mappings and nonlinearities will contain less and less class independent information. Hence, pushing reconstruction of inner representations as opposed to the input should be considered as the optimal strategy for real world application.




§ EXPERIMENTAL RESULTS


We first emphasize the need for cross validation encountered in the original framework. In order to reach state-of-the-art results on MNIST with 50 labels, as we report the results in Tab. <ref>, one should note the selected hyper-parameters (α,β). The best result was obtained by reducing the importance of the unsupervised losses likely due to the impacts of the reconstruction loss ℒ_R becoming detrimental for the behavior of the whole DNN training.





The optimal parameters being in favor of the supervised loss and further reducing the impact of ℒ_R by setting β=0.2 is indication of the need to have better behaving reconstruction loss.
This further motivates the need to adapt the loss in order to remove the need for semi-supervised specific cross-validation as well as prevent the unsupervised loss to overcome the natural learning of the DNN with the given labels. We now run experiments on the present framework with the λ versus Γ losses.



 §.§ Per Layer Reconstruction




In this section, we first describe the settings in which our experiments were performed. The Tab. <ref>,<ref> provide series of experiments on the two datasets SVHN and CIFAR, each time with two regime of labeled samples according to standard literature. We also provide evolution of the losses during training in Fig. <ref> and <ref>. Finally, image reconstruction is provided in order to qualitatively judge the abilities of the trained models to indeed reconstruct their input and provide further analysis between the λ versus Γ losses in Fig. <ref>.





To highlight the inter-dataset capacities of the model, we perform all experiments with different topologies but keeping identical the learning rate, batch size and input renormalization. 
We test 3 different network topologies: a large CNN made of 9 convolutional layers containing a total of  1M3 parameters; a wide Resnet with 2M1 parameters denoted as Resnet3-64 ; a deep resnet with 1M1 parameters denoted as Resnet6-32 with M stading for million.
For the CNN, we provide for each layer the tuple (number of filters, shape of filters, padding, pooling size), with a pooling size of 1 being synonym of no pooling performed. This leads, from the input layer to inner layer: (96,3,s,1), (96,3,f,1), (96,3,f,2), (192,3,v,1), (192,3,f,1), (192,3,v,2), (192,3,v,1), (192,1,s,6) where s stands for same, v for valid and f for full. Finally, a fully connected layer with 10 output neurons is used for the output prediction.
Note that this is a standard topology already used in <cit.> for semi-supervised learning.
For the Resnet blocks, we use a simplified version of <cit.>. 
The Resnet block is defined as f^(ℓ)(z^(ℓ-1))=Wz^(ℓ-1)+f_ conv(z^(ℓ-1)). The operator W is a linear convolution with filters of size (1,1). We follow standard procedure as in <cit.> for the number of filters which are always of spatial size (3,3) for the nonlinear convolutional layer and (1,1) for the linear one. The number of filters is multiplied by 2 after n blocks, and at the same time a down-sampling of the representation by a factor of (2,2) via mean pooling is performed. The total number of blocks is thus 3n. The initial number of filters is denoted by k, then a full topology is written as Resnet3-64 for n=3 and k=32. Note that we used mean-pooling in the convolutional layer to prevent artifact due to the max-pooling when performing reconstruction.

In all cases, dropout <cit.> is used after each nonlinearity with p=0.2 and batch norm <cit.> prior to nonlinearity taken as leaky-rectify <cit.>.
All inputs x_n are renormalized per observation by centering and reducing leading to x_n=x_n-x_n/max_d |x_n-x_n|. 
The batch size is taken as 50. 
Half of the batch is filled with labeled examples and the remaining with unlabeled ones draw randomly from 𝒟_u.
One epoch corresponds to having treated all unsupervised examples. As such, supervised examples 𝒟_s is augmented by replicating the labeled examples as many times as necessary to obtain Card(𝒟_s)= Card(𝒟_u). This is standard technique for non GAN based semi-supervised settings<cit.>.
Finally, the only hyper-parameter to cross-validate is the initial learning rate γ^(0). For this, we tried the following learning rates γ^(0)∈{0.02,0.002,0.0002} and chose the greatest one which did not lead to DNN divergence during learning. Hence we use  γ^(0)=0.002 for all models and all experiments, with adam optimizer <cit.>.
When e=150 and e=300 we perform a manual learning rate change by setting γ^(0) = γ^(0) /2 with e denoting the epoch number. We train for a total of 400 epochs.
Because we consider as one layer the succession of Convolution-Nonlinearity-Pooling for the CNN topology and a full block for the Resnet, our formula can be applied directly from Eqs. <ref>,<ref>.



By using the previously described training settings, we now present results on the two datasets SVHN and CIFAR10. For both we use benchmarks to compare our results with two regimes : 500 and 1000 labels for SVHN, and 4000 and 8000 labels for CIFAR10.
We also compare the two introduced renormalized loss we denote as Γ and λ, the latter being the one per layer.








We also provide the evolution of the training cross-entropy and entropy losses as well as reconstruction loss per layer and the test set accuracy evolution highlighting the fast convergence of the models in Fig. <ref>,<ref>.

We analyze the reconstruction of the best model for each dataset in Fig. <ref>. While the Γ loss provides DNNs with much better reconstruction capacities, the use of this ability for semi-supervised classification task is nonexistent. On the opposite as seen in Fig. <ref>, the inner layer's ability to reconstruct reduces dramatically. Yet, those inner representations are the ones of interest encoding the crucial information about the input, filtered after the cascade of projections and nonlinearities.
Thus the difference observed by changing from a global loss to a multiscale one, brings greater representation learning of the DNN and this regardless of the number of labeled samples. In fact, the reconstruction being applied for labeled and unlabeled examples, only the total number of samples will impact the observed benefits in accuracy.




We also present complementary experiment in order to highlight the ability of the proposed method to generalize not only between DNN architectures but also when changing nonlinearities. 
We already demonstrated the ability of the technique to deal with dropout and batch normalization as well as different tasks with no change in the framework. For this experiment we use the best model on the CIFAR 10 and SVHN tasks being respectively the CNN and wide resnet. Then we simply replace all nonlinearities originally being leaky rectifiers by sigmoids. We then apply the exact same experiment as before with no change whatsoever. We perform the learning by λ loss and compare with the original models for each dataset in Tab. <ref>,<ref> as well as provide for the CIFAR10 case evolution of the losses and test set accuracy in Fig. <ref>.

Such experiments have some importance: First, being able to generalize to non piecewise affine activation function allows the use of such a framework for DNN architectures requiring squashing functions such as recurrent networks <cit.>, LSTM <cit.> and GRUs <cit.>. Secondly, for more general task, one might consider to impose to specific behavior of the hidden layer representations such as saturation, upper bounded output and so on. Hence, nonconvex function might be of interest leading to the impossibility to use ReLU based activations.
For this specific experiment, one benefit of using a sigmoid function is the ability to bound the forward-backward pass. In fact, during backpropagation, the vanishing gradient property, usually considered as detrimental during learning, can here be considered as a self-regulatory behavior avoiding explosion of the reconstruction amplitudes through the layers.






 §.§ Going Further: Self-Learning and Data Augmentation


In this section we propose two additional experiments two demonstrate in practice how to obtain the best performances with two simple techniques. The first one is widely popular in supervised learning and is data augmentation. We hence simply aim at demonstrating that the benefits of this technique transport to the semi-supervised approach we propose. For this task, two tasks will be performed: labeled dataset augmentation, and labeled as well as unlabeled dataset augmentation. Through this we will be able to measure the impact of applying this method additionally on the unsupervised samples and thus the amount of information the DNN encodes from them.
We use horizontal flip as well as cropping and 24× 24 random crop. For test set we use 5 random crops and predict the majority vote.







The second proposed experiment consists of, during learning, pushing unlabeled examples into a supervised dataset by using the DNN prediction as ground truth label. This technique is also widely used in semi-supervised learning when using supervised algorithms. Basic method consists of training the supervised classifier on the supervised dataset and then using some policy to label unlabeled examples via this classifier and train it again. This process is repeated until all samples are labeled or a stopping criterion is reached. 

We demonstrate that this framework can be applied in out context as well. In particular, we propose to study two strategies. The first one is based only on the entropy error to select the best unlabeled examples to transfer. The second is based on reconstruction error as well.

Hence we have the new couples (x_n,Q(ŷ(x_n))) where Q stands for the quantisation process transforming the soft prediction into a hard prediction with a one-hot encoding of the class belonging predicton at the index of the argmax.
We stop this process at epoch 200, and let the network train for a total of 600 epochs. We add additional epochs as we remind that in our implementation, one epoch is defined when all unlabeled examples have been seen. Hence by reducing the unlabeled set size, the epochs represent smaller number of batches.
We perform this experiment on the two most restrictive cases, namely SVHN 500 and CIFAR 4000 with the topology that was giving the best results in the general experiments previous described.
We report the results in Tab. <ref>.














§ DISCUSSION AND FUTURE WORK


In this paper, we have developed a general, parameterless loss function for learning. We demonstrated that it enables DNNs of any topology to be trained in a semi-supervised manner and that it is robust, leading to state-of-the-art performances on various tasks across DNN topologies and data sets.
By providing a general framework dealing with no task or DNN specific pre-processing; as well as being computationally efficient, we hope to bring DNNs to semi-supervised applications.

There are many avenues for future work.
For instance, we can see that the introduced multiscale loss (λ) is able to outperform the global loss (Γ). Yet, further improvements seem to be reachable with the standard trade off between computational need and model abilities. While we chose to present an out-of-the-box approach reaching state-of-the-art performances, there still remains options if one aims at further performances.
To do so, one possibility remains in the introduction of hyper parameters (β_CE,β_E,(β_R^(ℓ))_ℓ=0^L-1) in order to find more robust weighting of the losses such that the input reconstruction does not penalizes learning. However, doing so would bring back the cumbersome task of cross-validation. Hence one solution would be to do so coupled with automatic hyper-parameters updates as was done for learning rate.
To do so, two approaches would be available. First, as is done in adam <cit.> and rmsprop  <cit.>, updates of the hyper-parameters based on their evolution through the updates, their statistics and behaviors could be used. Such as reducing the ones corresponding to a volatile loss or simply re-weighting the multiple losses to guarantees uniform speed of convergence among them. Secondly, a more explicit possibility would be to explicit optimize and update the hyper-parameters by line search or approximate line search with gradients as proposed in  <cit.>.
For example, this could take the form of
 updating the weighting while performing learning. Let index by t the value of the parameters Θ at batch t. Given the new hyper-parameterized loss

    ℒ(x_n,y_n   ;Θ)=β_CE(t) ℒ_CE(y_n,ŷ(x_n))1_{y_n ≠∅}
       +β_E(t) ℒ_E(x_n)1_{y_n = ∅} +∑_ℓ=0^L-1β_R^(ℓ)(t)ℒ_R^(ℓ)(x_n),

the updated weights are defined as

    Θ(t+1)=Θ(t) - γ g((x_n,y_n;Θ),Θ),

with typically g(ℒ(x_n,y_n;Θ),Θ)=d ℒ(x_n,y_n;Θ)/dΘ being a gradient descent update. One can thus adopt the following update strategy for the hyper-parameters as

    β^(ℓ)(t+1) = β^(ℓ)(t)-d ℒ(x_n,y_n;Θ(t+1))/dβ^(ℓ)(t),

and so for all hyper-parameters.
Finally, from another angle, studying the impact of batch size as was done for supervised learning <cit.> as well as the proportion of labeled versus unlabeled examples per batch is of crucial important to further provide robust yet adaptive learning of large scale networks.







ieee



