Instance Credibility Inference for Few-Shot Learning
Yikai Wang Chengming Xu Chen Liu Li Zhang Yanwei Fu
cs.CV cs.LG


plain

















papersize=,

ruled
algorithmtbploa

algorithm
lyxlist[1]
	
		#1
		 
		 
		 
	







6422 







 
 
Instance Credibility Inference for Few-Shot Learning
    Yikai Wang1,4    Chengming Xu1    Chen Liu1    Li Zhang2    Yanwei Fu1,3,4Corresponding author.

1School of Data Science, Fudan University

2Department of Engineering Science, University of Oxford

3MOE Frontiers Center for Brain Science, Fudan University

4Shanghai Key Lab of
Intelligent Information Processing, Fudan University


{yikaiwang19, cmxu18, chenliu18, yanweifu}@fudan.edu.cn,
lz@robots.ox.ac.uk


    December 30, 2023
==============================================================================================================================================================================================================================================================================================================================================================================================================================


Few-shot learning (FSL) aims to recognize new objects with extremely limited training data for each category. 
Previous efforts are made by either leveraging meta-learning paradigm or novel principles in data augmentation to alleviate this extremely data-scarce problem. 
In contrast, 
this paper presents a simple statistical approach, 
dubbed Instance Credibility Inference (ICI) to exploit the distribution support of unlabeled instances for few-shot learning. 
Specifically, 
we first train a linear classifier with the labeled few-shot examples and use it to infer the pseudo-labels for the unlabeled data. 
To measure the credibility of each pseudo-labeled instance, 
we then propose to solve another linear regression hypothesis by increasing the sparsity of the incidental parameters and rank the pseudo-labeled instances with their sparsity degree. 
We select the most trustworthy pseudo-labeled instances alongside the labeled examples to re-train the linear classifier. 
This process is iterated until all the unlabeled samples are included in the expanded training set, 
i.e. the pseudo-label is converged for unlabeled data pool.
Extensive experiments under two few-shot settings show that our simple approach can establish new state-of-the-arts on four widely used few-shot learning benchmark datasets including miniImageNet, tieredImageNet, CIFAR-FS, and CUB. 
Our code is available at: <https://github.com/Yikai-Wang/ICI-FSL>




§ INTRODUCTION



Learning from one or few examples is an important ability for humans.
For example, children have no problem forming the concept of “giraffe” by only taking a glance from a picture in a book, 
or hearing its description as looking like a deer with a long neck <cit.>.
In contrast, the most successful recognition systems <cit.> still highly rely on an avalanche of labeled training data.
This thus increases the burden in rare data collection (accident data in the autonomous driving scenario) and
expensive data annotation (disease data for medical diagnose), 
and more fundamentally limits their scalability to open-ended learning of the long tail categories in the real-world.

Motivated by these observations, there has been a recent resurgence of research interest in few-shot learning <cit.>.
It aims to recognize new objects with extremely limited training data for each category. 
Basically, a few-shot learning model has the chance to access the source/base dataset with many labeled training instances for model training and then is able to generalize to a disjoint but relevant target/novel 
dataset
with only scarce labeled data.
A simplest baseline to transfer learned knowledge to the novel set is fine-tuning <cit.>.
However, it would cause severely overfitting as one or a few instances are insufficient to model the data distributions of the novel classes.
Data augmentation and regularization techniques can alleviate overfitting in such a limited-data regime, but they do not solve it.
Several recent efforts are made in leveraging learning to learn, or meta-learning paradigm by simulating the few-shot scenario in the training process <cit.>. 
However, Chen  <cit.> empirically argue that such a learning paradigm often results in inferior performance compared to a simple baseline with a linear classifier coupled with a deep feature extractor.

Given such a limited-data regime (one or few labeled examples per category), one of the fundamental problems for few-shot learning is that one can hardly estimate the data distribution without introducing the inductive bias.
To address this problem, 
two types of strategy resort to model the data distribution of novel category beyond traditional inductive few-shot learning:
(i) semi-supervised few-shot learning (SSFSL) <cit.> supposes that we can utilize 


unlabeled data (about ten times more than labeled data) to help to learn the 

model;
furthermore,
(ii) transductive inference <cit.> for few-shot learning (TFSL) <cit.> assumes we can access to all the test data, rather than evaluate them one by one in the inference process. 
In other words, the few-shot learning model can utilize the data distributions of testing examples. 

Self-taught learning <cit.> is one of the most straightforward ways in leveraging the information of unlabeled data. Typically, a trained classifier infers the labels of unlabeled data, which are further taken to update the classifier. 
Nevertheless, the inferred pseudo-labels may not be always trustworthy; the wrongly labeled instances may jeopardize the performance of the classifier.
It is thus essential to investigate the labeling confidence of each unlabeled instance.

To this end, we present a simple statistical approach, dubbed Instance Credibility Inference (ICI) to exploit the distribution support of unlabeled instances for few-shot learning. 
Specifically, we first train a linear classifier (, logistic regression) with the labeled few-shot examples and use it to infer the pseudo-labels for the unlabeled data.
Our model aims to iteratively select the most trustworthy pseudo-labeled instances according to their credibility measured by the proposed ICI to augment the training set. 
The classifier thus can be progressively updated and further infer the unlabeled data.
We iterate this process until all the unlabeled samples are included in the expanded training set, the pseudo-label is converged for unlabeled data pool.
The schematic illustration is shown in Figure <ref>. 

Basically, we re-purpose the standard self-taught learning algorithm by our ICI algorithm.
How to select the pseudo-labeled data to exclude the wrong-predicted samples, , excluding the noise introduced by the self-taught learning strategy?
Our intuition is that the algorithm of sample selection can neither rely only on the label space (based on the probability of each class given by the classifier) nor the feature space (select samples most similar to training data). 
Instead, 
we introduce a linear regression hypothesis by regressing each instance (labeled and pseudo-labeled) from feature to label space and increase the sparsity of the incidental parameter <cit.> until it vanishes.
Thus we can rank pseudo-labeled instances with sparsity degree as their credibility.
We conduct extensive experiments on major few-shot learning datasets to validate the effectiveness of our proposed algorithm. 

The contributions of this work are as follows:
(i) 
We present a simple statistical approach, dubbed Instance Credibility Inference (ICI) to exploit the distribution support of unlabeled instances for few-shot learning. 
Specifically, our model iteratively selects the pseudo-labeled instances according to its credibility measured by the proposed ICI for classifier training.
(ii) 
We re-purpose the standard self-taught learning algorithm <cit.> by our proposed ICI. To measure the credibility of each pseudo-labeled instance, we solve another linear regression hypothesis by increasing the sparsity of the incidental parameter <cit.> and rank the sparsity degree as the credibility for each pseudo-labeled instance.
(iii)
Extensive experiments under two few-shot settings show that our simple approach can establish new state-of-the-arts on four widely used few-shot learning benchmark datasets including miniImageNet, tieredImageNet, CIFAR-FS, and CUB.



§ RELATED WORK

Semi-supervised learning. 
Semi-supervised learning (SSL) aims to improve the learning performance with limited labeled data by exploiting large amount of unlabeled data.
Conventional approaches focus on finding the low-density separator within both labeled and unlabeled data <cit.>,
and avoid to learn the “wrong” knowledge from the unlabeled data <cit.>.
Recently, 
semi-supervised learning with deep learning models use consistency regularization <cit.>,
moving average technique <cit.> and
adversarial perturbation regularization <cit.>
to train the model with large amount of unlabeled data.
The key difference between semi-supervised learning and few-shot learning with unlabeled data is that the unlabeled data is still limited in the latter.
To some extent, the low-density assumption widely utilized in SSL is hard to achieve in the few-shot scenario, making SSFSL a more difficult problem.


Self-taught learning <cit.>, also known as self-training <cit.>,
is a traditional semi-supervised strategy of utilizing unlabeled data to improve the performance of classifiers <cit.>. 
Typically, an initially trained classifier predicts class labels of unlabeled instances; the  unlabeled data with  pseudo-labels are further selected to update the classifier. <cit.>. 
Current algorithms based on self-taught learning includes training neural networks using labeled data and pseudo-labeled data jointly <cit.>, 
using mix-up between unlabeled data and labeled data to reduce the influence of noise <cit.>, 
using label propagation for pseudo-labeling based on a nearest-neighbor graph and measuring the credibility using entropy <cit.>,
and re-weighting the pseudo-labeled data based on the cluster assumption on the feature space <cit.>. 
Unfortunately, the predicted pseudo-labels may not be trustworthy.
Different and orthogonal to previous re-weighting or mix-up works,  we design a statistical algorithm in estimating the credibility of each instance assigned with its corresponding pseudo-label. Only the most confident instances are employed to update the classifier.

Few-shot learning.
Recent efforts on FSL are made towards the following aspects.
(1) Metric learning methods, putting emphasis on finding better distance metrics, 
include weighted nearest neighbor classifier (Matching Network <cit.>), 
finding prototype for each class (Prototypical Network <cit.>),
or learning specific metric for each task (TADAM <cit.>);
(2) Meta learning methods, such as Meta-Critic <cit.>, MAML <cit.>, Meta-SGD <cit.>, Reptile <cit.>, and LEO <cit.>,  optimize the models for the capacity of rapidly adapted to new tasks. 
(3) Data augmentation algorithms enlarge available data to alleviate the lack of data in the image level <cit.> or the feature level <cit.>. Additional, SNAIL <cit.> utilizes the sequence modeling to create a new framework.
The proposed statistical algorithm is orthogonal but potentially useful to improve these algorithms – it is always worth increasing the training set by utilizing the unlabeled data with confidently predicted labels.


Few-shot learning with unlabeled data.
Recently approaches  tackle few-shot learning problems by resorting to additional  unlabeled data. Specifically,  in semi-supervised few-shot learning settings, recent works <cit.> enables unlabeled data from the same categories to better handle the true distribution of each class. Furthermore, transductive settings have also been considered recently. For example,  LST <cit.> utilizes self-taught learning strategy in a meta-learning manner. 
Different from these methods, this paper presents a conceptually simple statistical approach derived from self-taught learning; our approach, empirically and significantly improves the performance of FSL on several benchmark datasets, by only using very simple classifiers, , logistic regression, or Support Vector Machine (SVM).


§ METHODOLOGY




 §.§ Problem formulation


We introduce the formulation of few-shot learning here. Assume a base category
set 𝒞_base, and a novel category set 𝒞_novel
with 𝒞_base⋂𝒞_novel=∅. 
Accordingly, the base and novel datasets are D_base={(𝐈_i,y_i),y_i∈𝒞_base},
and D_novel={(𝐈_i,y_i),y_i∈𝒞_novel},
respectively. 
In few-shot learning, the recognition models on D_base
should be generalized to the novel category C_novel with only one
or few training examples per class. 

For evaluation, we adopt the standard N-way-m-shot classification as <cit.> on D_novel. 
Specifically, in each episode, we randomly sample N classes L∼ C_novel;
and m and q labeled images per class are randomly sampled in L to construct the support set S and the query set Q, respectively.
Thus we have |S|=N× m and |Q|=N× q.
The classification accuracy is averaged on query sets Q of many
meta-testing episodes. In addition, we have unlabeled
data of novel categories U_novel={𝐈_u}. 



 §.§ Self-taught learning from unlabeled data


In general, labeled data for machine learning is often very difficult
and expensive to obtain, while the unlabeled data can be utilized
for improving the performance of supervised learning. Thus we recap
the self-taught learning formalism – one of the most classical semi-supervised
methods for few-shot learning <cit.>. Particularly,
assume f(·) is the feature extractor trained
on the base dataset D_base. One can train a supervised classifier
g(·) on the support set S, and pseudo-labeling unlabeled data, ŷ_i=g(f(𝐈_u))
with corresponding confidence p_i given by the classifier. The most confident unlabeled
instances will be further taken as additional data of corresponding
classes in the support set S. Thus we obtain the updated supervised
classifier ĝ(·). To this end, few-shot classifier
acquires additional training instances, and thus its performance can
be improved. 

However, it is problematic if directly utilizing self-taught learning
in one-shot cases. Particularly, the supervised classifier g(·)
is only trained by few instances. The unlabeled instances with high
confidence may  not be correctly categorized, and the classifier will
be updated by some wrong instances. Even worse, one can not assume
the unlabeled instances follows the same class labels or generative
distribution as the labeled data. Noisy instances or outliers may
also be utilized to update the classifiers. To this end, we propose
a systematical algorithm: Instance Credibility Inference (ICI) to reduce the noise.



 §.§ Instance credibility inference (ICI)

To measure the credibility of predicted labels over unlabeled data, we introduce a hypothesis of linear model by regressing each instance from feature to label spaces.  Particularly, given n  instances of N classes,
S={(𝐈_i,y_i,𝐱_i),y_i∈𝒞_novel}, where y_i is the ground truth when 𝐈_i come from the support set, or the pseudo-label when 𝐈_i come from the unlabeled set, we employ a simple linear regression model to “predict” the class label,




    y_i=𝐱_i^⊤β+γ_i+ϵ_i,


where β∈ℝ^d× N is the
coefficient matrix for classification; 𝐱_i∈ℝ^d×1
is the feature vector of instance i; y_i is N dimension
one-hot vector denoting the class label of instance i. 
Note that to facilitate the computations, we employ PCA <cit.> to reduce the dimension of extracted features f(𝐈_i) to d. ϵ_ij∼𝒩(0,σ^2)
is the Gaussian noise of zero mean and σ variance. 
Inspired by incidental parameters <cit.>, 
we introduce γ_i,j to amend the chance of instance i belonging to class  y_j.
Larger ‖γ_i,j‖, the higher difficulty in attributing instance i to  class  y_j.


Write Eq. <ref> in a matrix form for all instances, we are thus solving the problem of:

    (β̂,γ̂)=β,γmin‖ Y-Xβ-γ‖ _F^2+λ R(γ),

where ‖·‖ _F^2 denotes the Frobenius norm. Y=[y_i]∈ℝ^n× N and

X=[𝐱_i^⊤]∈ℝ^n× d
indicate label and feature input respectively. γ=[γ_i]∈ℝ^n× N
is the incidental matrix, with the penalty R(γ)=∑_i=1^n‖γ_i‖ _2.
λ is the coefficient of penalty. To solve Eq. <ref>,
we re-write the function as 

    L(β,γ)=‖ Y-Xβ-γ‖ _F^2+λ R(γ).

Let ∂ L/∂β=0, we have 




    β̂=(X^⊤X)^†X^⊤(Y-γ),


where (·)^† denotes the Moore-Penrose pseudo-inverse. 
Note that 
(1) we are interested in utilizing γ
to measure the credibility of each instance along its regularization path, rather than estimating
β̂, since the linear regression model is not good enough
for classification in general. 
(2) the β̂ also relies
on the estimation of γ. 
To this end, we take Eq. <ref>
into L(·) and solve the problem
as,

    γ∈ℝ^n× Nmin‖ Y-H(Y-γ)-γ‖ _F^2+λ R(γ),

where 

H=X(X^⊤X)^†X^⊤
is the hat matrix of
X.
We further define X̃=(I-H) and Ỹ=X̃Y.
Then the above equation can be simplified as

    γ∈ℝ^n× Nmin‖Ỹ-X̃γ‖ _F^2+λ R(γ),

which is a multi-response regression problem. 



We seek the best subset by checking the regularization path, 
which can be easily configured by 
a blockwise descent algorithm
implemented in Glmnet <cit.>. 
Specifically,
we have a theoretical value of λ_max=imax‖X̃_· i^⊤Ỹ‖ _2/n <cit.> to guarantee the solution of Eq. <ref> all 0.
Then we can get a list of λs from 0 to λ_max. 
We solve a specific Eq. <ref> with each λ,
and get the regularization path of γ along the way.
Particularly, we regard γ as a function of λ. 
When λ changes from 0 to ∞, the sparsity of γ is increased until all of its elements are forced to be vanished. 
Further, our penalty R(γ) encourages γ vanishes row by row, , instance by instance. 
Moreover, the penalty will tend to vanish the subset of X̃ with the lowest deviations, indicating less discrepancy between the prediction and the ground truth.
Hence we could rank the pseudo-labeled data by their λ value when the corresponding γ_i vanishes. 
As shown in one toy example of Figure <ref>, the γ value of the instance denoted by the red line vanishes first, and thus it is the most trustworthy sample by our algorithm. 




 §.§ Self-taught learning with ICI


The proposed ICI can thus be easily integrated to improve the self-taught learning algorithm. Particularly, the initialized classifier can predict the pseudo-labels of unlabeled instances; and we further employ the ICI algorithm to select the most confident subset of unlabeled instances, to update the classifier. The whole algorithm can be iteratively updated, as summarized in Algorithm <ref>. 





§ EXPERIMENTS



Datasets.
Our experiments are conducted on several widely few-shot learning benchmark datasets for general object recognition and fine-grained classification, including
miniImageNet <cit.>, tieredImageNet <cit.>, 
CIFAR-FS <cit.> and 
CUB <cit.>.
miniImageNet consists of 100 classes with 600 labeled instances in each category.
We follow the split proposed by <cit.>, using 64 classes as the base set to train the feature extractor, 16 classes as the validation set and 
report performance on the novel set which consists of 20 classes. 

tieredImageNet is a larger dataset compared with miniImageNet, and its categories are selected with hierarchical structure to split base and novel datasets semantically. 
We follow the split introduced in <cit.> with base set of 20 superclasses (351 classes), validation set of 6 superclasses (97 classes) and novel set of 8 superclasses (160 classes). 
Each class contains 1281 images on average. 
CUB is a fine-grained dataset of 200 bird categories with 11788 images in total. 
Following the previous setting in <cit.>, we use 100 classes as the base set, 50 for validation and 50 as the novel set. 
To make a fair comparison, we crop all images with the bounding box provided by  <cit.>.
CIFAR-FS is a dataset with lower-resolution images derived from CIFAR-100 <cit.> .
It contains 100 classes with 600 instances in each class. 
We follow the split given by <cit.>, using 64 classes to construct the base set, 16 for validation and 20 as the novel set.



Experimental setup.
Unless otherwise specified, we use the following settings and implementation  in the experiments for our approach to make a fair comparison. 
As in <cit.>, we use ResNet-12 <cit.> with 4 residual blocks as the feature extractor in our experiments. 
Each block consists of three 3×3 convolutional layers, each of which followed by a BatchNorm layer and a LeakyReLu(0.1) activation. In the end of each block, a 2×2 max-pooling layer is utilized to reduce the output size.
The number of filters in each block is 64, 128, 256 and 512 respectively.
Specifically, referring to <cit.>, we adopt the Dropout <cit.> in the first two block to vanish 10% of the output,
and adopt DropBlock <cit.> in the latter two blocks to vanish 10% of output in channel level.
Finally, an average-pooling layer is employed to produce the input feature embedding.

We select 90% images from each training class (, 64 categories for miniImageNet) to construct our training set for training the feature extractor and use the remaining 10% as the validation set to select the best model.
We use SGD with momentum as the optimizer to train the feature extractor from scratch.

Momentum factor and L_2 weight decay is set to 0.9 and 1e-4, respectively.
All inputs are resized to 84×84.
We set the initial learning rate of 0.1, decayed by 10 after every 30 epochs.
The total training epochs is 120 epochs. 
In all of our experiments, we normalize the feature with L_2 norm and reduce the feature dimension to d=5 using PCA <cit.>.
Our model and all baselines are evaluated over 600 episodes with 15 test samples from each class.


  


 §.§ Semi-supervised few-shot learning

Settings. 
In the inference process, the unlabeled data from the corresponding category pool is utilized to help FSL. 
In our experiments, we report following settings of SSFSL: 
(1) we use 15 unlabeled samples for each class, the same as TFSL, to compare our algorithm in SSFSL and TFSL settings; 
(2) we use 30 unlabeled samples in 1-shot task, and 50 unlabeled samples in 5-shot task, the same as current SSFSL approaches <cit.>; 
(3) we use 80 unlabeled samples, to show the effectiveness of ICI compared with FSL algorithms with a larger network and higher-resolution inputs.
We denote these as (15/15), (30/50) and (80/80) in Table <ref>. 
Note that CUB is a fine-grained dataset and does not have so sufficient samples in each class, so we simply choose 5 as support set, 15 as query set and other samples as unlabeled set (about 39 samples on average) in the 5-shot task in the latter two settings. 
For all settings, we select 5 samples for every class in each iteration. The process is finished when at most five instances for each class are excluded from the expanded support set. , select (10/10), (25/45), (75/75) unlabeled instances in total. 
Further, we utilize Logistic Regression (denoted as LR) and linear Support Vector Machine (denoted as SVM) to show the robustness of ICI against different linear classifiers. 

Competitors. 
We compare our algorithm with current approaches in SSFSL. TPN <cit.> uses labeled support set and unlabeled set to propagate label to one query sample each time. LST <cit.> also uses self-taught learning strategy to pseudo-label data and select confident ones, but they do this by a neural network trained in the meta-learning manner for many iterations. 
Other approaches include Masked Soft k-Means <cit.> and a combination of MTL with TPN and Masked Soft k-Means reported by LST.

Results. are shown in Table <ref> where denoted as Semi. in the first column. 
Analysis from the experimental results, we can find that:
(1) Compare SSFSL with TFSL with the same number of unlabeled data, we can see that our SSFSL results are only reduced by a little or even beat TFSL results, which indicates that the information we got from the unlabeled data are robust and we can indeed handle the true distribution with unlabeled data practically.
(2) The more unlabeled data we get, the better performance we have. Thus we can learn more knowledge with more unlabeled data almost consistently using a linear classifier (logistic regression). When lots of unlabeled data are accessible, ICI achieves state-of-the-art in all experiments even compared with competitors which use bigger network and higher-resolution inputs. 
(3) Compared with other SSFSL approaches, ICI also achieves varying degrees of improvements in almost all tasks and datasets. These results further indicate the robustness of our algorithm. Compared logistic regression with SVM, the robustness of ICI still holds.



 §.§ Transductive few-shot learning

Settings. 
In transductive few-shot learning setting, we have chance to access the query data in the inference stage. 
Thus the unlabeled set and the query dataset are the same. 
In our experiments, we select 5 instances for each class in each iteration and repeat our algorithm until all the expected query samples are included, , each class will be expanded by at most 15 images.
We also utilize both Logistic Regression and SVM as our classifier, respectively.

Competitors.
We compare ICI with current TFSL approaches. 
TPN <cit.> constructs a graph and uses label propagation to transfer label from support samples to query samples and learn their framework in a meta-learning way.
TEAM <cit.> utilizes class prototypes with a data-dependent metric to inference labels of query samples.

Results. are shown in Table <ref> where denoted as Tran. in the first column. Experiments cross four benchmark datasets indicate that:
(1) Compared with basic linear classifier, ICI enjoys consistently improvements, especially in the 1-shot setting where the labeled data is extremely limited and  such improvements are robust regardless of utilizing which linear classifiers.
Further, compared results between miniImageNet and tieredImageNet, we can find that the improvement margin is in the similar scale, indicating that the improvement of ICI does not rely on the semantic relationship between base set and novel set.
Hence the effectiveness and robustness of ICI is confirmed practically. 
(2) Compared with current TFSL approaches, ICI also achieves the state-of-the-art results.








































 §.§ Ablation study










Effectiveness of ICI. To show the effectiveness of ICI, we visualize the regularization path of γ in one episode of inference process in Figure <ref> where red lines are instances that are correct-predicted while black lines are wrong-predicted ones. 
It is obvious that that most of the correct-predicted instances lie in the lower-left part. Since ICI will select samples whose norm will vanish in a lower λ. 
We could get more correct-predicted instances than wrong-predicted instances in a high ratio. 


Compare to baselines. 
To further show the effectiveness of ICI, 
we compare ICI with other sample selection strategies under the self-taught learning pipeline. 
One simple strategy is randomly sampling the unlabeled data into the expanded support set in each iteration, denoted as ra. 
Another is selecting the data based on the confidence given by the classifier, denoted by co. 
In this strategy, 
the more confident the classifier is to one sample, 
the more trustworthy that sample is. 
The last one is replacing our algorithm of computing credibility by choosing the nearest-neighbor of each class in the feature space, denoted as nn. 
In this part, we have 15 unlabeled instances for each class and select 5 to re-train the classifier by different methods for Semi. and Tran. task on  miniImageNet. 
From Table <ref>,
we observe that ICI outperforms all the baselines in all settings.










Effectiveness of iterative manner.
Our intuition is the proposed ICI learns to generate a set of trustworthy unlabelled data for classifier training. 
Select

all the unlabelled data in one go
cannot take the distribution, or the credibility of the unlabeled data into account, and thus produce more noise labels to hurt the performance of the model.
The classifier thus be trained with its prediction, resulting in no improvements in TFSL setting.
We briefly validate this as ICI (15) in Figure <ref> whilst ICI obtained better accuracy with iterative selection manner. 
For example, select 6 images with two iterations (ICI(3)) is superior to select 8 images in one iteration (ICI(8)).










Robustness against initial classifier. 
What are the requirements for the initial linear classifier? Is it necessary to satisfy that the accuracy of the initial linear classifier is higher than 50% or even higher? 
The answer is no.
As long as the initial linear classifier can be trained, theoretically our method should work.  
It thus is a future open question of how the initial classifier affects. 
We briefly validate it in Table <ref>.
We run 600 episodes, with each episode training an initial classifier with different classification accuracy.
Table <ref>
shows that most classifiers can get improved by ICI regardless of the initial accuracy (even with accuracy of 30-40%).





















































































Influence of reduced dimension.  
In this part, we study the influence of reduced dimension d in our algorithm on 5-way 1-shot miniImageNet experiments.
The results with reduced dimension 2, 5, 10, 20, 50, and without dimensionality reduction , d=512, are shown in Table <ref>. 
Our algorithm achieves better performance when the reduced dimension is much smaller than the number of instances (, d≪ n), which is consistent with the theoretical property <cit.>. 
Moreover, we can observe that our model achieves the best accuracy 66.80% when d=5.
Practically, we adopt d=5 in our model.























Influence of dimension reduction algorithms. 
Furthermore, we study the robustness of ICI to different dimension reduction algorithms.
We compare
Isomap <cit.>,
principal components analysis <cit.> (PCA),
local tangent space alignment <cit.> (LTSA),
multi-dimensional scaling <cit.> (MDS),
locally linear embedding <cit.> (LLE) and
spectral embedding <cit.> (SE)
on 5-way 1-shot miniImageNet experiments.
From Table <ref> we can observe that
ICI is robust across most of the dimensionality reduction algorithms (from LTAS 64.61% to SE 67.7%) except MDS (59.99%).
We adopt PCA for dimension reduction in our method.






§ CONCLUSION


In this paper, we have proposed a simple method, called Instance Credibility Inference (ICI) to exploit the distribution support of unlabeled instances for few-shot learning. 
The proposed ICI effectively select the most trustworthy pseudo-labeled instances according to their credibility to augment the training set. 
In order to measure the credibility of each pseudo-labeled instance, 
we propose to solve a linear regression hypothesis by increasing the sparsity of the incidental parameters <cit.> and rank the pseudo-labeled instance with their sparsity degree. 
Extensive experiments show that our simple approach can establish new state-of-the-arts on four widely used few-shot learning benchmark datasets including miniImageNet, tieredImageNet, CIFAR-FS, and CUB.

Acknowledgement.
This work was supported in part by NSFC Projects (U1611461,61702108), 
Science and Technology Commission of Shanghai Municipality Projects (19511120700, 19ZR1471800), Shanghai Municipal Science and Technology Major Project (2018SHZDZX01), and  Shanghai Research and Innovation Functional Program (17DZ2260900).


ieee_fullname



