Induction Networks for Few-Shot Text Classification
Ruiying Geng Binhua Li Yongbin Li Xiaodan Zhu Ping Jian Jian Sun



Dispersion of Mobile Robots: The Power of Randomness
    Anisur Rahaman Molla1Research supported in part by DST Inspire Faculty research grant DST/INSPIRE/04/2015/002801. ORCID ID: 0000-0002-1537-3462William K. Moses Jr.2Research supported in part by a grant of his postdoctoral fellowship hosts from the Israeli Ministry of Science. ORCID ID: 0000-0002-4533-7593

    December 30, 2023
=======================================================================================================================================================================================================================================================================================================================




Text classification tends to struggle when data is deficient or when it needs to adapt to unseen classes. In such challenging scenarios, recent studies have used meta-learning to simulate the few-shot task, in which new queries are compared to a small support set at the sample-wise level. However, this sample-wise comparison may be severely disturbed by the various expressions in the same class. Therefore, we should be able to learn a general representation of each class in the support set and then compare it to new queries. In this paper, we propose a novel Induction Network to learn such a generalized class-wise representation, by innovatively leveraging the dynamic routing algorithm in meta-learning. In this way, we find the model is able to induce and generalize better. We evaluate the proposed model on a well-studied sentiment classification dataset (English) and a real-world dialogue intent classification dataset (Chinese). Experiment results show that on both datasets, the proposed model significantly outperforms the existing state-of-the-art approaches, proving the effectiveness of class-wise generalization in few-shot text classification.




§ INTRODUCTION

Deep learning has achieved a great success in many fields such as computer vision, speech recognition and natural language processing <cit.>. 

However, supervised deep learning is notoriously greedy for large labeled datasets, which limits the generalizability of deep models to new classes due to annotation cost. 
Humans on the other hand are readily capable of rapidly learning new classes of concepts with few examples or stimuli. This notable gap provides a fertile ground for further research.



Few-shot learning is devoted to resolving the data deficiency problem by recognizing novel classes from very few labeled examples. The limitation of only one or very few examples challenges the standard fine-tuning method in deep learning. Early studies <cit.> applied data augmentation and regularization techniques to alleviate the overfitting problem caused by data sparseness, only to a limited extent. Instead, researchers have explored meta-learning <cit.> to leverage the distribution over similar tasks, inspired by human learning. Contemporary approaches to few-shot learning often decompose the training procedure into an auxiliary meta-learning phase, which includes many meta-tasks, following the principle that the testing and training conditions must match. They extract some transferable knowledge by switching the meta-task from mini-batch to mini-batch. As such, few-shot models can classify new classes with just a small labeled support set.

However, existing approaches for few-shot learning still confront many important problems, including the imposed strong priors <cit.>, complex gradient transfer between tasks <cit.>, and fine-tuning the target problem <cit.>. The approaches proposed by <cit.> and <cit.>, which combine non-parametric methods and metric learning, provide potential solutions to some of those problems. The non-parametric methods allow novel examples to be rapidly assimilated, without suffering from catastrophic overfitting. Such non-parametric models only need to learn the representation of the samples and the metric measure. 
However, instances in the same class are interlinked and have their uniform fraction and their specific fractions. In previous studies, the class-level representations are calculated by simply summing or averaging representations of samples in the support set. In doing so, essential information may be lost in the noise brought by various forms of samples in the same class. Note that few-shot learning algorithms do not fine-tune on the support set. When increasing size of the support set, the improvement brought by a bigger data size will also be diminished by more sample level noises.



Instead, we explore a better approach by performing induction at the class-wise level: ignoring irrelevant details and encapsulating general semantic information from samples with various linguistic forms in the same class. 
As a result, there is a need for a perspective architecture that can reconstruct hierarchical representations of support sets and dynamically induce sample representations to class representations. 

Recently, capsule network <cit.> has been proposed, which possesses the exciting potential to address the aforementioned issue. A capsule network uses “capsules" that perform dynamic routing to encode the intrinsic spatial relationship between parts and whole that constitutes viewpoint invariant knowledge.
Following a similar spirit, we can regard samples as parts and class as a whole.
We propose the Induction Networks, which aims to model the ability of learning generalized class-level representation from samples in a small support set, based on the dynamic routing process.

First, an Encoder Module generates representations for a query and support samples. 
Next, an Induction Module executes a dynamic routing procedure, in which the matrix transformation can be seen as a map from the sample space to the class space, and then the generation of the class representation is all depending on the routing-by-agreement procedure other than any parameters, which renders a robust induction ability to the proposed model to deal with unseen classes. 
By regarding the samples' representations as input capsules and the classes' as output capsules, we expect to recognize the semantics of classes that is invariant to sample-level noise. 

Finally, the interaction between queries and classes is modelled—their representations are compared by a Relation Module to determine if the query matches the class or not. Defining an episode-based meta-training strategy, the holistic model is meta-trained end-to-end with the generalizability and scalability to recognize unseen classes.

The specific contributions of our work are listed as follows:



  * We propose the Induction Networks for few-shot text classification. To deal with sample-wise diversity in the few-shot learning task, our model is the first, to the best of our knowledge, that explicitly models the ability to induce class-level representations from small support sets.

  * The proposed Induction Module combines the dynamic routing algorithm with typical meta-learning frameworks. The matrix transformation and routing procedure enable our model to generalize well to recognize unseen classes. 

  * Our method outperforms the current state-of-the-art models on two few-shot text classification datasets, including a well-studied sentiment classification benchmark and a real-world dialogue intent classification dataset. 






§ RELATED WORK





 §.§ Few-Shot Learning


The seminal work on few-shot learning dates back to the early 2000s <cit.>. The authors combined generative models with complex iterative inference strategies. More recently, many approaches have used a meta-learning <cit.> strategy in the sense that they extract some transferable knowledge from a set of auxiliary tasks, which then helps them to learn the target few-shot problem well without suffering from overfitting. In general, these approaches can be divided into two categories. 




  
Optimization-based Methods This type of approach aims to learn to optimize the model parameters given the gradients computed from the few-shot examples. <cit.> proposed the Meta Network, which learnt the meta-level knowledge across tasks and shifted its inductive biases via fast parameterization for rapid generalization. <cit.> introduced a generic meta-learning architecture called SNAIL which used a novel combination of temporal convolutions and soft attention.



  
Distance Metric Learning These approaches are different from the above approaches that entail some complexity when learning the target few-shot problem. The core idea in metric-based few-shot learning is similar to nearest neighbours and kernel density estimation. The predicted probability over a set of known labels is a weighted sum of labels of support set samples. <cit.> produced a weighted K-nearest neighbour classifier measured by the cosine distance, which was called Matching Networks. <cit.> proposed the Prototypical Networks which learnt a metric space where classification could be performed by computing squared Euclidean distances to prototype representations of each class. Different from fixed metric measures, the Relation Network learnt a deep distance metric to compare the query with given examples <cit.>.

Recently, some studies have been presented focusing specifically on few-shot text classification problems. <cit.> studied lifelong domain word embeddings via meta-learning. <cit.> argued that the optimal meta-model may vary across tasks, and they employed the multi-metric model by clustering the meta-tasks into several defined clusters. <cit.> developed a few-shot text classification model for multi-label text classification where there was a known structure over the label space. <cit.> proposed a open-world learning model to deal with the unseen classes in the product classification problem. We solve the few-shot learning problem from a different perspective and propose a dynamic routing induction method to encapsulate the abstract class representation from samples, achieving state-of-the-art performances on two datasets.



 §.§ Capsule Network

The Capsule Network was first proposed by <cit.>, which allowed the network to learn robustly the invariants in part-whole relationships. Lately, Capsule Network has been explored in the natural language processing field. <cit.> successfully applied Capsule Network to fully supervised text classification problem with large labeled datasets. Unlike their work, we study few-shot text classification. <cit.> reused the supervised model similar to that of <cit.> for intent classification, in which a capsule-based architecture is extended to compute similarity between the target intents and source intents. Unlike their work, we propose Induction Networks for few-shot learning, in which we propose to use capsules and dynamic routing to learn generalized class-level representation from samples based. The dynamic routing method makes our model generalize better in the few-shot text classification task. 











§ PROBLEM DEFINITION



 §.§ Few-Shot Classification

Few-shot classification <cit.> is a task in which a classifier must be adapted to accommodate new classes not seen in training, given only a few examples of each of these new classes. We have a large labeled training set with a set of classes C_train. However, after training, our ultimate goal is to produce classifiers on the testing set with a disjoint set of new classes C_test, for which only a small labeled support set will be available. If the support set contains K labeled examples for each of the C unique classes, the target few-shot problem is called a C-way K-shot problem. Usually, the K is too small to train a supervised classification model. Therefore, we aim to perform meta-learning on the training set, and extract transferable knowledge that will allow us to deliver better few-shot learning on the support set and thus classify the test set more accurately.



 §.§ Training Procedure

The training procedure has to be chosen carefully to match inference at test time. An effective way to exploit the training set is to decompose the training procedure into an auxiliary meta-learning phase and mimic the few-shot learning setting via episode-based training, as proposed in <cit.>. We construct an meta-episode to compute gradients and update our model in each training iteration. The meta-episode is formed by randomly selecting a subset of classes from the training set first, and then choosing a subset of examples within each selected class to act as the support set S and a subset of the remaining examples to serve as the query set Q. 
The meta-training procedure explicitly learns to learn from the given support set S to minimise a loss over the query set Q. We call this strategy as episode-based meta training, and the details are shown in Algorithm 1.

It is worth noting that there are exponentially many possible meta tasks to train the model on, making it hard to overfit. 







For example, if a dataset contains 159 training classes, this leads to ( [ 159;   5 ])=794,747,031 possible 5-way tasks.













§ THE MODELS

Our Induction Networks, depicted in Figure <ref> (the case of 3-way 2-shot model), consists of three modules: Encoder Module, Induction Module and Relation Module. In the rest of this section, we will show how these modules work in each meta-episode. 



 §.§ Encoder Module

This module is a bi-direction recurrent neural network with self-attention as shown in <cit.>. Given an input text x=(w_1,w_2,...,w_T), represented by a sequence of word embeddings. We use a bidirectional LSTM to process the text:

    h_t=  LSTM(w_t, h_t-1)


    h_t=  LSTM(w_t, h_t+1)

And we concatenate h_t with h_t to obtain a hidden state h_t. Let the hidden state size for each unidirectional LSTM be u.
For simplicity, we note all the T h_ts as H=(h_1, h_2,...,h_T). Our aim is to encode a variable length of text into a fixed size embedding. We achieve that by choosing a linear combination of the T LSTM hidden vectors in H. Computing the linear combination requires the self-attention mechanism, which takes the whole LSTM hidden states H as input, and outputs a vector of weights a:

    a=softmax(W_a2tanh(W_a1H^T))

here W_a1∈ R^d_a × 2u and W_a2∈ R^d_a are weight matrices and d_a is a hyperparameter. The final representation e of the text is the weighted sum of H:

    e = ∑_t=1^Ta_t· h_t




 §.§ Induction Module

This section introduces the proposed dynamic routing induction algorithm. We regard these vectors e obtained from the support set S by Eq <ref> as sample vectors e^s, and the vectors e from the query set Q as query vectors e^q. The most important step is to extract the representation for each class in the support set. The main purpose of the induction module is to design a non-linear mapping from sample vector e_ij^s to class vector c_i:


    {e_ij^s∈R^2u}_i = 1,...C,j = 1...K↦{c_i∈R^2u}_i = 1^C.


We apply the dynamic routing algorithm <cit.> in this module, in the situation where the number of the output capsule is one. 
In order to accept any-way any-shot inputs in our model, a weight-sharable transformation across all sample vectors in the support set is employed. All of the sample vectors in the support set share the same
transformation weights W_s∈R^2u × 2u and bias b_s, so that the model is flexible enough to handle the support set at any scale. Each sample prediction vector ê_ij^s is computed by:

    ê_ij^s = squash(W_se_ij^s + b_s)

where squash is a non-linear squashing function through the entire vector, which leaves the direction of the vector unchanged but decreases its magnitude. Given input vector x, squash is defined as:

    squash(x) = x^2/1 + x^2x/x

Eq <ref> encodes important invariant semantic relationships between lower level sample features and higher level class features <cit.>.
To ensure the class vector encapsulates the sample feature vectors of this class automatically, dynamic routing is applied iteratively. In each iteration, the process dynamically amends the connection strength and makes sure that the coupling coefficients d_i sum to 1 between class i and all support samples in this class by a “routing softmax":

    d_i = softmax ( b_i)

where b_i is the logits of coupling coefficients, and initialized by 0 in the first iteration. 
Given each sample prediction vector ê_ij^s, each class candidate vector ĉ_i is a weighted sum of all sample prediction vectors ê_ij^s in class i:

    ĉ_i =  ∑_j d_ij·ê_ij^s

then a non-linear “squashing" function is applied to ensure that the length of the vector output of the routing process will not exceed 1:

    c_i = squash(ĉ_i)


The last step in every iteration is to adjust the logits of coupling coefficients b_ij by a “routing by agreement" method. If the produced class candidate vector has a large scalar output with one sample prediction vector, there is a top-down feedback which increases the coupling coefficient for that sample and decreases it for other samples. This type of adjustment is very effective and robust for the few-shot learning scenario because it does not need to restore any parameters. Each b_ij is updated by:

    b_ij = b_ij + ê_ij^s·c_i


Formally, we call our induction method as dynamic routing induction and summarize it in Algorithm 2.




 §.§ Relation Module

After the class vector c_i is generated by the Induction Module and each query text in the query set is encoded to a query vector e^q by the Encoder Module, the next essential procedure is to measure the correlation between each pair of query and class.
The output of the Relation Module is called the relation score, representing the correlation between c_i and e^q, which is a scalar between 0 and 1. Specifically, we use the neural tensor layer <cit.> in this module, which has shown great advantages in modeling the relationship between two vectors <cit.>. We choose it as an interaction function in this paper. The tensor layer outputs a relation vector as follows:

    v(c_i,e^q) = f( c_i^TM^[1:h]e^q)

where M^k∈R^2u × 2u,k ∈[ 1,...,h] is one slice of the tensor parameters and f is a non-linear activation function called RELU <cit.>. The final relation score r_iq between the i-th class and the q-th query is calculated by a fully connected layer activated by a sigmoid function.

    r_iq=sigmoid(W_rv(c_i,e^q)+b_r)




 §.§ Objective Function

We use the mean square error (MSE) loss to train our model, regressing the relation score r_iq to the ground truth y_q: matched pairs have similarity 1 and the mismatched pair have similarity 0. Given the support set S with C classes and query set Q={( x_q,y_q)}_q = 1^n in an episode, the loss function is defined as:

    L( S, Q) = ∑_i=1^C ∑_q=1^n (r_iq - 1(y_q==i))^2

conceptually we are predicting relation scores, which can be considered as a regression problem and the ground truth is within the space {. 0,1}..

All parameters of the three modules are trained jointly by backpropagation. The Adagrad <cit.> is used on all parameters in each training episode. 

Our model does not need any fine-tuning on the classes it has never seen due to its generalization nature. The induction and comparison ability are accumulated in the model along with the training episodes.



§ EXPERIMENTS

We evaluate our model by conducting experiments on two few-shot text classification datasets. All the experiments are implemented with Tensorflow.


 §.§ Datasets



  
Amazon Review Sentiment Classification (ARSC) Following <cit.>, we use the multiple tasks with the multi-domain sentiment classification <cit.> dataset. The dataset comprises English reviews for 23 types of products on Amazon. For each product domain, there are three different binary classification tasks. These buckets then form 23×3=69 tasks in total. Following <cit.>, we select 12(4×3) tasks from 4 domains (Books, DVD, Electronics and Kitchen) as the test set, and there are only five examples as support set for each label in the test set. We create 5-shot learning models on this dataset. 
  



  
Open Domain Intent Classification for Dialog System (ODIC) We create this dataset by fetching the log data on a real-world conversational platform. The enterprises submit various dialogue tasks with a great number of intents, but many intents have only a few labeled samples, which is a typical few-shot classification application. Following the definition of the few-shot learning task, we divide the ODIC into a training set and a testing set and ensure that the labels of the two sets have no intersection. The details of the set partition are shown in Table <ref>.


  





 §.§ Experiment Setup



  
Baselines In this section, the baseline models in our experiments are introduced as follows.




  * Matching Networks: a few-shot learning model using a metric-based attention method <cit.>. 

  * Prototypical Networks: a deep metric-based method using sample average as class prototypes <cit.>.

  * Graph Network: a graph-based few-shot learning model that implements a task-driven message passing algorithm on the sample-wise level <cit.>.

  * Relation Network: a few-shot learning model which uses a neural network as the distance metric and sums up sample vectors in the support set as class vectors <cit.>.

  * SNAIL: a class of simple and generic meta-learner architectures that use a novel combination of temporal convolutions and soft attention <cit.>.

  * ROBUSTTC-FSL: This approach combines several metric-based methods by clustering the tasks <cit.>.



The baseline results on ARSC are reported in <cit.> and we implemented the baseline models on ODIC with the same text encoder module.




  
Implementation Details We use 300-dimension Glove embeddings <cit.> for ARSC dataset and 300-dimension Chinese word embeddings trained by <cit.> for ODIC. We set the hidden state size of LSTM u=128 and the attention dimension d_a=64. The iteration number iter used in dynamic routing algorithm is 3. The relation module is a neural tensor layer with h=100 followed by a fully connected layer activated by sigmoid. We build 2-way 5-shot models on ARSC following <cit.>, and build episode-based meta training with C=[5, 10] and K=[5, 10] for comparison on ODIC. In addition to K sample texts as support set, the query set has 20 query texts for each of the C sampled classes in every training episode. This means, for example, that there are 20×5 + 5 × 5 = 125 texts in one training episode for the 5-way 5-shot experiments.



  
Evaluation Methods We evaluate the performance by few-shot classification accuracy following previous studies in few-shot learning <cit.>. To evaluate the proposed model with the baselines objectively, we compute mean few-shot classification accuracies on ODIC over 600 randomly selected episodes from the testing set. We sample 10 test texts per class in each episode for evaluation in both 5-shot and 10-shot scenarios. Note that for ARSC, the support set for testing is fixed by <cit.>. Consequently, we just need to run the test episode once for each of the target tasks. The mean accuracy of the 12 target task is compared to the baseline models following <cit.>.



 §.§ Experiment Results




  
Overall Performance Experiment results on ARSC are presented in Table <ref>. The proposed Induction Networks achieves a 85.63% accuracy, outperforming the existing state-of-the-art model, ROBUSTTC-FSL, by a notable 3% improvement. We due the improvement to the fact that ROBUSTTC-FSL builds a general metric method by integrating several metrics at the sample level, which faces the difficulty of getting rid of the noise among different expressions in the same class. In addition to that, the task-clustering-based method used by ROBUSTTC-FSL must be found on the relevance matrix, which is inefficient when applied to real-world scenarios where the tasks change rapidly. Our Induction Networks, however, is trained in the meta-learning framework with more flexible generalization and its induction ability can hence be accumulated through different tasks.

We also evaluate our method with a real-world intent classification dataset ODIC. The experiment results are listed in Table <ref>. We can see that our proposed Induction Networks achieves best classification performances on all of the four experiments. In the distance metric learning models (Matching Networks, Prototypical Networks, Graph Network and Relation Network), all the learning occurs in representing features and measuring distances at the sample-wise level. Our work builds an induction module focusing on the class-wise level of representation, which we claim to be more robust to variation of samples in the support set. Our model also outperforms the latest optimization-based method—SNAIL. The difference between Induction Networks and SNAIL shown in Table <ref> is statistically significant under the paired at the 99% significance level. 
In addition, the performance difference between our model and other baselines in the 10-shot scenario is more significant than in the 5-shot scenario. This is because in the 10-shot scenario, for the baseline models the improvement brought by a bigger data size is also diminished by more sample level noises.






  
Ablation Study To analyze the effect of varying different components of the Induction Module and Relation Module, we further report the ablation experiments on the ARSC dataset as shown in Table <ref>. We can see that the best performance is achieved when we used 3 iterations, corresponding to the best result reported in Table <ref> (more rounds of iterations did not further improve the performance), and the table shows the effectiveness of the routing component.
We also changed the Induction Module with sum and self-attention and changed Relation Module with cosine distance. Changes in the performances validate the benefit of both the Relation Module and Induction Module. The Attention+Relation models the induction ability by self-attention mechanism, but the ability is limited by the learnt attention parameters. Conversely, the proposed dynamic routing induction method captures class-level information by automatically adjusting the coupling coefficients according to inputted support sets, which is more suitable for the few-shot learning task.























 §.§ Further Analysis



We further analyze the effect of transformation and visualize query text vectors to show the advantage of the Induction Networks.

  



  
Effect of Transformation

Figure <ref> shows the t-SNE <cit.> visualization of support sample vectors before and after matrix transformation under the 5-way 10-shot scenario. We randomly select a support set with 50 texts (10 texts per class) from the ODIC testing set, and obtain the sample vectors {e_ij^s}_i = 1,...5,j = 1...10 after the encoder module and the sample prediction vector {ê_ij^s}_i = 1,...5,j = 1...10 after transformation. We can see that the vectors after matrix transformation are more separable, demonstrating the effectiveness of matrix transformation to encode semantic relationships between lower-level sample features and higher-level class features.






  
Query Text Vector Visualization

We also find out that our induction module does not only work well in generating effective class-level features, but also helps the encoders to learn better text vectors, as it can give different weights to instances and features during backpropagation. 
Figure <ref> shows the t-SNE <cit.> visualization of text vectors from the same randomly selected five classes, learnt by the Relation Network and our Induction Networks. 
It is clear that the text vectors learnt by Induction Networks are better separated semantically than those of Relation Network.





































§ CONCLUSION

In this paper, we propose the Induction Networks, a novel neural model for few-shot text classification. We propose to induce the class-level representations from support sets to deal with sample-wise diversity in few-shot learning tasks. The Induction Module combines the dynamic routing algorithm with a meta-learning framework, and the routing mechanism makes our model more general to recognize unseen classes.
The experiment results show that the proposed model outperforms the existing state-of-the-art few-shot text classification models. We found that both the matrix transformation and routing procedure contribute consistently to the few-shot learning tasks.





§ ACKNOWLEDGMENTS

The authors would like to thank the organizers of EMNLP-IJCNLP2019 and the reviewers for their helpful suggestions. This research work is supported by the National Key Research and Development Program of China under Grant No. 2017YFB1002103, the National Natural Science Foundation of China under Grant No. 61751201, and the Research Foundation of Beijing Municipal Science & Technology Commission under Grant No. Z181100008918002.


acl_natbib

