A Semantic Similarity Supervised Autoencoder for Zero-Shot Learning

This Letter proposes a autoencoder model supervised by semantic similarity for zero-shot learning. With the help of semantic similarity vectors of seen and unseen classes and the classification branch, our experimental results on two datasets are 7.3% and 4% better than the stateof-the-art on conventional zero-shot learning in terms of the averaged top-1 accuracy. key words: zero-shot learning, autoencoder, image classification


Introduction
With a large number of labelled training data, supervised classification accuracy has rapidly increased by utilizing deep neural network in recent years. However, the downside of deep learning technology is that it can only classify classes appearing in training data. Therefore, the concept of Zero-Shot Learning (ZSL) was proposed in the scenario where test classes are not provided during the training stage. In order to transfer the knowledge learned from seen classes to unseen classes, we need auxiliary information, such as manually annotated attributes and word2vec learned from text, to link between seen classes and unseen ones in the semantic space.
According to how to establish the mapping function between the visual space and the semantic space, ZSL methods can be divided into four categories. In the first category, models learn a projection function from the visual feature space to the semantic space [1]. In order to alleviate the hubness problem in the semantic space, the second category chooses the reverse mapping direction [2]. Methods in the third category find some intermediate spaces for both visual feature vectors and semantic embeddings to be mapped to [3]. The last category is the combination of the first category and the second category, which learns mapping functions between the visual space and the semantic space simultaneously, represented by generative adversarial networks (GAN) and autoencoder based models [4]- [8]. We use an autoencoder as our main architecture. Thus our model belongs to the last category.
In this work, we present a semantic similarity supervised model to zero learning based on the autoencoder paradigm. Specifically, semantic similarities of all classes Manuscript  are calculated and serve as supervision to guide the classification branch in our autoencoder model. We decompose the relations of the semantic space to semantic similarities among classes. Our aim is to preserve these similarities in the visual feature space so as to appropriately inherit the structure of the semantic space. Unlike other methods just using indirect semantic relations between seen classes and unseen classes, our model can directly and indirectly utilize these semantic relations. The relations are directly used by calculating the semantic similarities between seen classes and unseen classes and indirectly used by obtaining the similarities between images features and projected semantic embeddings of all classes. Our contributions are: • We propose an autoencoder model with a classification branch which is trained by semantic similarity for ZSL. • Our model can directly and indirectly utilize semantic relations between seen classes and unseen classes. • Our experimental results on two ZSL datasets show a significant improvement.

Related Work
The fourth method mentioned in Sect. 1 is widely used in today's ZSL.

Proposed Method
First, the definition of ZSL is as follows. Given an visual feature set X = X tr ∪ X te , where X tr is the training visual feature set and X te is the testing visual feature set. In coventional ZSL setting, visual features in X te only come from unseen classes, while they may come from seen classes or unseen classes in generalized ZSL(GZSL) setting. Semantic embeddings for all classes is is the semantic embedding set of seen classes is for unseen classes, N s is the number of seen classes, N u is the number of unseen classes, and N is the number of all classes, which equals to N s +N u . The label set of X tr is Y tr and the label set of X te is Y te .
The architecture of our proposed model is shown in Fig. 1. It consists of an encoder F(a) and a decoder G(p), where a is a semantic embedding and p is a class prototype in visual space, which is equal to F(a).
Our learning objective has two parts. One is the reconstruction objective to ensure that a semantic embedding a i belonging to i-th class can be mapped to i-th class prototype p i and this class prototype is similar to a visual feature x i of i-th class. Specifically, where p i = F(a i ). In order to make sure that p i contains semantic information of its class, it should be able to be restored to its original semantic embedding a i . Specifically: whereã i = G(F(a i )). Our second objective needs p i to be unlike visual feature vectors of other classes. To boost the discriminative power of class prototypes, we add a classification branch to the autoencoder and use semantic similarity vectors to supervise its training. Let s i j be a semantic similarity measure between different class embeddings. In order to calculate the softmax probability stably, we subtract the Fig. 1 Architecture of our model. the mean value from semantic similarities. Specificially, where τ is the temperature, when τ → 0, s i j → 1/(N), which leads to a uniform distribution of s i j with same i. As τ → ∞, only s ii is 1 and other s i j is 0. Then, we define q i = (s i1 , s i2 , . . . , s iN ) as the semantic similarity vector of i-th class. We consider the softmax of dot products between x i and p j as the predicted category probability of x i . Finally, the cross-entropy loss between q i and the predicted probability is considered as our classificiation loss to trian the classification branch: To sum up, our model minimizes the following objective function during training: Here |B| refers to the size of a mini-batch B.
In the testing stage, given a test sample x i , we infer its class as follows:

Experimental Settings and Datasets
Our encoder and decoder are both implemented as 2 fully connected layers with 1800 hidden units. We use LeakyReLU as the nonlinear activation function. As for the optimization, we adopt Adam optimizer with a constant learning rate 0.0001 and our training mini-batch is 64.The temperature parameter is fine-tuned with a cross-validation procedure in the split of train and validation provided by [10]. Specifically, we have found that the proposed model works well when the temperature is set as 125.
We test our proposed model in two ZSL datasets, CUB-200-2011 [11] and FLO [12]. For both CUB and FLO, split  More information about datasets in terms of the number of seen classes, unseen classes, the number of training and testing images can be found in Table 1. |X tr | is the number of training images. X u te and X s te represent the number of testing images belonging to unseen classes and seen classes respectively.
We follow the evaluation protocol proposed by [10] to evaluate our model. For the conventional ZSL setting, the averaged top-1 accuracy for each unseen class is computed, denoted as T1. For the GZSL setting, the averaged accuracy of seen classes is denoted by s and the averaged accuracy of unseen classes is denoted by u, and their harmonic mean defined as H = 2su/(s+u).

Results and Analysis
We compare our model with four state-of-the-art (SOTA) methods, i.e., CLSWGAN [5], Cycle-CLSWGAN [6], CADA-VAE [7] and f-VAEGAN-D2 [8] in Table 2, because we use the same semantic embeddings as they do. We also conduct experiments for ablation analysis. First, we remove the classification branch to show the its importance to the performance of our model. Then, we set different temperatures to show the influence of semantic relations provided by semantic similarity vectors. Model without our classification branch is called AE in Table 2. Our whole model containing the classification branch is referred as SSAE in Table 2.
In Table 2, our model apparently establishes new SOTA results on both datasets in the ZSL setting. Our model achieves 77.6% on FLO, and 65.0% on CUB in terms of T1, which is 7.3% and 4% higher than previous SOTA, 70.3% and 61%. In the more challenging GZSL setting, our model gets better results than others on both datasets in terms of H as well. On FLO, the performance of our model is 73.8% in terms of H, which achieves significantly higher result than the previous best 61%. On CUB, our result 56.2% is 2.6% higher than SOTA result 53.6% in terms of H.
Our model outperforms other models in Table 2, because we use semantic similarity vectors to guide the training of our autoencoder and our model can utilize semantic embeddings of unseen classes during training. The CLSWGAN model uses a pretrained classification module to guide their generation module, which can not use semantic embeddings of unseen classes. The Cycle-CLSWGAN model is based on the CLSWGAN model, so it has CLSWGAN model's shortcoming as well. The f-VAEGAN-D2 model combines the benefits of the variational autoencoder and GAN, but their model does not contain a classification module to guide their generation model to generate different visual features. It can be seen that CADA-VAE model performs best in the terms of u, because their model can learn a encoding that retains the information contained in all modalities they used.
In ablation study, as we expected our AE model does not perform well, which is 38.7% and 35.8% in terms of T1 in the ZSL setting on FLO and CUB in Table 2. In the GZSL setting, its H is well below the average, confirming that it is important to add a classification branch to our model. In Fig. 2 we show the performance of our model under different temperaures. As shown in Fig. 2 both for CUB and FLO, our model has a significant edge when the temperature is small, for example, 25 and 50. This shows that with the increasing of the temperature, semantic similarity vectors provide our model with more information about semantic relations. Going towards the large tempearture, the performance keeps stable and reaches the maximum at 125 for both datasets. This is expected since semantic similarity vectors can provide more information about semantic relations than one-hot labels.

Conclusion
In this work, we focus on utilizing semantic similarity to assist the autoencoder to map from the semantic space to the visual space correctly. We add a classification branch to our main autoencoder model and calculate semantic similarity vectors to guide its training. Evaluations on different settings of the proposed model are carried out and results outperforming state-of-the-art methods are achieved. Our results show the importance of our classificaiton branch and semantic information provided by our semantic similarity vectors.