Abstract
Classifying a visual concept merely from its associated online textual source, such as a Wikipedia article, is an attractive research topic in zero-shot learning because it alleviates the burden of manually collecting semantic attributes. Recent work has pursued this approach by exploring various ways of connecting the visual and text domains. In this paper, we revisit this idea by going further to consider one important factor: the textual representation is usually too noisy for the zero-shot learning application. This observation motivates us to design a simple yet effective zero-shot learning method that is capable of suppressing noise in the text.
Abstract
Specifically, we propose an l 2,1 -norm based objective function which can simultaneously suppress the noisy signal in the text and learn a function to match the text document and visual features. We also develop an optimization algorithm to efficiently solve the resulting problem. By conducting experiments on two large datasets, we demonstrate that the proposed method significantly outperforms those competing methods which rely on online information sources but with no explicit noise suppression. Furthermore, we make an in-depth analysis of the proposed method and provide insight as to what kind of information in documents is useful for zero-shot learning.
Introduction
Unlike traditional object classification tasks in which the training and test categories are identical, zero-shot learning aims to recognize objects from classes not seen at the training stage. It is recognized as an effective way for large scale visual classification since it alleviates the burden of collecting sufficient training data for every possible class. The key component ensuring the success of zero-shot learning is to find an intermediate semantic representation to bridge the * The first two authors contributed to this work equally. Correspondence should be addressed to C. Shen. gap between seen and unseen classes. In a nutshell, with this semantic representation we can first learn its connection with image features and then transfer this connection to unseen classes. So once the semantic representation of an unseen class is given, one can easily classify the image through the learned connection.
Introduction
Attributes, which essentially represent the discriminative properties shared among both seen and unseen categories, have become the most popular semantic representation in zero-shot learning [7, 8, 24, 12, 26] . Although the recent use of attributes has led to exciting advances in zero-shot learning [10, 2, 27] , the creation of attributes still relies on much human labour. This is inevitably discouraging since the motivation for zero-shot learning is to free large-scale recognition tasks from cumbersome annotation requirements.
Introduction
To remedy this drawback and move towards the goal of fully automatic zero-shot learning, several recent works [23, 9, 16] have explored the possibility of using the easily accessed online information sources to create the intermediate semantic representation. One possible choice is to directly use online textual documents, e.g., those found in Wikipedia, to build such a representation [6, 3] . This is promising because online text documents can be easily obtained and contain rich information about the object. To conduct zero-shot learning with textual documents, existing works [2, 10] develop various ways to measure the similarity between text and visual features. Our work is also based on this idea. We take a step further, however, to consider one additional important factor: the document representation is much more noisy than the human specified semantic representation and negligence of this fact would inevitably lead to inferior performance. For example, when the bagof-words model is adopted as the document representation, the occurrence of every word in a document will trigger a signal in one dimension of the document representation. However, it is clear that most words in a document are not directly relevant for identifying the object category. Thus it is necessary to design a noise suppression mechanism to down weight the importance of those less relevant words for zero-shot learning. This mechanism is closely related to feature selection. However, it is not exactly the same. As will be discussed in the following sections, the solution of our method does not discard the less relevant dimensions of the document representation but only suppress their impact for zero-shot learning.
Introduction
To this end, we propose a zero-shot learning method which particularly caters for the need for noise suppression. More specifically, we proposed a simple yet effective l 2,1norm based objective function which simultaneously suppresses the noisy signal within text descriptions and learns a function to match the visual and text domains. Furthermore, we develop an efficient optimization algorithm to solve this problem. By conducting experiments on two large scale zero-shot learning evaluation benchmarks, we demonstrate the benefit of the proposed noise suppression mechanism as well as its superior performance over other zero-shot learning methods which also rely on online information sources. In addition, we also conduct an in-depth analysis of the proposed method which provides an insight as to what kinds of information within a document are useful for zero-shot learning.
Related work
Most zero-shot learning approaches rely on human specified attributes. As one of the earliest attempts in zero-shot learning, Lampert et al. [12] adopted a set of attributes obtained from a psychology study. By learning probabilistic predictors of those attributes, they developed a framework to estimate the posterior of the test class. Later, a number of works has been proposed to improve the way of learning the connection between attributes and object categories. For example, the work in [11] addresses unreliability of attributes by exploring the idea of random forest. The work in [1] converted the zero-shot learning into a cross-domain matching problem and they proposed to learn a matching function to compare the attribute and the image feature. Built upon this idea, Romera-Paredes and Torr [20] proposed a simpler but more effective objective function to learn the matching function. Zhang and Saligrama [27] advocated the benefits of using attribute-attribute relationships, termed semantic similarity, as the intermediate semantic representation and they learn a function to match the image features with the semantic similarity.
Related work
To go beyond the human specified attributes, recent works also explore the use of other forms of semantic representations which can be easily obtained [14, 2, 9, 10] . For example, the co-occurrence statistics of words has been explored in [14, 2] to capture the semantic relevance of two concepts. The distributed word representation, e.g., word2vec, has been utilized as a substitution of attributes [9] and more recently the word2vec representation has been shown to be complementary to the human specified attributes [10] .
Related work
The other information source for creating the semantic representation is the online textual document, such as Wikipedia articles. In an earlier work, Berg et al. [4] attempt to discover attribute representation from a noisy web source by ranking the visual-ness scores of attribute candidates. Rohrbach et al. [18, 19] mine semantic relatedness for attribute-class association from different internet sources. More recent works [6, 3] directly learn a function to measure the compatibility between documents and visual features. However, compared with the state-of-the-art zeroshot learning methods, their performance seems to be disappointing even though some advanced technologies, such as deep learning, has been applied [3] .
Our approach

Overview
The overview of our method is depicted in Figure 1 . It starts with a raw document representation which is simply a binarized histogram of words. This representation is fed into our zero-shot learning algorithm to generate a classifier to detect relevant images. In the process of generating this classifier, the noise suppression regularizer in our method will automatically suppress the impact of less relevant words (illustrated as the red words in Figure 1 ).
Text representation
We extract our text representation based on a simple bagof-words model. We start by a preprocessing step of tokenizing the words and removing stop words and punctuations. Then a histogram of the remaining word occurrences is calculated and is subsequently binarized as the text representation. In other words, once a word appears in a document, its corresponding dimension within the text representation is set to "1". One more commonly used choice for the text representation is based on TF-IDF as in [6, 3] . However, we find that it produces worse performance 1 than directly using the binarized representation. This is probably because the weighting calculated of TF-IDF is not suitable for our zero-shot learning although it is considered to be less noisy for applications like document classification. In the binarized histogram we essentially treat each word in a document equally and this inevitably introduces a lot of noisy signals. However, thanks to our noise suppressing zeroshot learning algorithm, we can substantially down-weight the less relevant words and achieve good performance even with a noisy document representation.
Learning to match text and visual features
We first formally define our problem and introduce the notation used in the following sections. At the training Figure 1 . Overview of our zero-shot learning approach. The text representations are processed by the noise suppression mechanism to generate a classifier to detect relevant images and the noisy components of text representations are suppressed to gain better performance. stage, both image features and document descriptions for C seen categories are available. Let X ∈ R d×N denote the image features of N training examples and Z ∈ {0, 1}d ×C the aforementioned document representations for C seen classes, whered and d are the dimensionality of the document representation and the image features respectively. We also define Y ∈ {0, 1} N ×C as the indicator matrix for the C seen classes. Each row of Y has a unique "1" indicating its corresponding class label. At the test stage, the document representations of theĈ unseen classes are given and our task is to assignĈ unseen class labels to the test images.
Formulation
Our method is inspired by a recently proposed zero-shot learning approach [20] which has demonstrated impressive performance despite a very simple learning process. More specifically, it learns a matrix V which optimizes the following objective function.
Formulation
where S denotes the semantic attribute matrix and it can be either a binary matrix or a real value matrix. The scalars γ and λ are weights controlling the prominence of the various terms. The underlying idea of this algorithm can be understood as follows. If the task is to classify X into C categories, we can simply learn a linear classifier by fitting to Y, that is, min W X W − Y 2 F . However, in this case W cannot be transferred to unseen classes. Thus we fur-ther impose that W = VS. In other words, the classifier of a class is generated from its attributes. With this requirement, the classifier of an unseen class can be easily obtained and utilized to predict the category of a test image. Similarly, we can also treat X V as the classifier operated on the attributes S. The above understanding naturally gives rise to the regularization terms λ VS 2 F and γ X V 2 F which play the same role of the Frobenius norm regularizer as commonly introduced in multi-class classification or regression.
Formulation
Since our document representation can also be seen as an attribute vector, the method in [20] can be readily applied to our problem by simply setting S = Z. However, this naive solution ignores an important fact that the document representation is much more noisy than the human specified attribute vectors. To handle this issue, we introduce a noise suppression mechanism into Eq. (1). More specifically, we first decompose V into two terms:
Formulation
where W x ∈ R m×d and W z ∈ R m×d . These two matrices will play different roles in our method. W z is used to suppress the noisy components of Z and transform Z into a m × C intermediate representation. W x is used to generate the image classifier from the noise-suppressed intermediate representation. Thus, two different regularization terms are imposed to suit these two different roles. The first term is the l 2,1 -norm of W z which achieves the noise suppression effect. The second term is the Frobenius norm of W x W z Z which is similar to the λ VS 2 F term in Eq. (1).
Formulation
The formulation of our method is expressed as follows:
Formulation
It is known that the l 2,1 -norm will encourage the column vectors of W z to have few large values, which means that the impact of noisy dimensions of Z will be substantially suppressed or even completely eliminated. In fact, if λ 2 becomes sufficient large, it achieves the effect of feature selection on the document representation. However, by cross-validating λ 1 and λ 2 , our method does not lead to an exactly sparse solution as it seems that the algorithm prefers to keep the majority of the dimensions in Z for zero-shot learning. This is probably due to the joint regularization effect of W x W z Z 2 F or the fact that dimensions corresponding to lower values of w i z 2 are still useful for zero-shot learning. Therefore we consider the use of the l 2,1 -norm here as a noise suppression mechanism rather than a feature selection mechanism. We drop out the other regularization terms in Eq. (1) since we find them have little impact on performance.
Formulation
Similar to [20] , once V, in our case V = W x W z , is learned, we can infer the class label of a test image x using the following rule:
Formulation
where z c is the document representation for the c-th candidate test class.
Optimization
Eq. (3) is convex for W x and W z individually but not convex for both of them. Therefore we can solve it using an alternating method, that is, we first fix W x and solve for W z ; then fix W z and solve for W x .
Optimization
(1) Fix W x and solve for W z :
Optimization
This sub-problem is a regression problem with l 2,1 -norm regularization. Nie et al. [15] proposes an iterative framework to efficiently solve it. It has been shown that the original problem is equivalent to sequentially solving the following problem until convergence
Optimization
where D t is a diagonal matrix whose i-th diagonal element is 1/(2 (w i z ) (t−1) 2 ) 2 at the t-th iteration, where Algorithm 1 Fix W x and solve W z Input: W x ; X of seen classes; Z of seen classes; λ 1 and λ 2 ; maximum number of iterations τ . Initialize D 0 as identity matrix I ∈ Rd ×d . for t = 1 · · · τ do · Solve Sylvester equation (6) for
Optimization
is the i-th column of the optimal W z solved at the (t − 1)-th iteration. The problem in Eq. (5) further reduces to a Sylvester equation of W z
Optimization
The Sylvester equation has a unique solution if and only if A and −B do not share any eigenvalues. Many stateof-the-art toolboxes are able to solve it efficiently. In our setting, since both A and B are positive definite, A has only positive eigenvalues and −B has only negative eigenvalues. Therefore Eq. (6) has a unique solution. In summary, the sub-problem of fixing W x to solve W z can be solved via the algorithm listed in Algorithm 1.
Optimization
(2) Fix W z and solve for W x : This sub-problem is a conventional least squares minimization problem which has the following closed-form solution
Optimization
By alternating between the above two matrices, the overall alternating optimization algorithm for Eq. (3) is listed in Algorithm 2.
Optimization
as the i-th diagonal element to avoid the case of zero columns, and the l 2,1 norm is therefore approximated by
Optimization
It has been proved in [15] that this approximation guarantees the convergence and the result approaches to that of l 2,1 -norm as σ → 0 .
Algorithm 2 Alternating algorithm for solving Eq. (3)
Input: X of seen classes; Z of seen classes; λ 1 and λ 2 ; maximum number of iterations τ . Initialize W 0
Algorithm 2 Alternating algorithm for solving Eq. (3)
x with Gaussian distribution.
Experiments
We divide our experiments into two parts. In the first part we evaluate the proposed method and compare it against both of the methods utilizing online textual sources and human-specified semantic attributes. In the second part we analyse in-depth the noise suppression effect of the proposed method and provide insight into what kind of information in a document is useful for zero-shot learning.
Experimental setting
Datasets: We test our approach on two widely used benchmarks for attribute learning and zero-shot learning: Animals with Attributes [12] (AwA) and Caltech-UCSD birds-200-2011 [25] (CUB-200-2011). AwA consists of 30,475 images of 50 mammals classes with 85 attributes including color, skin texture, body size, body part, affordance, food source, habitat, and behaviour. CUB-200-2011 contains 11,788 images of 200 categories of bird subspecies with 312 fine-grained attributes such as color/shape/texture of body parts. We follow the train/test split according to [12] and [25] , where 10 and 50 testing classes are treated as unseen for AwA and CUB-200-2011, respectively.
Experimental setting
Textual document sources: We extract the text representation according to scheme introduced in Section 3.2. The raw textual sources are collected from Wikipedia articles describing each of the categories. When constructing the vocabulary, we use the articles of seen classes only. The dimensionality of the text representation is 3506 for AwA and 6815 for CUB-200-2011, respectively.
Experimental setting
Image features: To make fair a comparison, two types of image features, the low-level features in [19] and the fully connected layer activations from the "imagenet-vggverydeep-19" [22] CNN are used in our experiments.
Experimental setting
Implementation details: The Sylvester equation in Eq. (6) is solved by a MATLAB built-in function, which takes only around 5 seconds on an Intel Core i7 duranscott@example.net. The number of rows of matrices W x and W z is equal to the number of seen classes. We choose the 29.12 ± 0.07 Table 2 . Zero-shot learning classification results of AwA, measured by mean accuracy. In [19] , the approach mines attributes names from WordNet and additionally mines class-attribute from online sources of Wikipedia, WordNet, Yahoo, and Flickr. All methods in this table use the same low-level features in [19] . 
Performance evaluation
We first compare our method against [3] and [19] . The former is most relevant to our work in the sense that it learns a mapping to match images and textual documents. The work in [19] is a comprehensive comparison study of various information sources for zero-shot learning. Besides these two method, we also treat S = Z in Eq. (1), and apply the ESZSL method in [20] to our zero-shot learning problem. To make a fair comparison, we use the same lowlevel features in [19] when comparing with it and then use the "imagenet-vgg-verydeep-19" to compare with [3] . The comparison results are given in Table 1 and Table 2 . As can be seen in Table 1 , the proposed method significantly outperforms the methods in [3] , although they have used a more complicated deep learning framework. Also, we find that our baseline ESZSL achieves good performance. However, it is still 5% inferior to our approach, which clearly demonstrates the advantage of the noise suppression mechanism introduced in this paper. The results in Table 2 further show that our method is superior over other approaches which rely on automatically mined information from the web. Again, our method achieves a significant improvement (more than 4%) over ESZSL.
Performance evaluation
We now compare our work with a few other state-of-theart approaches on zero-shot learning, even though some of them are not based on online information sources. The results are summarized in Table 3 . Results [18, 11, 14, 1] listed in the upper part of the table utilize hand-crafted features and not surprisingly their performance is much inferior to that of the proposed method. The lower part of Table 3 are methods with visual features extracted from a pretrained CNN and thus are more comparable to our method. In this setting, we find that our method is comparable to most of the state-of-the-art results on AwA and results better than ours are all obtained from the methods using cleaner human defined attributes. The work in [2] evaluates various semantic representations such as Word2Vec embedding, GloVe word co-occurrence from Wikipedia sources, taxonomy embedding inferred from WordNet Hierarchy, and pre-defined binary and real-valued attributes. Our approach outperforms all methods that use online text sources. This shows that although online text sources provide transferable semantic representations, their discriminative ability is affected by the inherent noise and our method is better at handling the noisy information source for zero-shot learning.
Performance evaluation
Similar results are observed on the CUB-200-2011 dataset. Our approach again outperforms the methods using online sources and those methods that beat ours are all based on human specified fine-grained attributes. Note that many of the bird categories in CUB-200-2011 have very subtle differences which may not be well captured in Wikipedia articles. However, better performance may be expected by using a higher quality text corpus, such as bird watching articles.
In-depth analysis of the proposed method
In this section we provide an in-depth analysis of the proposed method by examining its noise suppression mechanism and the words that are most discriminative in the view of our method.
Effectiveness of the noise suppression method
In our method, the l 2,1 -norm is expected to allow only a few dimensions of the document representation to have large values. The importance of each individual dimension of the document representation can therefore be measured by the l 2 -norm of each column of learned W Z (we call it the im-portance weight in the following). We visualize this measurement for each dimension of the document representation in the top two subfigures in Figure 2 . As can be seen, most of the importance weights are not exactly zero as one might expect given that the l 2,1 -norm is applied. In fact, there are only 702 zero columns (out of 3506) for AwA and 949 (out of 6815) for CUB-200-2011. As also mentioned in Section 3, this is probably because of the joint regularization effect of ||W x W z Z|| 2 F in Eq. (3) and/or because by cross-validation most dimensions are still identified as being useful although their weighting should be very low. The second postulate might be supported by the observation that poorer performance will be obtained if we manually remove the dimensions which have low importance weights.
Effectiveness of the noise suppression method
Although our formulation does not achieve the feature selection effect, it does only assign large importance weights to a small number of dimensions. To visually compare its effect, we replace the l 2,1 -norm and with the Frobenius norm and carry out our learning algorithm again. The resulting importance weights are shown in the two subfigures at the bottom of Figure 2 . As can be seen, large importance weights appear in more dimensions in this case. This observation verifies the noise suppression effect of the regularizer introduced in Eq. (3) and explains the superior performance of our method over other text-based zero-shot learning approaches.
Understanding the important dimensions of the document representation
Since each individual dimension of the textual document representation corresponds to an unique word, we can visualize the dimensions/words with large importance weights for better understanding our zero-shot learning algorithm. Table 4 lists at most 15 top scored words for 15 out of 40 seen classes in AwA and we could make several observations from it: (1) even though the document representations are extremely noisy, most of the top-ranked words are semantically meaningful to describe discriminative properties of a category (an animal in this case), such as body parts, habitat, behaviour, affordance, taxonomy, and environment.
Understanding the important dimensions of the document representation
In fact, we find many top weighted words are consistent with some of the human specified attributes in AwA. (2) Many top-ranked words are not explicitly "visualizable" but they imply visual information of a category. For example, the abstract concept "ruminant" implicitly tells that the creature with this property is "deer-like" or "cattle-like" and builds a visual connection between antelope and deer in Table 4 . This observation has also been made in the literature [12, 13, 17, 4, 21] . (3) Interestingly, we also notice that although some concepts are not commonly considered as attributes, they exhibit large importance weight as inferred by our algorithm. By taking a close examina-
Method/Dataset
AwA CUB Rohrbach et al. [18] 42.7 Jayaraman & Grauman [11] 43.01 Mensink et al. [14] 14.4 Akata et al. [1] 43.5 18.0 Lampert et al. [13] tion, we categorize these words into two types. The first (labelled green in Table 4 ) are some concepts that are more likely to co-occur with meaningful attributes. For example, the word "stomach" is only shared by antelope and deer in Table 4 , despite its existence in all mammals. This is probably because "stomach" is more likely to be co-occurred with Table 4 . Category-wisely top ranked words, sorted by average importance weights within each class. The blue words are generally considered as meaningful attributes of this class. The green words are concepts somewhat related to this class, but are less informative to define it. The red words are concepts that are not semantically related to the corresponding class.
Method/Dataset
"ruminant", a discriminative property of ruminant animals. Another type of words (labelled red in Table 4 ) are not sufficiently meaningful for human interpreter. For example, "belong" and "general" are assigned with high importance weight for all cetaceans (blue whale, dolphin, killer whale etc.) and rodents (mouse, rabbit, hamster etc.), respectively. We suspect the reason is due to the dataset bias of documents. For example, documents of similar categories may be edited by authors from the same background who prefer a certain word choice. In sum, we find most of the top ranked words carry weak information by their own, but it seems that using them collaboratively produces impressive discriminative power for zero-shot learning.
Conclusion
In this paper, we have introduced a noise suppression mechanism to text-based zero-shot learning. The proposed l 2,1 -norm based objective function generates classifiers that are robust against textual noise and achieve state-of-theart zero-shot learning performance. We have made several findings in the experiments. (1) The inherent noise within text sources has a significant impact on zero-shot learning performance. As all the text-methods without noise suppression are inferior to our approach, we speculate that noise in a component of the mid-level representation decreases its discriminative power. (2) Most noisy components are suppressed rather than completely eliminated by our mechanism. Some words, although unimportant individually, can produce meaningful discriminative power when put together. (3) We find three kinds of words in the de-noised representation that can provide useful information for zero-shot learning. The first kind are the attributelike words that explicitly describe the category. The second are words that are weakly related to the category. They usually occur with definitive words. The last kind of words is non-informative to humans, but shows certain distribution patterns among related categories.
Conclusion
Overall, this paper points out an important factor in textbased zero-shot learning that has been previously ignored. By dealing directly with the inevitable variations in human expression, and suppressing words that contain little or no value, the performance of text-based automatic zero-shot learning can be significantly improved.

{'paper_id': '13897053', 'title': 'Less is More: Zero-Shot Learning from Online Textual Documents with Noise Suppression', 's2_url': 'https://api.semanticscholar.org/CorpusID:13897053', 'source': 'S2ORC'}