abstract: This paper presents an algorithm for tagging words whose part-of-speech properties are unknown. Unlike previous work, the algorithm categorizes *word tokens in context* instead of *word types*. The algorithm is evaluated on the Brown Corpus.
author: Hinrich Schütze
CSLI, Ventura Hall
Stanford, CA 94305-4115 , USA
email: `firstname.lastname@example.com`
URL: `ftp://csli.stanford.edu/pub/prosit/DisPosTag.ps`
title: Distributional Part-of-Speech Tagging

# Introduction

Since online text becomes available in ever increasing volumes and an ever increasing number of languages, there is a growing need for robust processing techniques that can analyze text without expensive and time-consuming adaptation to new domains and genres. This need motivates research on fully automatic text processing that may rely on general principles of linguistics and computation, but does not depend on knowledge about individual words.

In this paper, we describe an experiment on fully automatic derivation of the knowledge necessary for part-of-speech tagging. Part-of-speech tagging is of interest for a number of applications, for example access to text data bases , robust parsing , and general parsing . The goal is to find an unsupervised method for tagging that relies on general distributional properties of text, properties that are invariant across languages and sublanguages. While the proposed algorithm is not successful for all grammatical categories, it does show that fully automatic tagging is possible when demands on accuracy are modest.

The following sections discuss related work, describe the learning procedure and evaluate it on the Brown Corpus .

# Related Work

The simplest part-of-speech taggers are bigram or trigram models . They require a relatively large tagged training text. Transformation-based tagging as introduced by also requires a hand-tagged text for training. No pretagged text is necessary for Hidden Markov Models . Still, a lexicon is needed that specifies the possible parts of speech for every word. have shown that the effort necessary to construct the part-of-speech lexicon can be considerably reduced by combining learning procedures and a partial part-of-speech categorization elicited from an informant.

The present paper is concerned with tagging languages and sublanguages for which no a priori knowledge about grammatical categories is available, a situation that occurs often in practice .

Several researchers have worked on learning grammatical properties of words. trains a connectionist net to predict words, a process that generates internal representations that reflect grammatical category. try to infer grammatical category from bigram statistics. and use vector models in which words are clustered according to the similarity of their close neighbors in a corpus. present a probabilistic model for entropy maximization that also relies on the immediate neighbors of words in a corpus. applies factor analysis to collocations of two target words ("certain" and "right") with their immediate neighbors.

```latex
\begin{table*}[tbp]\small
\centering
\begin{tabular}{l|l|l}
{\bf word} & {\bf side} & {\bf nearest neighbors} \\\hline\hline
   onto &left &into  toward  away  off
together    against beside around down\\
  onto &right &reduce among regarding against
towards plus  toward using
unlike\\\hline
 seemed &left &appeared might would remained
had became could must   should\\
 seemed    &right &seem   seems   wanted   want
going    meant   tried   expect   likely
\end{tabular}
\caption{\label{nearest} Words with most similar left and right
neighbors for ``onto'' and ``seemed''.}
\end{table*}
```

What these approaches have in common is that they classify *words* instead of individual occurrences. Given the widespread part-of-speech ambiguity of words this is problematic.[^1] How should a word like "plant" be categorized if it has uses both as a verb and as a noun? How can a categorization be considered meaningful if the infinitive marker "to" is not distinguished from the homophonous preposition?

In a previous paper , we trained a neural network to disambiguate part-of-speech using context; however, no information about the word that is to be categorized was used. This scheme fails for cases like "The soldiers *rarely* come home." vs. "The soldiers *will* come home." where the context is identical and information about the lexical item in question ("rarely" vs. "will") is needed in combination with context for correct classification. In this paper, we will compare two tagging algorithms, one based on classifying word types, and one based on classifying words-plus-context.

# Tag induction

We start by constructing representations of the syntactic behavior of a word with respect to its left and right context. Our working hypothesis is that syntactic behavior is reflected in co-occurrence patterns. Therefore, we will measure the similarity between two words with respect to their syntactic behavior to, say, their left side by the degree to which they share the same neighbors on the left. If the counts of neighbors are assembled into a vector (with one dimension for each neighbor), the cosine can be employed to measure similarity. It will assign a value close to 1.0 if two words share many neighbors, and 0.0 if they share none. We refer to the vector of left neighbors of a word as its *left context vector*, and to the vector of right neighbors as its *right context vector*. The unreduced context vectors in the experiment described here have 250 entries, corresponding to the 250 most frequent words in the Brown corpus.

This basic idea of measuring distributional similarity in terms of shared neighbors must be modified because of the sparseness of the data. Consider two infrequent adjectives that happen to modify different nouns in the corpus. Their right similarity according to the cosine measure would be zero. This is clearly undesirable. But even with high-frequency words, the simple vector model can yield misleading similarity measurements. A case in point is "a" vs. "an". These two articles do not share any right neighbors since the former is only used before consonants and the latter only before vowels. Yet intuitively, they are similar with respect to their right syntactic context despite the lack of common right neighbors.

Our solution to these problems is the application of a singular value decomposition. We can represent the left vectors of all words in the corpus as a matrix $C$ with $n$ rows, one for each word whose left neighbors are to be represented, and $k$ columns, one for each of the possible neighbors. SVD can be used to approximate the row and column vectors of $C$ in a low-dimensional space. In more detail, SVD decomposes a matrix $C$, the matrix of left vectors in our case, into three matrices $T_{0}$, $S_{0}$, and $D_{0}$ such that: $$C= T_{0} S_{0} D_{0}'$$ $S_{0}$ is a diagonal k-by-k matrix that contains the singular values of $C$ in descending order. The *i*th singular value can be interpreted as indicating the strength of the *i*th principal component of $C$. $T_{0}$ and $D_{0}$ are orthonormal matrices that approximate the rows and columns of $C$, respectively. By restricting the matrices $T_{0}$, $S_{0}$, and $D_{0}$ to their first $m<k$ columns (= principal components) one obtains the matrices $T$, $S$, and $D$. Their product $\hat{C}$ is the best least square approximation of $C$ by a matrix of rank $m$: $\hat{C} = T S D'$. We chose $m=50$ (reduction to a 50-dimensional space) for the SVD's described in this paper.

SVD addresses the problems of generalization and sparseness because broad and stable generalizations are represented on dimensions with large values which will be retained in the dimensionality reduction. In contrast, dimensions corresponding to small singular values represent idiosyncrasies, like the phonological constraint on the usage of "an" vs. "a", and will be dropped. We also gain efficiency since we can manipulate smaller vectors, reduced to 50 dimensions. We used SVDPACK to compute the singular value decompositions described in this paper .

Table  shows the nearest neighbors of two words (ordered according to closeness to the head word) after the dimensionality reduction. Neighbors with highest similarity according to both left and right context are listed. One can see clear differences between the nearest neighbors in the two spaces. The right-context neighbors of "onto" contain verbs because both prepositions and verbs govern noun phrases to their right. The left-context neighborhood of "onto" reflects the fact that prepositional phrases are used in the same position as adverbs like "away" and "together", thus making their left context similar. For "seemed", left-context neighbors are words that have similar types of noun phrases in subject position (mainly auxiliaries). The right-context neighbors all take "to"-infinitives as complements. An adjective like "likely" is very similar to "seemed" in this respect although its left context is quite different from that of "seemed". Similarly, the generalization that prepositions and transitive verbs are very similar if not identical in the way they govern noun phrases would be lost if "left" and "right" properties of words were lumped together in one representation. These examples demonstrate the importance of representing generalizations about left and right context separately.

The left and right context vectors are the basis for four different tag induction experiments, which are described in detail below:

-   induction based on word type only

-   induction based on word type and context

-   induction based on word type and context, restricted to "natural" contexts

-   induction based on word type and context, using generalized left and right context vectors

## Induction based on word type only

The two context vectors of a word characterize the distribution of neighboring words to its left and right. The concatenation of left and right context vector can therefore serve as a representation of a word's distributional behavior . We formed such concatenated vectors for all 47,025 words (surface forms) in the Brown corpus. Here, we use the raw 250-dimensional context vectors and apply the SVD to the 47,025-by-500 matrix (47,025 words with two 250-dimensional context vectors each). We obtained 47,025 50-dimensional reduced vectors from the SVD and clustered them into 200 classes using the fast clustering algorithm Buckshot (group average agglomeration applied to a sample). This classification constitutes the baseline performance for distributional part-of-speech tagging. All occurrences of a word are assigned to one class. As pointed out above, such a procedure is problematic for ambiguous words.

## Induction based on word type and context

In order to exploit contextual information in the classification of a token, we simply use context vectors of the two words occurring next to the token. An occurrence of word $w$ is represented by a concatenation of four context vectors:

-   The right context vector of the preceding word.

-   The left context vector of $w$.

-   The right context vector of $w$.

-   The left context vector of the following word.

The motivation is that a word's syntactic role depends both on the syntactic properties of its neighbors and on its own potential for entering into syntactic relationships with these neighbors. The only properties of context that we consider are the right-context vector of the preceding word and the left-context vector of the following word because they seem to represent the contextual information most important for the categorization of $w$. For example, for the disambiguation of "work" in "her work seemed to be important", only the fact that "seemed" expects noun phrases to its left is important, the right context vector of "seemed" does not contribute to disambiguation. That only the immediate neighbors are crucial for categorization is clearly a simplification, but as the results presented below show it seems to work surprisingly well.

```latex
\begin{table*}[tbp]\small
\centering
\begin{tabular}{l|ll||l|ll}
tag & description & Penn Treebank tags &
 tag & description & Penn Treebank tags\\\hline

ADN & adnominal modifier & ADN$^{*}$ \$ &
POS & possessive marker & POS\\
CC & conjunction & CC &
PRP & pronoun & PRP\\
CD & cardinal & CD &
RB & adverbial & RB RP RBR RBS\\
DT &  determiner & DT PDT PRP\$ &
TO & infinitive marker & TO\\
IN & preposition & IN &
VB & infinitive & VB \\
ING & ``-ing'' forms & VBG &
VBD & inflected verb form & VBD VBZ VBP \\
MD & modal & MD &
VBN & predicative & VBN PRD$^{*}$ \\
N & nominal & NNP(S) NN(S) &
WDT & wh-word & WP(\$) WRB WDT
\end{tabular}
\caption{\label{tagset} Evaluation tag set.
Structural tags derived from parse trees are marked with $*$.
}
\end{table*}
```

Again, an SVD is applied to address the problems of sparseness and generalization. We randomly selected 20,000 word triplets from the corpus and formed concatenations of four context vectors as described above. The singular value decomposition of the resulting 20,000-by-1,000 matrix defines a mapping from the 1,000-dimensional space of concatenated context vectors to a 50-dimensional reduced space. Our tag set was then induced by clustering the reduced vectors of the 20,000 selected occurrences into 200 classes. Each of the 200 tags is defined by the centroid of the corresponding class (the sum of its members). Distributional tagging of an occurrence of a word $w$ proceeds then by retrieving the four relevant context vectors (right context vector of previous word, left context vector of following word, both context vectors of $w$) concatenating them to one 1000-component vector, mapping this vector to 50 dimensions, computing the correlations with the 200 cluster centroids and, finally, assigning the occurrence to the closest cluster. This procedure was applied to all tokens of the Brown corpus.

We will see below that this method of distributional tagging, although partially successful, fails for many tokens whose neighbors are punctuation marks. The context vectors of punctuation marks contribute little information about syntactic categorization since there are no grammatical dependencies between words and punctuation marks, in contrast to strong dependencies between neighboring words.

For this reason, a second induction on the basis of word type and context was performed, but only for those tokens with informative contexts. Tokens next to punctuation marks and tokens with rare words as neighbors were not included. Contexts with rare words (less than ten occurrences) were also excluded for similar reasons: If a word only occurs nine or fewer times its left and right context vectors capture little information for syntactic categorization. In the experiment, 20,000 natural contexts were randomly selected, processed by the SVD and clustered into 200 classes. The classification was then applied to all natural contexts of the Brown corpus.

## Generalized context vectors

The context vectors used so far only capture information about distributional interactions with the 250 most frequent words. Intuitively, it should be possible to gain accuracy in tag induction by using information from more words. One way to do this is to let the right context vector record which *classes of left context vectors* occur to the right of a word. The rationale is that words with similar left context characterize words to their right in a similar way. For example, "seemed" and "would" have similar left contexts, and they characterize the right contexts of "he" and "the firefighter" as potentially containing an inflected verb form. Rather than having separate entries in its right context vector for "seemed", "would", and "likes", a word like "he" can now be characterized by a generalized entry for "inflected verb form occurs frequently to my right".

This proposal was implemented by applying a singular value decomposition to the 47025-by-250 matrix of left context vectors and clustering the resulting context vectors into 250 classes. A generalized right context vector $v$ for word $w$ was then formed by counting how often words from these 250 classes occurred to the right of $w$. Entry $v_{i}$ counts the number of times that a word from class $i$ occurs to the right of $w$ in the corpus (as opposed to the number of times that the word with frequency rank $i$ occurs to the right of $w$). Generalized left context vectors were derived by an analogous procedure using word-based right context vectors. Note that the information about left and right is kept separate in this computation. This differs from previous approaches in which left and right context vectors of a word are always used in one concatenated vector. There are arguably fewer different types of right syntactic contexts than types of syntactic categories. For example, transitive verbs and prepositions belong to different syntactic categories, but their right contexts are virtually identical in that they require a noun phrase. This generalization could not be exploited if left and right context were not treated separately.

Another argument for the two-step derivation is that many words don't have any of the 250 most frequent words as their left or right neighbor. Hence, their vector would be zero in the word-based scheme. The class-based scheme makes it more likely that meaningful representations are formed for all words in the vocabulary.

The generalized context vectors were input to the tag induction procedure described above for word-based context vectors: 20,000 word triplets were selected from the corpus, encoded as 1,000-dimensional vectors (consisting of four generalized context vectors), decomposed by a singular value decomposition and clustered into 200 classes. The resulting classification was applied to all tokens in the Brown corpus.

# Results

The results of the four experiments were evaluated by forming 16 classes of tags from the Penn Treebank as shown in Table . Preliminary experiments showed that distributional methods distinguish adnominal and predicative uses of adjectives (e.g. "the black cat" vs. "the cat is black"). Therefore the tag "ADN" was introduced for uses of adjectives, nouns, and participles as adnominal modifiers. The tag "PRD" stands for predicative uses of adjectives. The Penn Treebank parses of the Brown corpus were used to determine whether a token functions as an adnominal modifier. Punctuation marks, special symbols, interjections, foreign words and tags with fewer than 100 instances were excluded from the evaluation.

```latex
\begin{table*}[tbp]\small
\centering
\begin{tabular}{l||r|r|r|r|r|r|l}
tag &frequency &\# classes &correct&incorrect &precision &recall &
F\\\hline\hline

ADN  &108586 &34 & 38282 & 19528 &0.66 & 0.35 & 0.46 \\
CC  &36808 &0 & 0 & 0 &0.00 & 0.00 & 0.00 \\
CD  &15085 &4 & 3376 & 1431 &0.70 & 0.22 & 0.34 \\
DT  &129626 &2 & 125540 & 31783 &0.80 & 0.97 & 0.87 \\
IN  &132079 &3 & 118726 & 75829 &0.61 & 0.90 & 0.73 \\
ING  &14753 &5 & 2111 & 1016 &0.68 & 0.14 & 0.24 \\
MD  &13498 &2 & 13383 & 13016 &0.51 & 0.99 & 0.67 \\
N  &231434 &98 & 193838 & 79652 &0.71 & 0.84 & 0.77 \\
POS  &5086 &1 & 4641 & 1213 &0.79 & 0.91 & 0.85 \\
PRP  &47686 &3 & 43839 & 21723 &0.67 & 0.92 & 0.77 \\
RB  &54525 &7 & 35364 & 56505 &0.38 & 0.65 & 0.48 \\
TO  &25196 &0 & 0 & 0 &0.00 & 0.00 & 0.00 \\
VB  &35342 &8 & 29138 & 17945 &0.62 & 0.82 & 0.71 \\
VBD  &80058 &12 & 36653 & 3855 &0.90 & 0.46 & 0.61 \\
VBN  &41146 &21 & 7773 & 8841 &0.47 & 0.19 & 0.27 \\
WDT  &14093 &0 & 0 & 0 &0.00 & 0.00 & 0.00 \\
\hline
avg. &&&&& 0.53 & 0.52 & 0.49

\end{tabular}
\caption{\label{wordtype} Precision and recall for induction based on
word type.}
\end{table*}
```

Tables  and present results for word type-based induction and induction based on word type and context. For each tag $t$, the table lists the frequency of $t$ in the corpus ("frequency")[^2], the number of induced tags $i_{0}, i_{1},
\ldots,i_{l}$, that were assigned to it ("\# classes"); the number of times an occurrence of $t$ was correctly labeled as belonging to one of $i_{0},i_{1},\ldots,i_{l}$ ("correct"); the number of times that a token of a different tag $t^{\prime}$ was miscategorized as being an instance of $i_{0},i_{1},\ldots,i_{l}$ ("incorrect"); and precision and recall of the categorization of $t$. Precision is the number of correct tokens divided by the sum of correct and incorrect tokens. Recall is the number of correct tokens divided by the total number of tokens of $t$ (in the first column). The last column gives van Rijsbergen's F measure which computes an aggregate score from precision and recall: $F = \frac{1}{\alpha \frac{1}{P} + (1-\alpha) \frac{1}{R}}$. We chose $\alpha = 0.5$ to give equal weight to precision and recall.

It is clear from the tables that incorporating context improves performance considerably. The F score increases for all tags except CD, with an average improvement of more than 0.20. The tag CD is probably better thought of as describing a word class. There is a wide range of heterogeneous syntactic functions of cardinals in particular contexts: quantificational and adnominal uses, bare NP's ("is one of"), dates and ages ("Jan 1", "gave his age as 25"), and enumerations. In this light, it is not surprising that the word-type method does better on cardinals.

```latex
\begin{table*}[tbp]\small
\centering
\begin{tabular}{l||r|r|r|r|r|r|l}
tag &frequency &\# classes &correct&incorrect &precision &recall &
F\\\hline\hline

ADN  &108532 &42 & 87128 & 24743 &0.78 & 0.80 & 0.79 \\
CC  &36808 &2 & 28671 & 1501 &0.95 & 0.78 & 0.86 \\
CD  &15084 &1 & 747 & 809 &0.48 & 0.05 & 0.09 \\
DT  &129626 &6 & 119534 & 6178 &0.95 & 0.92 & 0.94 \\
IN  &132079 &11 & 125554 & 25316 &0.83 & 0.95 & 0.89 \\
ING  &14753 &4 & 3096 & 4876 &0.39 & 0.21 & 0.27 \\
MD  &13498 &2 & 12983 & 936 &0.93 & 0.96 & 0.95 \\
N  &231424 &68 & 207822 & 51695 &0.80 & 0.90 & 0.85 \\
POS  &5086 &2 & 4623 & 533 &0.90 & 0.91 & 0.90 \\
PRP  &47686 &7 & 44946 & 12759 &0.78 & 0.94 & 0.85 \\
RB  &54524 &16 & 31184 & 17403 &0.64 & 0.57 & 0.60 \\
TO  &25196 &1 & 23291 & 61 &1.00 & 0.92 & 0.96 \\
VB  &35342 &8 & 29392 & 6152 &0.83 & 0.83 & 0.83 \\
VBD  &80058 &17 & 64150 & 8663 &0.88 & 0.80 & 0.84 \\
VBN  &41145 &11 & 25578 & 11972 &0.68 & 0.62 & 0.65 \\
WDT  &14093 &2 & 1621 & 1017 &0.61 & 0.12 & 0.19 \\
\hline
avg. &&&&& 0.78 & 0.71 & 0.72

\end{tabular}
\caption{\label{context} Precision and recall for induction based on
word type and context.}
\end{table*}
```

Table  shows that performance for generalized context vectors is better than for word-based context vectors (0.74 vs. 0.72). However, since the number of tags with better and worse performance is about the same (7 and 5), one cannot conclude with certainty that generalized context vectors induce tags of higher quality. Apparently, the 250 most frequent words capture most of the relevant distributional information so that the additional information from less frequent words available from generalized vectors only has a small effect.

```latex
\begin{table*}[tbp]\small
\centering
\begin{tabular}{l||r|r|r|r|r|r|l}
tag &frequency &\# classes &correct&incorrect &precision &recall &
F\\\hline\hline

ADN  &108586 &50 & 91893 & 26790 &0.77 & 0.85 & 0.81 \\
CC  &36808 &4 & 34127 & 6430 &0.84 & 0.93 & 0.88 \\
CD  &15085 &3 & 3707 & 1530 &0.71 & 0.25 & 0.36 \\
DT  &129626 &10 & 120968 & 5780 &0.95 & 0.93 & 0.94 \\
IN  &132079 &8 & 123516 & 22070 &0.85 & 0.94 & 0.89 \\
ING  &14753 &2 & 3798 & 7161 &0.35 & 0.26 & 0.30 \\
MD  &13498 &3 & 13175 & 1059 &0.93 & 0.98 & 0.95 \\
N  &231434 &70 & 201890 & 33206 &0.86 & 0.87 & 0.87 \\
POS  &5086 &2 & 4932 & 1636 &0.75 & 0.97 & 0.85 \\
PRP  &47686 &5 & 37535 & 9221 &0.80 & 0.79 & 0.79 \\
RB  &54524 &9 & 29892 & 18398 &0.62 & 0.55 & 0.58 \\
TO  &25196 &1 & 25181 & 27 &1.00 & 1.00 & 1.00 \\
VB  &35342 &7 & 28879 & 6560 &0.81 & 0.82 & 0.82 \\
VBD  &80058 &15 & 66457 & 12079 &0.85 & 0.83 & 0.84 \\
VBN  &41145 &10 & 26960 & 17356 &0.61 & 0.66 & 0.63 \\
WDT  &14093 &1 & 2223 & 563 &0.80 & 0.16 & 0.26 \\
\hline
avg. &&&&& 0.78 & 0.73 & 0.74

\end{tabular}
\caption{\label{general} Precision and recall for induction based on
generalized context vectors.}
\end{table*}
```
```latex
\begin{table*}[tbp]\small
\centering
\begin{tabular}{l||r|r|r|r|r|r|l}
tag &frequency &\# classes &correct&incorrect &precision &recall &
F\\\hline\hline

ADN  &63771 &36 & 54398 & 12203 &0.82 & 0.85 & 0.83 \\
CC  &16148 &4 & 15657 & 1798 &0.90 & 0.97 & 0.93 \\
CD  &7011 &1 & 1857 & 918 &0.67 & 0.26 & 0.38 \\
DT  &87914 &9 & 82206 & 2664 &0.97 & 0.94 & 0.95 \\
IN  &91950 &9 & 86793 & 6842 &0.93 & 0.94 & 0.94 \\
ING  &7268 &2 & 1243 & 1412 &0.47 & 0.17 & 0.25 \\
MD  &11244 &3 & 10363 & 476 &0.96 & 0.92 & 0.94 \\
N  &111368 &49 & 100105 & 14452 &0.87 & 0.90 & 0.89 \\
POS  &3202 &1 & 2912 & 255 &0.92 & 0.91 & 0.91 \\
PRP  &23946 &7 & 22877 & 4062 &0.85 & 0.96 & 0.90 \\
RB  &32331 &16 & 21037 & 9922 &0.68 & 0.65 & 0.66 \\
TO  &19859 &2 & 19537 & 53 &1.00 & 0.98 & 0.99 \\
VB  &26714 &11 & 24036 & 4119 &0.85 & 0.90 & 0.88 \\
VBD  &56540 &33 & 51016 & 8488 &0.86 & 0.90 & 0.88 \\
VBN  &24804 &14 & 18889 & 7448 &0.72 & 0.76 & 0.74 \\
WDT  &8329 &3 & 3691 & 670 &0.85 & 0.44 & 0.58 \\
\hline
avg. &&&&& 0.83 & 0.78 & 0.79

\end{tabular}
\caption{\label{natural} Precision and recall for induction for
natural contexts.}
\end{table*}
```

Table  looks at results for "natural" contexts, i.e. those not containing punctuation marks and rare words. Performance is consistently better than for the evaluation on all contexts, indicating that the low quality of the distributional information about punctuation marks and rare words is a difficulty for successful tag induction.

Even for "natural" contexts, performance varies considerably. It is fairly good for prepositions, determiners, pronouns, conjunctions, the infinitive marker, modals, and the possessive marker. Tag induction fails for cardinals (for the reasons mentioned above) and for "-ing" forms. Present participles and gerunds are difficult because they exhibit both verbal and nominal properties and occur in a wide variety of different contexts whereas other parts of speech have a few typical and frequent contexts.

It may seem worrying that some of the tags are assigned a high number of clusters (e.g., 49 for N, 36 for ADN). A closer look reveals that many clusters embody finer distinctions. Some examples: Nouns in cluster 0 are heads of larger noun phrases, whereas the nouns in cluster 1 are full-fledged NPs. The members of classes 29 and 111 function as subjects. Class 49 consists of proper nouns. However, there are many pairs or triples of clusters that should be collapsed into one on linguistic grounds. They were separated on distributional criteria that don't have linguistic correlates.

An analysis of the divergence between our classification and the manually assigned tags revealed three main sources of errors: rare words and rare syntactic phenomena, indistinguishable distribution, and non-local dependencies.

Rare words are difficult because of lack of distributional evidence. For example, "ties" is used as a verb only 2 times (out of 15 occurrences in the corpus). Both occurrences are miscategorized, since its context vectors do not provide enough evidence for the verbal use. Rare syntactic constructions pose a related problem: There are not enough instances to justify the creation of a separate cluster. For example, verbs taking bare infinitives were classified as adverbs since this is too rare a phenomenon to provide strong distributional evidence ("we do not DARE speak of", "legislation could HELP remove").

The case of the tags "VBN" and "PRD" (past participles and predicative adjectives) demonstrates the difficulties of word classes with indistinguishable distributions. There are hardly any distributional clues for distinguishing "VBN" and "PRD" since both are mainly used as complements of "to be".[^3] A common tag class was created for "VBN" and "PRD" to show that they are reasonably well distinguished from other parts of speech, even if not from each other. Semantic understanding is necessary to distinguish between the states described by phrases of the form "*to be* adjective" and the processes described by phrases of the form "*to be* past participle".

Finally, the method fails if there are no local dependencies that could be used for categorization and only non-local dependencies are informative. For example, the adverb in "Mc\*N. Hester, CURRENTLY Dean of …" and the conjunction in "to add that, IF United States policies …" have similar immediate neighbors (comma, NP). The decision to consider only immediate neighbors is responsible for this type of error since taking a wider context into account would disambiguate the parts of speech in question.

# Future Work

There are three avenues of future research we are interested in pursuing. First, we are planning to apply the algorithm to an as yet untagged language. Languages with a rich morphology may be more difficult than English since with fewer tokens per type, there is less data on which to base a categorization decision.

Secondly, the error analysis suggests that considering non-local dependencies would improve results. Categories that can be induced well (those characterized by local dependencies) could be input into procedures that learn phrase structure (e.g. ). These phrase constraints could then be incorporated into the distributional tagger to characterize non-local dependencies.

Finally, our procedure induces a "hard" part-of-speech classification of occurrences in context, i.e., each occurrence is assigned to only one category. It is by no means generally accepted that such a classification is linguistically adequate. There is both synchronic and diachronic evidence suggesting that words and their uses can inherit properties from several prototypical syntactic categories. For example, "fun" in "It's a fun thing to do." has properties of both a noun and an adjective (superlative "funnest" possible). We are planning to explore "soft" classification algorithms that can account for these phenomena.

# Conclusion

In this paper, we have attempted to construct an algorithm for fully automatic distributional tagging, using unannotated corpora as the sole source of information. The main innovation is that the algorithm is able to deal with part-of-speech ambiguity, a pervasive phenomenon in natural language that was unaccounted for in previous work on learning categories from corpora. The method was systematically evaluated on the Brown corpus. Even if no automatic procedure can rival the accuracy of human tagging, we hope that the algorithm will facilitate the initial tagging of texts in new languages and sublanguages.

# Acknowledgments

I am grateful for helpful comments to Steve Finch, Jan Pedersen and two anonymous reviewers (from ACL and EACL). I'm also indebted to Michael Berry for SVDPACK and to the Penn Treebank Project for the parsed Brown corpus.

[^1]: Although classifies collocations, these can also be ambiguous. For example, "for certain" has both senses of "certain": "particular" and "sure".

[^2]: The small difference in overall frequency in the tables is due to the fact that some word-based context vectors consist entirely of zeros. There were about a hundred word triplets whose four context vectors did not have non-zero entries and could not be assigned a cluster.

[^3]: Because of phrases like "I had sweet potatoes", forms of "have" cannot serve as a reliable discriminator either.


abstract: Facial expression recognition is a topic of great interest in most fields from artificial intelligence and gaming to marketing and healthcare. The goal of this paper is to classify images of human faces into one of seven basic emotions. A number of different models were experimented with, including decision trees and neural networks before arriving at a final Convolutional Neural Network (CNN) model. CNNs work better for image recognition tasks since they are able to capture spacial features of the inputs due to their large number of filters. The proposed model consists of six convolutional layers, two max pooling layers and two fully connected layers. Upon tuning of the various hyperparameters, this model achieved a final accuracy of 0.60.
author:
;
;
title: Facial Emotion Recognition using Convolutional Neural Networks

# Introduction

Human beings communicate with each other in the form of speech, gestures and emotions. As such systems that can recognize the same are in great demand in many fields. With respect to artificial intelligence, a computer will be able to interact with humans much more naturally if they are capable of understanding human emotion. It would also help during counseling and other health care related fields. In an E-Learning system, the presentation style may be varied depending on the student's state. However in many cases, static emotion detection is not very useful. It is essential to know the user's feelings over a period of time in a live environment. Thus, the paper proposes a model that is aimed at real-time facial emotion recognition.

For real-time purposes, facial emotion recognition has a number of applications. Facial emotion recognition could be used in conjunction with other systems to provide a form of safety. For instance, ATMs could be set up such that they won't dispense money when the user is scared. In the gaming industry, emotion-aware games can be developed which could vary the difficulty of a level depending on the player's emotions. It also has uses in video game testing. At present, players usually give some form of verbal or written feedback. Using facial emotion recognition, the number of testers can be increased to accomodate people who use different languages or people who are unable to cohesively state their opinions on the game. By judging their expressions during different points of the game, a general understanding of the game's strong and weak points can be discerned. Emotions can also be gauged while a viewer watches ads to see how they react to them. This is especially helpful since ads do not usually have feedback mechanisms apart from tracking whether the ad was watched and whether there was any user interaction. Software for cameras can use emotion recognition to take photos whenever a user smiles.

However not all emotions can be inferred just by looking at someone. In his 1971 paper, Paul Ekman et al. identified six basic, universal facial expressions - anger, disgust. fear, happiness, sadness and surprise. Even today, researchers aim to identify these six emotions with reliable accuracy. Emotions can be inferred from a person's actions, speech, writing and facial expressions. In terms of facial emotion recognition, one major challenge lies in the data collected. Most datasets contain labelled images which are generally posed. This generally involves photos taken in a stable environment such as a laboratory. While it is much easier to accurately predict the emotion in such scenarios, these systems tend to be unreliable in predicting emotions in the "wild" (Uncontrolled environments). Another issue is that most datasets are from these controlled environments and it is relatively harder to obtain labelled datasets of emotions in the wild. Furthermore, most datasets have relatively lesser training data for emotions such as fear and disgust when compared to emotions such as happiness. Another factor to take into account is a person's pose. It is significantly harder to determine the emotion of a person when only half of their face is visible. In addition, lighting plays a major role in facial emotion recognition. Systems may fail to identify an emotion that it normally would identify if the lighting conditions are poor. Finally, one must remember that a user's emotional state is a combination of many factors, a smile does not always mean that a person is genuinely happy.

The objective of this project is to classify human faces into one of the six universal emotions or a seventh neutral emotion. In recent years, many papers have been published that use deep learning for facial emotion recognition . These papers used freely available datasets with state of the art models achieving an accuracy of 0.66. With this in mind, a number of different models both new and old were experimented with to arrive at a final model with comparable results.

# Related Work

Yu and Zhang used a five layer ensemble CNN to achieve a 0.612 accuracy. They pre-trained their models on the FER-2013 dataset and then finetuned the model on the Static Facial Expressions in the Wild 2.0 (SFEW) dataset. They used an ensemble of three face detectors to detect and extract faces from the labelled movie frames of SFEW. They then proposed a data perturbation and voting method to increase the recognition performance of the CNN. They also chose to use stochastic pooling layers over max pooling layers citing its better performance on their limited data.

Kahou et al. used a CNN-RNN architecture to train a model on individual frames of videos as well as static images. They made use of the Acted Facial Expressions in the Wild (AFEW) 5.0 dataset for the video clips and a combination of the FER-2013 and Toronto Face Database for the images. Instead of using long short term memory (LSTM) units, they used IRNNs which are composed of rectified linear units (ReLUs). These IRNNs provided a simple mechanism for dealing with the vanishing and exploding gradient problem. They achieved an overall accuracy of 0.528.

Mollahosseini et al. proposed a network consisting of two convolutional layers each followed by max pooling and then four Inception layers. They used this network on seven different datasets including the FER-2013 dataset. They also compared the accuracies of their proposed network with an AlexNet network trained on the same datasets. They found that their architecture had better performance on the MMI and FER-2013 datasets with comparable performances on the remaining five datasets. The FER-2013 dataset in particular managed to reach an accuracy of 0.664.

Ming Li et al. propose a neural network model to overcome two shortcomings in still image based FERs which are the inter-variability of emotions between subjects and misclassification of emotions. The model consists of two convolutional neural networks - the first is trained with facial expression databases whereas the second is a DeepID network used for learning identity features. These two networks are then concatenated together as a Tandem Facial Expression of TFE Feature which is fed to the fully connected layers to form a new model. The proposed model was evaluated on two datasets, namely the FER+ database and the Extended Cohn-Kanade (CK+) database. The identity features were learned from the CASIA-WebFace database. The model was trained for 200 epochs and achieved an accuracy measure of 71.1% on the FER2013 dataset, 99.31% on the CK+ database. These experimental results show that the model outperforms many state-of-the-art methods on the CK+ and FER+ databases.

Tan et al. propose a neural network model to classify a group image into a particular emotion - positive, neutral or negative. The model consists of two convolutional neural networks - the first is based on group images and the second is based on individual facial emotions. The facial emotion CNN comprises of two CNNs - one for aligned faces which is trained using the ResNet64 model using the Webface dataset and the other for non-aligned faces which is trained using the ResNet34 model on the FER+ dataset. The group images are trained using VGG19 model on the Places and ImageNet datasets. Fine-tuning is done with batch normalisation and average pooling of the ResNet101 and BN-Inception models, with a dropout of 0.5. The validation set consists of 2068 images - combined from all of the datasets used for training and the model achieved an accuracy measure of 80.9.

Most other works in the same field attempted to solve the facial emotion recognition problem by the use of a combination of different datsets. In this paper, a single dataset, FER-2013 was chosen over such a combination of different datasets and then experiments were conducted with different models to find the highest accuracy that each model could reach.

# Theoretical Background

Three different types of models are used in this paper. This section details the theoretical background for each of these models.

## Decision Tree

Decision trees are a supervised learning technique that predicts a value given a set of inputs by "learning" rules based on a set of training data. To put it simply, it is a massive tree of if-then-else rules. The decision making process starts off at the root of the tree and descends by answering a series of yes-no questions. At the end of this if-then-else chain, it arrives at a single predicted label. This is the output of a decision tree.

## Feedforward Neural Network

A neural network is a system of algorithms that attempts to identify underlying relationships in a set of data by using a method that mimics the way in which a human brain operates. Neural networks consist of nodes connected to each other through edges. Each connection has a weight and a bias. A weight is the strength of the connection. The greater the weight, the greater impact it will have on the final output. A bias is a minimum threshhold which the sum of all the weighted inputs must cross. Neural networks are primarily employed for classification tasks. Neural networks consist of three layers - input, output and hidden layers. Hidden layers are sets of features based on the previous layer. They are intermediate layers in the network. A neural network recognises objects based on the concept of learning. Learning consists of six steps. Initially weights are initialised and a batch of data is fetched. This data is known as training data. A forward propagation is done on the data by passing through the network. A metric of difference between expected output and actual output is computed by the use of activation functions which perform computations on the data based on standard mathematical distributions such as Hyperbolic tangent and Sigmoid. This is known as cost. The goal is to minimize or reduce the cost. For this purpose, gradients of cost and weight are backpropagated to know how to adjust the weights to reduce the cost. Backpropagation refers to a backward pass of the network. Later, the weights are updated and the whole process is repeated. Feed-forward networks are also termed as multi-layer perceptrons.

## Convolutional Neural Network

A Convolutional neural network is a neural network comprised of convolution layers which does computational heavy lifting by performing convolution. Convolution is a mathematical operation on two functions to produce a third function. It is to be noted that the image is not represented as pixels, but as numbers representing the pixel value. In terms of what the computer sees, there will simply just be a matrix of numbers. The convolution operation takes place on these numbers. We utilize both fully-connected layers as well as convolutional layers. In a fully-connected layer, every node is connected to every other neuron. They are the layers used in standard feedforward neural networks. Unlike the fully-connected layers, convolutional layers are not connected to every neuron. Connections are made across localized regions. A sliding "window" is moved across the image. The size of this window is known as the kernel or the filter. They help recognise patterns in the data. For each filter, there are two main properties to consider - padding and stride. Stride represents the step of the convolution operation, that is, the number of pixels the window moves across. Padding is the addition of null pixels to increase the size of an image. Null pixels here refers to pixels with value of 0. If we have a 5x5 image and a window with a 3x3 filter, a stride of 1 and no padding, the output of the convolutional layer will be a 3x3 image. This condensation of a feature map is known as pooling. In this case, "max pooling" is utilized. Here, the maximum value is taken from each sliding window and is placed in the output matrix.

Convolution is very effective in image recognition and classification compared to a feed-forward neural network. This is because convolution allows to reduce the number of parameters in a network and take advantage of spatial locality. Further, convolutional neural networks introduce the concept of pooling to reduce the number of parameters by downsampling. Applications of Convolutional neural networks include image recognition, self-driving cars and robotics. CNN is popularly used with videos, 2D images, spectrograms, Synthetic Aperture Radars.

# Experimental Setup

This section details the data used for training and testing, how the data was preprocessed, the various models that were used and an evaluation of each model.

## Dataset

In general, neural networks, especially deep neural networks, tend to perform better when larger amounts of training data set is present. With this in mind, the more popular Extended Cohn-Kanade (CK+) and the Static Facial Expressions in the Wild (SFEW) datasets were overlooked.

Instead the Facial Expression dataset (FER-2013) was chosen. The FER-2013 dataset was introduced in the ICML 2013 Challenges in Representation Learning . It contains 35,887 images with the following basic expressions: angry, disgusted, fearful, happy, sad, surprised and neutral. Figure shows the distribution of each expression. Each image is a frontal view of a subject, taken from the wild and annotated to one of the seven expressions. A sample of these expressions are shown in Figure . It is to be noted that the number of disgusted expressions (547) is much lower in comparison to the other expressions. There was also an obvious bias towards happy expressions due to the sheer number of sample data present for the expression.

## Preprocessing

The dataset consisted of a number of images represented as strings of 2304 space separated numbers which were then converted to a 48\*48 matrix. Each number represented a pixel value. The original data of 35,887 images was split into a training set of 28,709 images and a testing set of 7,178 images - an 80:20 split. Generally, when it comes to deep learning, data is the biggest factor. The bigger the training set, the better the output. If there is less training data, there is a lot more variance in the final outputs due to a smaller set to train on. Bearing that in mind, having a testing set of 20% of the total images may be seen as excessive. However, to prevent overfitting it is necessary to have a sizable testing set as well. It is also to be noted that in general, there is a 60-20-20 split up for training, testing & validation respectively. For instance, Mollahosseini et al. divided their 275k image dataset into 60% for training, 20% for testing and 20% for validation. In this case, the validation set was forewent in favor of retraining the entire model every time the hyper-parameters were tuned. While this required more time and computational power, it provided a bigger training set in the end. A one-hot encoding scheme was used for the labels rather than classifiying emotions with numbers from 0-6. During the live testing, Haar Cascades were used to identify a face. This identified face was then taken as an image, converted to gray-scale and downscaled to a 48\*48 image. Thus the image was converted to a format identical to that which was used to train the model.

## Choosing a Model

4 different models were used in order to select the best foundation to work on - 3 neural networks and a decision tree. The neural networks were implemented using Keras with a TensorFlow backend running in Python. The decision tree was implemented with the help of Sci-Kit Learn . All the implementations are written in Python 3.6 and are wholy reproducible with freely available software. A comparison of the final accuracy of all the models can be seen in Table .

Since there are 4 different models used, the individual training algorithm for each model is detailed with the model description itself. The testing algorithm for the same is described in subsection D.

|      Model      | Accuracy |
|:---------------:|:--------:|
|  Decision Tree  |  30.84%  |
| Feed Forward NN |  17.38%  |
|   Simple CNN    |  24.72%  |
|  Proposed CNN   |  55.61%  |

Comparison of Models

### Decision Tree

A decision tree was chosen at first chosen due to a combination of it requiring little to no effort in preparation of data as well as its reputation for working well in almost all scenarios. The first attempt was to implement a decision tree using Sci-Kit Learn. A standard decision tree classifier was used and the parameters were tuned. However the only parameter that showed a noticeable difference while tuning was the emphmin_samples_split parameter. With the parameter set to 40, the overall accuracy was only a mere 0.309. As state of the art models reached accuracies greater than 0.6, it was decided to try out a neural network instead.

| **Feed-Forward Neural Network** |
|:-------------------------------:|
|         FULLY CONNECTED         |
|              RELU               |
|             DROPOUT             |
|         FULLY CONNECTED         |
|              RELU               |
|             DROPOUT             |
|         FULLY CONNECTED         |
|             SOFTMAX             |

Architecture of the Feedforward Neural Network

### Feedforward Neural Network

The next attempt was to try out the most basic form of an artificial neural network, a Feedforward Neural Network. Three layers were used - an input layer, a single hidden layer and an output layer. Each of these layers was a fully connected (dense) layer. The architecture of this model can be seen in Table . A dropout of rate 0.2 was applied for the input and hidden layer in an attempt to prevent overfitting. The output layer uses a softmax activation function while the remaining layers use ReLU (Rectified Linear Units). However this model ended up predicting the same expression for every input - angry. Further tuning of the hyperparameters lead to no difference and so it was decided that a convolutional neural network might work better.

| **Simple Convolutional Neural Network** |
|:---------------------------------------:|
|                CONV2D-32                |
|                  RELU                   |
|                CONV2D-64                |
|                  RELU                   |
|                MAXPOOL2D                |
|                 DROPOUT                 |
|                 FLATTEN                 |
|             FULLY CONNECTED             |
|                  RELU                   |
|                 DROPOUT                 |
|             FULLY CONNECTED             |
|                 SOFTMAX                 |

Architecture of the Simple CNN

### Simple Convolutional Network

Next, a basic convolutional network was tried out. This model's architecture can be seen in Table . This model consisted of two two-dimensional convolutional layers followed by a two-dimensional max pooling layer which was followed by two fully-connnected(Dense) layers. The output was flattened before entering the fully connected layers. Dropout was applied to the max pooling and to the first fully connected layer to reduce overfitting. This model while more complicated, ended up having the same issue as the feedforward neural network except instead of predicting angry, it predicted happy for all inputs. This makes sense as a quarter of the inputs are for the happy expression. In order to allow the model to actually learn, attempts were made to make the model deeper.

### Final Model

The final model is depicted in Table . The network consists of six two-dimensional convolutional layers, two max pooling layers and two fully connected layers. Max pooling uses the maximum value from each of a cluster of neurons at the prior layer. This reduces the dimensionality of the output array. The input to the network is a preprocessed face of 48 x 48 pixels. The model was developed based on the observation of the performance of the previous models. It was decided to go with a deeper network over a wide one. The advantage of using more layers is that it prevents memorization. A wide but shallow network memorizes well but does not generalize well. Multi-layer networks learn features at levels of abstractions allowing them to generalize well. The number of layers were selected so as to maintain a high level of accuracy while still being fast enough for real-time purposes. The proposed CNN differs from a simple CNN in that it uses 4 more convolutional layers and each of its convolutional layers differ in filter size. In addition, it utilized max pooling and dropout more effectively in order to minimize overfitting.

| **Proposed Convolutional Neural Network** |
|:-----------------------------------------:|
|                 CONV2D-64                 |
|                   RELU                    |
|                 CONV2D-64                 |
|                   RELU                    |
|                 MAXPOOL2D                 |
|                  DROPOUT                  |
|                CONV2D-128                 |
|                   RELU                    |
|                CONV2D-128                 |
|                   RELU                    |
|                CONV2D-256                 |
|                   RELU                    |
|                CONV2D-256                 |
|                   RELU                    |
|                 MAXPOOL2D                 |
|                  DROPOUT                  |
|                  FLATTEN                  |
|              FULLY CONNECTED              |
|                   RELU                    |
|                  DROPOUT                  |
|              FULLY CONNECTED              |
|                  SOFTMAX                  |

Architecture of the Proposed CNN

The network consists of two convolutional layers with a filter size of 64 each. This is then followed by a max pooling layer. A dropout of rate 0.25 is applied to reduce overfitting. This is followed by a sequence of four convolutional layers. The first two have a filter size of 128 each and the latter two have a filter size of 256 each. A single max pooling layer follows these four layers with a dropout of rate 0.25. In order to convert the output into a single dimensional vector, the output of the previous layers was flattened. A fully connected layer with a L2 regularizer of penalty of 0.001 is then used alogn with an additional dropout of rate 0.5. Finally, a fully connected layer with a softmax activation function serves as the output layer.

The kernel size, that is, the width and height of the 2D convolutional window is set to 3 x 3 for all convolutional layers. Each max pooling layer is two dimensional and uses a pool size of 2 x 2. This halves the size of the output after each pooling layer. All the layers bar the output layer used a ReLU activation function. The ReLU activation function is used here due to benefits such as sparsity and a reduced likelihood of vanishing gradient. The softmax activation function was used in the final output layer to receive the predicted probability of each emotion.

This model provided a base accuracy of 0.55 on the testing set. The hyperparamters were then tuned, namely the batch size, the optimizer and the number of epochs. Each model was set to run for 100 epochs. However, in the interest of saving time and computational power, the network was allowed to stop training if there was no change in the accuracy over consecutive epochs. That is, the network would stop training if there was no change in the accuracy over 4 continuous epochs. This saved both time and computational power, especially in cases where there was no change in the accuracy within the earlier epochs themselves. The decision turned out to be a good one as none of the models exceeded 20 epochs.

## Testing

The dataset was initially split into an 80%-training set and a 20%-testing set. During the testing phase, each of the trained networks was loaded and fed the entire testing set one image at a time. This image was a new one which the model had never seen before. The image fed to the model was preprocessed in the same way as detailed in . Thus the model did not know already what the correct output was and had to accurately predict it based on its own training. It attempted to classify the emotion shown on the image simply based on what it had already learned along with the characteristics of the image itself. Thus in the end, it gave a list of classified emotion probabilities for each image. The highest probability emotion for each image was then compared with the actual emotions associated with the images to count the number of accurate predictions.

The accuracy formula is detailed below. It simply counts the number of samples where the model correctly predicted the emotion and divides it by the total number of samples in the testing set. Here, the testing set consists of about 7,178 images. $$\label{accuracy}
Accuracy = \frac{Num. Correctly Predicted Emotions }{Total Num. Samples}$$

# Results

Upon tuning the hyperparameters, the highest accuracy was achieved for each optimizer. Using the RMSProp optimizer, an accuracy of 0.57 was reached over 20 epochs and a batch size of 96. The Stochastic Gradient Descent optimizer gave an accuracy of 0.55 out of the box and it could not be increased significantly by further tuning of the hyperparameters. Using the Adam optimizer with the default settings, a batch size of 64 and 10 epochs lead to an astoundingly low accuracy of 0.17. However upon setting the learning rate to 0.0001 and the decay to $10e-6$, the highest accuracy of 0.60 was attained. A comparison of the various hyperparameters that were tuned can be seen in Table .

| Optimizer | Batch Size | Epochs | Accuracy |
|:---------:|:----------:|:------:|:--------:|
|  RMSProp  |     64     |   24   |  55.96%  |
|  RMSProp  |     32     |   9    |  42.07%  |
|  RMSProp  |     96     |   20   |  57.39%  |
|    SGD    |     64     |   10   |  55.90%  |
|   Adam    |     64     |   10   |  17.38%  |
|   Adam    |    128     |   20   |  60.58%  |

Comparison of Hyperparameters

Based on these results it can be concluded that the Adam optimizer which initially provided an abysmal accuracy turned out to be the best fit for the data. This makes sense as Adam is based off of RMSProp & AdaGrad both of which are extensions of Stochastic Gradient Descent (SGD). It realizes the benefits of both RMSProp & AdaGrad by utilizing an adaptive learning rate as well as bringing in momentum. In Adam (and RMSProp), the learning rate of each parameter is adaptively decided. Parameters that would ordinarily receive smaller or less frequent updates receive larger updates with Adam (the reverse is also true). This speeds up learning in cases where the appropriate learning rates vary across parameters. This is not the case with SGD which requires careful tuning of learning rates.

Adam and RMSProp differ only in that Adam uses the concept of Momentum. Like the physical phenomenon, momentum adds a fraction of the previous update to the current update, so that repeated updates in a particular direction compound. Thus a "momentum" is created, causing it to move faster and faster in that direction. Local optima are thus skipped due to the momentum and convergence is also sped up. It is also to be noted that the learning rate and decay had to be adjusted to arrive at a good accuracy. The reason why the Adam optimizer was initially unable to offer good results with the default learning rate of 0.001 and a decay of 0 was that it failed to converge. By setting the learning rate to a much smaller value in 0.0001 and the decay to $10e-6$, convergence actually occured. The decay value was set in such a way that over time the learning rate would further reduce. However accuracy alone doesn't paint the whole picture. A confusion matrix for the final model (using the Adam optimizer) was generated to take a look at how each individual emotion is dealt with. This matrix can be seen in Figure .

Understandably happiness is very easy to determine as a direct result of the number of sample data present. Interestingly the emotion of surprise reached nearly the same accuracy. The other emotions had lower but similar accuracies. Another point of interest is that it manages to determine the emotion of disgust a little more than half the time. The model when given an image (or a frame from a video) to predict from, does not simply give one final prediction. Rather it predicts a list of probabilities of each individual emotion. We then take the emotion with the highest probability as the final prediction. Thus we classify the status of the facial reaction based on the most probable emotion predicted by the model. Considering the sparse number of sample data, it is possible that the model may have been overfit.

To put this theory to the test, a small tool was developed that took a webcam's feed, detected faces, processed the faces and then fed it to the model. It was found that the model managed to predict almost all instances of happiness and most instances of surprise. It correctly predicted sadness and neutrality about half the time but it rarely predicted the other emotions correctly. The emotions of anger and fear in particular tended to mix while disgust was almost never predicted. It is also to be noted that in most cases of a wrong prediction, the second most likely prediction was often the right one. This indicates that the top-2 predicted emotions will be much more accurate. This is confirmed by Mollahosseini et al. Finally, during the live testing, it was observed that the model was able to predict emotions upon detecting a face instantaneously with no delays.

# Conclusion & Future Work

In this paper, the aim was to classify facial expressions into one of seven emotions by using various models on the FER-2013 dataset. Models that were experimented with include decision trees, feed forward neural networks and smaller convolutional networks before arriving at the proposed model. The effects of different hyperparameters on the final model was then investigated. The final accuracy of 0.60 was achieved using the Adam optimizer with modified hyperparameters. It should also be noted that a nearly state-of-the-art accuracy was achieved with the use of a single dataset as opposed to a combination of many datasets. While it is true that other related works have managed to obtain higher accuracies - Mollahosseini et al.(0.66) and Yu and Zhang (0.61), they have used a combination of different datasets and large models in order to increase their overall accuracy. shows a comparison between the proposed approach and existing methodologies.

Given that only the FER-2013 dataset was used in this case without the use of other datasets, an accuracy of 0.60 is admirable as it demonstrates the efficiency of the model. In other words, the model demonstrated has used significantly less data for training and a deep but simple architecture to attain near-state-of-the-art results.

At the same time, it also has its shortcomings. While the model did attain near-state-of-the-art results, it also means that it did not achieve state-of-the-art. Additionally, the relatively lower amount of data for emotions such as "disgust" make the model have difficulty predicting it. This however does illuminate a path for future work. If provided with more training data while still retaining the same network structure, the efficiency of the proposed system will be enhanced considerably. Sang, Dat and Thuan who used the same dataset, augmented the data to greatly increase the size of the training set to achieve similar results. Thus augmenting the existing data to enlarge the dataset might also prove to be a worthwhile avenue to explore.

The ability of the model to make predictions in effectively real-time, indicates that real world uses of facial emotion recognition is barred only by the relative inaccuracies of the model itself. In the future, an indepth analysis of the top-2 predicted emotions may lead to a much more accurate and reliable system. Further training samples for the more difficult to predict emotion of disgust will definitely be required in order to perfect such a system.

The real-time capacity of the model in addition to its quick training time and near-state-of-the-art accuracy allows the model to be adapted and used in nearly any use-case. This also implies that with some work, the model could very well be deployed into real-life applications for effective utilization in domains such as in healthcare, marketing and the video game industry.


abstract: We express community detection as an inference problem of determining the most likely arrangement of communities. We then apply belief propagation and mean-field theory to this problem, and show that this leads to fast, accurate algorithms for community detection.
author: M. B. Hastings
date: April 6, 2006
title: Community Detection as an Inference Problem

Community detection is a well-studied problem in networks. This is the problem of dividing a network into communities, such that nodes within the same community tend to be connected by links, while those within different communities tend not to be connected by links. This problem has applications in understanding the structure of social and biological networks, while the closely related graph partitioning problem discussed below has applications in parallel processing, to allocate assignments to different processors while minimizing interprocessor communication.

Unfortunately, despite all this interest, there is no formal definition of the problem. Instead, each author tends to define communities as being whatever is found by a particular community detection algorithm. In this work, we exploit a standard method of testing communities, the four groups test, to express community detection as an inference, or maximum likelihood problem. This leads to a derivation of a Potts model similar to those derived previously on phenomenological grounds.

To solve this inference problem, we must find the ground state of the Potts model. To do this, we turn to the techniques of *belief propagation*, also known as sum-product, and mean-field theory. Belief propagation was originally developed to perform decoding in a certain class of error correcting codes, called low-density parity check codes. In this problem, one has a sender and receiver communicating over a noisy channel, and the receiver must determine which of all the possible messages is the most likely. This problem can be mapped onto finding the ground state of a spin system on a particular graph. The belief propagation algorithm exploits the fact that the graph has a low density of loops to solve this problem, in a manner similar to the famous Bethe-Peierls solution for the thermodynamics of a spin system on a Bethe lattice.

We will find that the resulting belief propagation algorithm for community detection is highly accurate on the four groups problem, while also performing well on other test networks. We then discuss the scaling of computational time with system size, extensions of this algorithm, and other problems.

*Inference Problem—* To express community detection as an inference problem we consider the following method, often used to test community detection algorithms. We invent a network as follows: we consider $N$ nodes, divided into $q$ different communities with $n_k$, $k=1,...,q$ nodes in each community. We consider each pair of nodes in turn. We connect those nodes with probability $p_{in}$ if they lie within the same community and $p_{out}$ if they lie within different communities. We then run the community detection algorithm and see if it correctly assigns nodes to communities. Let $q_i$ be the initial assignment of a community to node $i$. The probability that a given graph arises from this procedure is equal to $$\begin{aligned}
\label{pqi}
p(\{q_i\})=
\\ \nonumber
\left( \prod_{k=1}^q (1-p_{in})^{n_k^2/2-n_k/2} \right)
\left( \prod_{k=1}^{q-1} \prod_{l=k+1}^q
(1-p_{out})^{n_k n_l}
\right)
\times
\\ \nonumber
\left( \prod_{<ij>} (\frac{p_{in}}{1-p_{in}})^{\delta_{q_i,q_j}} \right)
\left( \prod_{<ij>} (\frac{p_{out}}{1-p_{out}})^{1-\delta_{q_i,q_j}} \right)
\end{aligned}$$ where $\prod_{<ij>}$ denotes a product over pairs of vertices $i,j$ connected by an edge. To verify the correctness of this formula, first consider the case in which there are no edges at all in the graph. Then, the probability is given correctly by the first two products of Eq. (). Adding edges to the graph changes the probability as given by the second two products in Eq. ().

We now consider a given graph and formulate the problem as follows: find the most likely community assignment. Following Bayes' theorem, given a graph, the probability that any given community assignment is the "correct" assignment is proportional to the probability $p(\{q_i\})$ multiplied by the a priori probability of having a given $n_k$. Throughout this paper we assume the a priori probability of a given $n_k$ is constant, and thus the optimal community assignment maximizes the probability in Eq. (); any non-constant priori probability can be easily incorporated by adding additional terms.

We rewrite Eq. () as an exponential: $$\label{pqie}
p(\{q_i\})=
c \exp[\sum_{<ij>} J \delta_{q_i,q_j}]
\exp[\sum_{i\neq j} J' \delta_{q_i,q_j}/2],$$ where $\sum_{<ij>}$ denotes a sum over pairs of $i,j$ connected by an edge in the graph, and $c=\exp[\log(1-p_{out})N(N-1)/2]
\exp[\sum_{<ij>} \log(p_{out}/(1-p_{out})]$, and with $$\begin{aligned}
\label{constants}
J=\log[(p_{in}(1-p_{out}))/((1-p_{in})p_{out})],\\ \nonumber
J'=\log[(1-p_{in})/(1-p_{out})].
\end{aligned}$$ The factor of $1/2$ in Eq. () is to avoid double counting. Eq. () presents the probability as a Potts model problem with combined short- and long-range interactions, with coupling constants $J,J'$. The problem of community detection is then reduced to finding the ground state of this Potts model. Assuming $p_{in}>p_{out}$, the short-range interactions are ferromagnetic, favoring the assignment of neighboring nodes to the same community, while the long-range interactions are anti-ferromagnetic and prevent one from simply taking all nodes to lie within the same community. The problem of finding the ground state is very closely related to the NP-complete problem of graph partitioning, to break a graph up into partitions, minimizing the number of edges connecting partitions and minimizing the difference in number of nodes between partitions.

*Belief Propagation—* Having arrived at Eq. () we have a very similar problem to that studied in . Instead of using Monte Carlo methods to find the ground state, we adopt the method of belief propagation, which we believe to be more efficient for many of the community detection problems that arise. Indeed, for the inference problems which arise many in error correcting codes, belief propagation is the most efficient method.

We begin by taking a mean-field approximation for the long-range interactions, justified for large $N$. We approximate $p(\{q_i\})\approx
p_{mft}(\{q_i\})$, where $$\begin{aligned}
p_{mft}(\{q_i\})\equiv
Z^{-1} \exp[\sum_{<ij>} J \delta_{q_i,q_j}]
\exp[\sum_{i} h_i(q_i)]
\end{aligned}$$ where $Z$ is a normalization, and for $r=1...q$ we define $h_i(r)=J' N \rho(r)-J' p_i(r)$ and $\rho(r)=\sum_i p_i(r)/N$, with $$p_i(r)=\frac{\sum_{\{q_j\}, q_i=r} p_{mft}(\{q_j\})}{\sum_{\{q_j\}} p_{mft}(\{q_j\})},$$ so that $p_i(r)$ is the probability in the mean-field approximation that node $i$ belongs to community $r$. These are a set of self-consistent equations for $p_i(q)$.

We will find that solving these equations, at least in the belief propagation approximation below, leads to a spontaneous symmetry breaking: for sufficiently large $J,J'$, the probability $p_i(q)$ depends on $q$. Of course, given any solution which breaks symmetry, one can arrive at other valid solutions by relabeling the communities. We then make a "nodewise" maximum a posteriori probability (MAP) approximation: for each node $i$, we compute $p_i(q)$ and then assign the node $i$ to the community $q$ which maximizes $p_i(q)$. This is an approximation: the set of community assignments which maximizes $p(\{q_i\})$ may not be given by maximizing the probability for each node separately. However, similar approximations work very well for error correcting codes, and these approximations are justified for large $J,J'$.

To compute the $p_i(q)$, we apply belief propagation. Suppose the graph forms a tree, with no loops. Then, the problem of solving for $p_i(q)$ given $h_i(q)$ can be solved: for each pair of nodes $i,j$ connected by an edge, we define $p_{ij}(r)$ to be probability $p_i$ for the network modified by removing the edge connecting $i$ to $j$. That is, $$p_{ij}(r)=\frac{\sum_{\{q_k\}, q_i=r} \exp(-J \delta_{q_i,q_j})
p_{mft}(\{q_k\})}{\sum_{\{q_k\}} \exp(-J \delta_{q_i,q_j}) p_{mft}(\{q_k\})},$$ as $\exp(-J \delta_{q_i,q_j}) p_{mft}(\{q_k\})$ is the probability distribution on the network with the edge removed. Then, for a tree-like structure, the Bethe-Peierls solution gives $$\label{bppi}
p_i(r)=Z_i^{-1} \exp[h_i(r)]\prod_{j}^{i,j \, n.n.}
\bigl[ \exp(J) p_{ji}(r)+(1-p_{ji}(r) \bigr] ,$$ where $Z_i$ is chosen so that $\sum_{r=1}^q p_i(r)=1$, and where the product $\prod_{j}^{i,j \, n.n.}$ is taken over all nodes $j$ which are connected to node $i$. Note that $\exp(J) p_{ji}(r)+(1-p_{ji}(r))=\sum_s p_{ji}(s) (\delta_{r,s} \exp(J)+(1-\delta_{r,s}))$. Similarly, $$\label{bppij}
p_{ij}(r)=Z_{ij}^{-1} \exp[h_i(r)]\prod_{k\neq j}^{i,k \, n.n.}
\bigl[\exp(J) p_{ki}(r)+(1-p_{ki}(r)\bigr],$$ where again we choose $Z_{ij}$ such that $\sum_{r=1}^q p_{ij}(r)=1$.

These belief propagation equations are exact for a tree. If the network is not a tree, however, belief propagation can still be very effective, especially if the density of loops is small. Of course, not only may the graph have loops, but the long-range interactions also induce loops. However, for large $N$, the mean-field approximation we have used is justified for the long-range interactions, and as discussed below the procedure works well even for test networks with loops.

We can simplify the algorithm by replacing the belief propagation equations by a set of naive mean-field equations: for each site we track only the beliefs $p_i(q_i)$ and iterate the equations $$\label{mfteq}
p_i(r)=Z_i^{-1} \prod_{j,<ij>} \exp[h_i(r)] \bigl(\exp(J) p_j(r)+ (1-p_j(r)) \bigr).$$ This method is most appropriate for networks with large $z$ where it becomes faster than the belief propagation method by roughly a factor $z$ as there are fewer equations to solve; also, as we will see, for large $z$ the mean-field approximation used here performs comparably to belief propagation.

There are many ways to solve the belief propagation equations. We choose to initialize each of the $p_i(r)=1/q+\epsilon_i(r)$, where $\epsilon_i(r)$ is chosen randomly and is small, and similarly for each of the $p_{ij}(r)$. We then perform a fixed number of iterations: on each iteration we first randomly select one directed edge $i,j$ and then update the corresponding function $p_{ij}(r)$ by replacing its current value with $0.75$ times its current value plus $0.25$ times the value given by solving Eq. (). We then randomly select one node $i$ and replace the function $p_i(r)$ by its $0.75$ times its current value plus $0.25$ times the value given by solving Eq. (). After updating $p_i(r)$ we update $\rho(r)$ and $m_i(r)$, thus solving the belief propagation and mean-field equations simultaneously. Other relaxation methods may prove better in certain applications.

*Applications—* We have tested this algorithm on several problems. We first consider the four groups problem. In this case, we take a randomly generated network of $N=128$ nodes, divided into four communities of 32 nodes each. After generating the network, we run the community detection algorithm, to find the most likely assignment of communities. We measure the algorithm's performance by determining the fraction of nodes whose community it identifies correctly. There is some arbitrariness in how this fraction of correctly identified nodes is defined, which is connected with the arbitrariness in the labeling of communities: one can permute the community labels as desired, given the Potts symmetry of Eq. (). To resolve this arbitrariness, we follow the stringent definition adopted in for the accuracy.

We choose $p_{in}$ and $p_{out}$ so that each node has on average $z_{in}$ connections to other nodes in the same community and $z_{out}$ connections to nodes in different communities. We pick $z_{in}+z_{out}=16$, and consider a range of values of $z_{in}$, the average number of intra-community links. For large $z_{in}$, the community detection problem is easier, as the effect of the community structure is much more clear.

The results of this procedure are shown in Fig. 1. As seen, with 100000 iterations, the algorithm is highly accurate. With 50000 iterations, a slight decrease in accuracy is noticed for small $z_{in}$, and the accuracy goes down significantly at $10000$ iterations (by increasing to $200000$ iterations, a slight improvement is noted for $z_{in}<8$). For the first three curves, we used the choice of constants in Eq. (). Since these constants depend on the given $z_{in},z_{out}$, something which may not be known for an arbitrary network, we repeated the algorithm with a particular fixed choice of constants, $\exp[J]=6, \exp[NJ']=10^{-10}$. This choice of constants was chosen completely arbitrarily, but as seen the algorithm still works well, with a much higher accuracy than the Newman-Girvan algorithm. However, the Newman-Girvan algorithm has some advantages in terms of picking the most optimal number of different communities into which to divide the network, while the present algorithm takes the number of communities as an input. We ignore, however, the a priori knowledge that each community has 32 nodes.

Finally, we have tested the mean-field algorithm. As seen, this is almost as accurate as the belief propagation algorithm for $z_{in}\geq 8$, and actually performs better for $z_{in}\leq 7.5$. We believe that the improved performance is a result of the fact that the mean-field equations more easily break symmetry. Tests on networks with a lower coordination number showed the difference more clearly, as given in Fig. 2 for a network with four groups and $z_{in}+z_{out}=4$.

We have also tested the belief propagation algorithm on simple networks, such as dividing $N$ nodes arranged on a straight line with connections between nearest neighbors on the line into two different communities, as well as the Zachary karate club network. The relaxation of the algorithm needed to be done slightly more slowly than described above (ten randomly chosen nodes were relaxed before each edge was relaxed), and some care was taken on the constants due to the lower coordination number in order to obtain convergence: with poor choices the beliefs $p_i(r)$ oscillated randomly. After finding these constants on the straight line network, the algorithm was tested on the Zachary network, and identified the communities accurately, with one error of placing node $10$ in the wrong community, as labeled in the figure in . However, this node has one link to each of the two communities and so the network gives nodes no obvious preference for either community. Interestingly, after convergence of the equations, for almost all nodes $i$, the maximum over $r$ of $p_i(r)$ was greater than $0.99$, except node $9$ where it was only $0.98$ and node $10$ where it was only $0.87$; these two nodes have roughly equal connections to both communities.

*Discussion—* We have expressed community detection as an inference problem, providing a formulation of the problem in statistical mechanical terms. We have then applied the belief propagation method to solve the resulting statistical problem. The results are accurate, however there are a number of questions that should be addressed as well as possible extensions. First, there is some questions about picking the constants $J,J'$. In some cases, especially on test networks with a low coordination number, a poor choice of constants leads to either a lack of convergence of the belief propagation equations, or else convergence to a solution in which $p_i(r)\rightarrow 1/q$ so that the spontaneous symmetry breaking is absent. In both cases the algorithm performs poorly at finding the communities. The former case requires a slower relaxation of the equations, while the latter case requires an increase in the constants $J,J'$. Fortunately, both of these cases can be detected by looking at the $p_i(r)$ as the algorithm runs, and then corrected, so that the algorithm warns of its possible failure in these cases. We did not find any case in which the belief propagation equations converged to a poor solution which spontaneously broke symmetry.

The next question is the scaling of the algorithm with system size. Accurate results were found with $50000$ iterations. Since each iteration updates one directed edge, there are roughly $50000/(128*16)\approx 24$ iterations per edge. Even with $10000$ iterations, or roughly $5$ per edge, some information is found. For a general network, we expect that if there are few links between communities, then the number of iterations required per edge will be proportional to the phase-ordering time for a given community under the appropriate dynamics. This phase ordering time typically scales as some power of the relevant length scale for a community, and for many networks this length scale is of order $\log(N)$. We thus expect for a network with average coordination number $z$ that the time will typically be of order $zN\log(N)^{\alpha}$ for some $\alpha$.

There are a number of possible ways of modifying the algorithm, also. It may be desirable to incorporate additional a priori knowledge about the communities, or to study the modularity of the different divisions, as in . In some cases, the belief propagation equations may be reduced to linear programming. A final interesting question relates to our need for spontaneous symmetry breaking. It would be desirable to be able to go beyond belief propagation using methods such as . However, doing this may cause us to lose the symmetry breaking, and thus the MAP approximation may need to be replaced. Survey propagation may help in this regard, as it allows us to consider a distribution of magnetic fields for each site, each possibility corresponding to a different symmetry breaking. Even without these extensions, however, the present algorithm leads to useful results.

*Acknowledgements—* I thank M. Chertkov and E. Ben-Naim for useful discussions. This work was supported by US DOE W-7405-ENG-36.