abstract: This paper presents a novel approach for automatically generating image descriptions: visual detectors, language models, and multimodal similarity models learnt directly from a dataset of image captions. We use multiple instance learning to train visual detectors for words that commonly occur in captions, including many different parts of speech such as nouns, verbs, and adjectives. The word detector outputs serve as conditional inputs to a maximum-entropy language model. The language model learns from a set of over 400,000 image descriptions to capture the statistics of word usage. We capture global semantics by re-ranking caption candidates using sentence-level features and a deep multimodal similarity model. Our system is state-of-the-art on the official Microsoft COCO benchmark, producing a BLEU-4 score of 29.1%. When human judges compare the system captions to ones written by other people on our held-out test set, the system captions have equal or better quality 34% of the time.
author: Hao Fang[^1]Saurabh GuptaForrest IandolaRupesh K. Srivastava
Li DengPiotr Dollár[^2]Jianfeng GaoXiaodong He
Margaret MitchellJohn C. Platt[^3]C. Lawrence ZitnickGeoffrey Zweig

Microsoft Research
bibliography: biblio.bib
title: From Captions to Visual Concepts and Back

# Introduction

When does a machine "understand" an image? One definition is when it can generate a novel caption that summarizes the salient content within an image. This content may include objects that are present, their attributes, or their relations with each other. Determining the salient content requires not only knowing the contents of an image, but also deducing which aspects of the scene may be interesting or novel through commonsense knowledge .

This paper describes a novel approach for generating image captions from samples. We train our caption generator from a dataset of images and corresponding image descriptions. Previous approaches to generating image captions relied on object, attribute, and relation detectors learned from separate hand-labeled training data .

The direct use of captions in training has three distinct advantages. First, captions only contain information that is inherently salient. For example, a dog detector trained from images with captions containing the word `dog` will be biased towards detecting dogs that are salient and not those that are in the background. Image descriptions also contain variety of word types, including nouns, verbs, and adjectives. As a result, we can learn detectors for a wide variety of concepts. While some concepts, such as `riding` or `beautiful`, may be difficult to learn in the abstract, these terms may be highly correlated to specific visual patterns (such as a person on a horse or mountains at sunset).

Second, training a language model (LM) on image captions captures commonsense knowledge about a scene. A language model can learn that a person is more likely to sit on a chair than to stand on it. This information disambiguates noisy visual detections.

Third, by learning a joint multimodal representation on images and their captions, we are able to measure the global similarity between images and text, and select the most suitable description for the image.

An overview of our approach is shown in Figure . First, we use weakly-supervised learning to create detectors for a set of words commonly found in image captions. Learning directly from image captions is difficult, because the system does not have access to supervisory signals, such as object bounding boxes, that are found in other data sets . Many words, e.g., `crowded` or `inside`, do not even have well-defined bounding boxes. To overcome this difficulty, we use three ideas. First, the system reasons with image sub-regions rather than with the full image. Next, we featurize each of these regions using rich convolutional neural network (CNN) features, fine-tuned on our training data . Finally, we map the features of each region to words likely to be contained in the caption. We train this map using multiple instance learning (MIL) which learns discriminative visual signature for each word.

Generating novel image descriptions from a bag of likely words requires an effective LM. In this paper, we view caption generation as an optimization problem. In this view, the core task is to take the set of word detection scores, and find the highest likelihood sentence that covers each word exactly once. We train a maximum entropy (ME) LM from a set of training image descriptions . This training captures commonsense knowledge about the world through language statistics . An explicit search over word sequences is effective at finding high-likelihood sentences.

The final stage of the system (Figure ) re-ranks a set of high-likelihood sentences by a linear weighting of sentence features. These weights are learned using Minimum Error Rate Training (MERT) . In addition to several common sentence features, we introduce a new feature based on a Deep Multimodal Similarity Model (DMSM). The DMSM learns two neural networks that map images and text fragments to a common vector representation in which the similarity between sentences and images can be easily measured. As we demonstrate, the use of the DMSM significantly improves the selection of quality sentences.

To evaluate the quality of our automatic captions, we use three easily computable metrics and *better/worse/equal* comparisons by human subjects on Amazon's Mechanical Turk (AMT). The evaluation was performed on the challenging Microsoft COCO dataset containing complex images with multiple objects. Each of the 82,783 training images has 5 human annotated captions. For measuring the quality of our sentences we use the popular BLEU , METEOR and perplexity (PPLX) metrics. Surprisingly, we find our generated captions outperform humans based on the BLEU metric; and this effect holds when evaluated on unseen test data from the COCO dataset evaluation server, reaching 29.1% BLEU-4 vs. 21.7% for humans. Human evaluation on our held-out test set has our captions judged to be of the same quality or better than humans 34% of the time. We also compare to previous work on the PASCAL sentence dataset , and show marked improvements over previous work. Our results demonstrate the utility of training both visual detectors and LMs directly on image captions, as well as using a global multimodal semantic model for re-ranking the caption candidates.

# Related Work

There are two well-studied approaches to automatic image captioning: retrieval of existing human-written captions, and generation of novel captions. Recent retrieval-based approaches have used neural networks to map images and text into a common vector representation . Other retrieval based methods use similarity metrics that take pre-defined image features . Farhadi et al. represent both images and text as linguistically-motivated semantic triples, and compute similarity in that space. A similar fine-grained analysis of sentences and images has been done for retrieval in the context of neural networks .

Retrieval-based methods always return well-formed human-written captions, but these captions may not be able to describe new combinations of objects or novel scenes. This limitation has motivated a large body of work on generative approaches, where the image is first analyzed and objects are detected, and then a novel caption is generated. Previous work utilizes syntactic and semantic constraints in the generation process , and we compare against prior state of the art in this line of work. We focus on the Midge system , which combines syntactic structures using maximum likelihood estimation to generate novel sentences; and compare qualitatively against the Baby Talk system , which generates descriptions by filling sentence template slots with words selected from a conditional random field that predicts the most likely image labeling. Both of these previous systems use the same set of test sentences, making direct comparison possible.

Recently, researchers explored purely statistical approaches to guiding language models using images. Kiros et al.  use a log-bilinear model with bias features derived from the image to model text conditioned on the image. Also related are several contemporaneous papers . Among these, a common theme has been to utilize a recurrent neural network for generating images captions by conditioning its output on image features extracted by a convolutional neural network. More recently, Donahue et al. also applied a similar model to video description. Lebret et al. have investigated the use of a phrase-based model for generating captions, while Xu et al. have proposed a model based on visual attention.

Unlike these approaches, in this work we detect words by applying a CNN to image regions  and integrating the information with MIL . We minimize *a priori* assumptions about how sentences should be structured by training directly from captions. Finally, in contrast to , we formulate the problem of generation as an optimization problem and search for the most likely sentence .

# Word Detection

The first step in our caption generation pipeline detects a set of words that are likely to be part of the image's description. These words may belong to any part of speech, including nouns, verbs, and adjectives. We determine our vocabulary $\mathcal{V}$ using the 1000 most common words in the training captions, which cover over $92\%$ of the word occurrences in the training data (available on project webpage [^4]).

## Training Word Detectors

Given a vocabulary of words, our next goal is to detect the words from images. We cannot use standard supervised learning techniques for learning detectors, since we do not know the image bounding boxes corresponding to the words. In fact, many words relate to concepts for which bounding boxes may not be easily defined, such as `open` or `beautiful`. One possible approach is to use image classifiers that take as input the entire image. As we show in Section , this leads to worse performance since many words or concepts only apply to image sub-regions. Instead, we learn our detectors using the weakly-supervised approach of Multiple Instance Learning (MIL) .

For each word $w \in \mathcal{V}$, MIL takes as input sets of "positive" and "negative" bags of bounding boxes, where each bag corresponds to one image $i$. A bag $b_i$ is said to be positive if word $w$ is in image $i$'s description, and negative otherwise. Intuitively, MIL performs training by iteratively selecting instances within the positive bags, followed by retraining the detector using the updated positive labels.

We use a noisy-OR version of MIL , where the probability of bag $b_i$ containing word $w$ is calculated from the probabilities of individual instances in the bag: $$1 - \prod_{j \in b_i} \left(1-p^w_{ij}\right)$$ where $p^w_{ij}$ is the probability that a given image region $j$ in image $i$ corresponds to word $w$. We compute $p^w_{ij}$ using a multi-layered architecture [^5], by computing a logistic function on top of the `fc7` layer (this can be expressed as a fully connected `fc8` layer followed by a sigmoid layer): $$\frac{1}{1+\exp\left(-(\mathbf{{v}_w^t}\phi(b_{ij})+u_w)\right)},
 \label{eqn:sigmoid}$$ where $\phi(b_{ij})$ is the `fc7` representation for image region $j$ in image $i$, and $\mathbf{v_w}$, $u_w$ are the weights and bias associated with word $w$.

We express the fully connected layers (`fc6`, `fc7`, `fc8`) of these networks as convolutions to obtain a fully convolutional network. When this fully convolutional network is run over the image, we obtain a coarse spatial response map. Each location in this response map corresponds to the response obtained by applying the original CNN to overlapping shifted regions of the input image (thereby effectively scanning different locations in the image for possible objects). We up-sample the image to make the longer side to be $565$ pixels which gives us a $12\times12$ response map at `fc8` for both and corresponds to sliding a $224\times224$ bounding box in the up-sampled image with a stride of 32. The noisy-OR version of MIL is then implemented on top of this response map to generate a single probability $p^{w}_i$ for each word for each image. We use a cross entropy loss and optimize the CNN end-to-end for this task with stochastic gradient descent. We use one image in each batch and train for 3 epochs. For initialization, we use the network pre-trained on ImageNet .

## Generating Word Scores for a Test Image

Given a novel test image $i$, we up-sample and forward propagate the image through the CNN to obtain $p^{w}_{i}$ as described above. We do this for all words $w$ in the vocabulary $\mathcal{V}$. Note that all the word detectors have been trained independently and hence their outputs need to be calibrated. To calibrate the output of different detectors, we use the image level likelihood $p^w_i$ to compute precision on a held-out subset of the training data . We threshold this precision value at a global threshold $\tau$, and output all words $\tilde{\mathcal{V}}$ with a precision of $\tau$ or higher along with the image level probability $p^w_{i}$, and raw score $\max_{j}{p^{w}_{ij}}$.

Figure  shows some sample MIL detections. For each image, we visualize the spatial response map $p^{w}_{ij}$. Note that the method has not used any bounding box annotations for training, but is still able to reliably localize objects and also associate image regions with more abstract concepts.

# Language Generation

```latex
\begin{table*}\centering\footnotesize
\caption{Features used in the maximum entropy language model.}
\label{tab:features}\vspace{1mm}
\begin{tabular}{@{}l@{\hspace{4mm}}c@{\hspace{4mm}}c@{\hspace{4mm}}l}
\toprule
 Feature & Type & Definition & Description\\
\midrule
 Attribute & 0/1 & $\bar{w}_{l} \in \tilde{\mathcal{V}}_{l-1}$ &
 Predicted word is in the attribute set, i.e. has been visually detected and not yet used. \\
 N-gram+& 0/1 & $\bar{w}_{l-N+1},\cdots,\bar{w}_l = \kappa$ and $\bar{w}_{l} \in \tilde{\mathcal{V}}_{l-1}$ &
 N-gram ending in predicted word is $\kappa$ and the predicted word is in the attribute set. \\
 N-gram-& 0/1 & $\bar{w}_{l-N+1},\cdots,\bar{w}_l = \kappa$ and $\bar{w}_{l} \notin \tilde{\mathcal{V}}_{l-1}$ &
 N-gram ending in predicted word is $\kappa$ and the predicted word is not in the attribute set. \\
 End & 0/1 & $\bar{w}_l = \kappa$ and $\tilde{\mathcal{V}}_{l-1} = \emptyset$ &
 The predicted word is $\kappa$ and all attributes have been mentioned. \\
 Score & $\mathbb{R}$ & score($\bar{w}_{l}$) when $\bar{w}_{l} \in \tilde{\mathcal{V}}_{l-1}$ &
 The log-probability of the predicted word when it is in the attribute set. \\
\bottomrule
\end{tabular}
\vspace{-1em}
\end{table*}
```

We cast the generation process as a search for the likeliest sentence conditioned on the set of visually detected words. The language model is at the heart of this process because it defines the probability distribution over word sequences. Note that despite being a statistical model, the LM can encode very meaningful information, for instance that `running` is more likely to follow `horse` than `talking`. This information can help identify false word detections and encodes a form of commonsense knowledge.

## Statistical Model

To generate candidate captions for an image, we use a maximum entropy (ME) LM conditioned on the set of visually detected words. The ME LM estimates the probability of a word $w_l$ conditioned on the preceding words $w_1, w_2, \cdots, w_{l-1}$, as well as the set of words with high likelihood detections $\tilde{\mathcal{V}}_l \subset \tilde{\mathcal{V}}$ that have yet to be mentioned in the sentence. The motivation of conditioning on the unused words is to encourage all the words to be used, while avoiding repetitions. The top 15 most frequent closed-class words[^6] are removed from the set $\tilde{\mathcal{V}}$ since they are detected in nearly every image (and are trivially generated by the LM). It should be noted that the detected words are usually somewhat noisy. Thus, when the end of sentence token is being predicted, the set of remaining words may still contain some words with a high confidence of detection.

Following the definition of an ME LM , the word probability conditioned on preceding words and remaining objects can be written as:

$$\begin{aligned}
    &\Pr(w_l = \bar{w}_l | \bar{w}_{l-1}, \cdots, \bar{w}_1, \scalebox{0.85}{\ensuremath{<\!\!s\!\!>}},
    \tilde{\mathcal{V}}_{l-1}) =\nonumber \\
    & \frac{\exp \left[
        \sum_{k=1}^K \lambda_k f_k(\bar{w}_l, \bar{w}_{l-1}, \cdots, \bar{w}_1, \scalebox{0.85}{\ensuremath{<\!\!s\!\!>}},
        \tilde{\mathcal{V}}_{l-1})
    \right]}
    {\sum_{v \in \mathcal{V} \cup \scalebox{0.85}{\ensuremath{<\!\!/s\!\!>}}} \exp \left[
        \sum_{k=1}^K \lambda_k f_k(v, \bar{w}_{l-1}, \cdots, \bar{w}_1, \scalebox{0.85}{\ensuremath{<\!\!s\!\!>}},
        \tilde{\mathcal{V}}_{l-1})
    \right]}
    \label{eq:melm}
\end{aligned}$$where $\scalebox{0.85}{\ensuremath{<\!\!s\!\!>}}$ denotes the start-of-sentence token, $\bar{w}_j \in \mathcal{V} ~ \cup \scalebox{0.85}{\ensuremath{<\!\!/s\!\!>}}$, and $f_k(w_l, \cdots, w_1, \tilde{\mathcal{V}}_{l-1})$ and $\lambda_k$ respectively denote the $k$-th max-entropy feature and its weight. The basic discrete ME features we use are summarized in Table . These features form our "baseline" system. It has proven effective to extend this with a "score" feature, which evaluates to the log-likelihood of a word according to the corresponding visual detector. We have also experimented with distant bigram features and continuous space log-bilinear features , but while these improved PPLX significantly, they did not improve BLEU, METEOR or human preference, and space restrictions preclude further discussion.

To train the ME LM, the objective function is the log-likelihood of the captions conditioned on the corresponding set of detected objects, i.e.:

where the superscript $(s)$ denotes the index of sentences in the training data, and $\#{(s)}$ denotes the length of the sentence. The noise contrastive estimation (NCE) technique is used to accelerate the training by avoiding the calculation of the exact denominator in () . In the generation process, we use the unnormalized NCE likelihood estimates, which are far more efficient than the exact likelihoods, and produce very similar outputs. However, all PPLX numbers we report are computed with exhaustive normalization. The ME features are implemented in a hash table as in . In our experiments, we use N-gram features up to 4-gram and 15 contrastive samples in NCE training.

## Generation Process

During generation, we perform a left-to-right beam search similar to the one used in . This maintains a stack of length $l$ partial hypotheses. At each step in the search, every path on the stack is extended with a set of likely words, and the resulting length $l+1$ paths are stored. The top $k$ length $l+1$ paths are retained and the others pruned away.

We define the possible extensions to be the end of sentence token $\scalebox{0.85}{\ensuremath{<\!\!/s\!\!>}}$, the 100 most frequent words, the set of attribute words that remain to be mentioned, and all the words in the training data that have been observed to follow the last word in the hypothesis. Pruning is based on the likelihood of the partial path. When $\scalebox{0.85}{\ensuremath{<\!\!/s\!\!>}}$ is generated, the full path to $\scalebox{0.85}{\ensuremath{<\!\!/s\!\!>}}$ is removed from the stack and set aside as a completed sentence. The process continues until a maximum sentence length $L$ is reached.

After obtaining the set of completed sentences $\mathcal{C}$, we form an $M$-best list as follows. Given a target number of $T$ image attributes to be mentioned, the sequences in $\mathcal{C}$ covering at least $T$ objects are added to the $M$-best list, sorted in descending order by the log-likelihood. If there are less than $M$ sequences covering at least $T$ objects found in $\mathcal{C}$, we reduce $T$ by 1 until $M$ sequences are found.

# Sentence Re-Ranking

Our LM produces an $M$-best set of sentences. Our final stage uses MERT to re-rank the $M$ sentences. MERT uses a linear combination of features computed over an entire sentence, shown in Table . The MERT model is trained on the $M$-best lists for the validation set using the BLEU metric, and applied to the $M$-best lists for the test set. Finally, the best sequence after the re-ranking is selected as the caption of the image. Along with standard MERT features, we introduce a new multimodal semantic similarity model, discussed below.

|                                                                 |
|:----------------------------------------------------------------|
| 1\. The log-likelihood of the sequence.                         |
| 2\. The length of the sequence.                                 |
| 3\. The log-probability per word of the sequence.               |
| 4\. The logarithm of the sequence's rank in the log-likelihood. |
| 5\. 11 binary features indicating whether the number            |
| $\quad$ of mentioned objects is $x$ ($x=0,\ldots,10)$.          |
| 6\. The DMSM score between the sequence and the image.          |

Features used by MERT.

## Deep Multimodal Similarity Model

To model global similarity between images and text, we develop a Deep Multimodal Similarity Model (DMSM). The DMSM learns two neural networks that map images and text fragments to a common vector representation. We measure similarity between images and text by measuring cosine similarity between their corresponding vectors. This cosine similarity score is used by MERT to re-rank the sentences. The DMSM is closely related to the unimodal Deep Structured Semantic Model (DSSM) , but extends it to the multimodal setting. The DSSM was initially proposed to model the semantic relevance between textual search queries and documents, and is extended in this work to replace the query vector in the original DSSM by the image vector computed from the deep convolutional network.

The DMSM consists of a pair of neural networks, one for mapping each input modality to a common semantic space, which are trained jointly. In training, the data consists of a set of image/caption pairs. The loss function minimized during training represents the negative log posterior probability of the caption given the corresponding image.

**Image model**: We map images to semantic vectors using the same CNN (AlexNet / VGG) as used for detecting words in Section . We first finetune the networks on the COCO dataset for the full image classification task of predicting the words occurring in the image caption. We then extract out the `fc7` representation from the finetuned network and stack three additional fully connected layers with *tanh* non-linearities on top of this representation to obtain a final representation of the same size as the last layer of the text model. We learn the parameters in these additional fully connected layers during DMSM training.

**Text model**: The text part of the DMSM maps text fragments to semantic vectors, in the same manner as in the original DSSM. In general, the text fragments can be a full caption. Following we convert each word in the caption to a letter-trigram count vector, which uses the count distribution of context-dependent letters to represent a word. This representation has the advantage of reducing the size of the input layer while generalizing well to infrequent, unseen and incorrectly spelled words. Then following , this representation is forward propagated through a deep convolutional neural network to produce the semantic vector at the last layer.

**Objective and training**: We define the relevance $R$ as the cosine similarity between an image or query ($Q$) and a text fragment or document ($D$) based on their representations $y_Q$ and $y_D$ obtained using the image and text models: $R(Q, D) = \text{cosine}(y_Q, y_D) = ({y_Q}^Ty_D) / \Vert y_Q \Vert\Vert y_D \Vert$. For a given image-text pair, we can compute the posterior probability of the text being relevant to the image via:

$$P(D | Q) = \frac{\exp(\gamma R(Q,D))}{\Sigma_{D' \in \mathbb{D}}\exp(\gamma R(Q, D'))}$$

Here $\gamma$ is a smoothing factor determined using the validation set, which is 10 in our experiments. $\mathbb{D}$ denotes the set of all candidate documents (captions) which should be compared to the query (image). We found that restricting $\mathbb{D}$ to one matching document $D^+$ and a fixed number $N$ of randomly selected non-matching documents $D^-$ worked reasonably well, although using noise-contrastive estimation could further improve results. Thus, for each image we select one relevant text fragment and $N$ non-relevant fragments to compute the posterior probability. $N$ is set to 50 in our experiments. During training, we adjust the model parameters $\Lambda$ to minimize the negative log posterior probability that the relevant captions are matched to the images:

$$\label{eq:dmsm_objective}
L(\Lambda) = -\log\prod_{(Q, D^{+})}P(D^+|Q)$$

# Experimental Results

```latex
\begin{table*}\centering\small
\caption{Average precision (AP) and Precision at Human Recall (PHR) \cite{capeval2015} for words with different parts of speech (NN: Nouns, VB: Verbs, JJ: Adjectives, DT: Determiners, PRP: Pronouns, IN: Prepositions). Results are shown using a chance classifier, full image classification, and Noisy OR multiple instance learning with AlexNet \cite{krizhevskyNIPS12} and VGG \cite{simonyan14very} CNNs.}
\label{tab:ap}\vspace{2mm}
\scalebox{0.8}{
\begin{tabular}{llccccccccccccccccc}
                       & \multicolumn{8}{c}{Average Precision} && \multicolumn{8}{c}{Precision at Human Recall} \\ \cmidrule(r){2-9} \cmidrule(r){11-18}
                          & NN   & VB   & JJ   & DT   & PRP  & IN   & Others & All  && NN   & VB   & JJ   & DT   & PRP  & IN   & Others & All \\ \midrule
 Count                    & 616  & 176  & 119  & 10   & 11   & 38   & 30     & 1000 &&      &      &      &      &      &      &        & \\ \midrule
 Chance                   & 2.0  & 2.3  & 2.5  & 23.6 & 4.7  & 11.9 & 7.7    & 2.9  &&      &      &      &      &      &      &        & \\
 Classification (AlexNet) & 32.4 & 16.7 & 20.7 & 31.6 & 16.8 & 21.4 & 15.6   & 27.1 && 39.0 & 27.7 & 37.0 & 37.3 & 26.2 & 31.5 & 25.0   & 35.9 \\
 Classification (VGG)     & 37.0 & 19.4 & 22.5 & 32.9 & 19.4 & 22.5 & 16.9   & 30.8 && 45.3 & 31.0 & 37.1 & 40.2 & 29.6 & 33.9 & 25.5   & 40.6 \\
 MIL            (AlexNet) & 36.9 & 18.0 & 22.9 & 31.7 & 16.8 & 21.4 & 15.2   & 30.4 && 46.0 & 29.4 & 40.1 & 37.9 & 25.9 & 31.5 & 21.6   & 40.8 \\
 MIL            (VGG)     & 41.4 & 20.7 & 24.9 & 32.4 & 19.1 & 22.8 & 16.3   & 34.0 && 51.6 & 33.3 & 44.3 & 39.2 & 29.4 & 34.3 & 23.9   & 45.7 \\
 Human Agreement          &      &      &      &      &      &      &        &      && 63.8 & 35.0 & 35.9 & 43.1 & 32.5 & 34.3 & 31.6   & 52.8 \\
\bottomrule
\end{tabular}}
\end{table*}
```

We next describe the datasets used for testing, followed by an evaluation of our approach for word detection and experimental results on sentence generation.

## Datasets

Most of our results are reported on the Microsoft COCO dataset . The dataset contains 82,783 training images and 40,504 validation images. The images create a challenging testbed for image captioning since most images contain multiple objects and significant contextual information. The COCO dataset provides 5 human-annotated captions per image. The test annotations are not available, so we split the validation set into validation and test sets[^7].

For experimental comparison with prior papers, we also report results on the PASCAL sentence dataset , which contains 1000 images from the 2008 VOC Challenge , with 5 human captions each.

## Word Detection

To gain insight into our weakly-supervised approach for word detection using MIL, we measure its accuracy on the word classification task: If a word is used in at least one ground truth caption, it is included as a positive instance. Note that this is a challenging task, since conceptually similar words are classified separately; for example, the words cat/cats/kitten, or run/ran/running all correspond to different classes. Attempts at adding further supervision, e.g., in the form of lemmas, did not result in significant gains.

Average Precision (AP) and Precision at Human Recall (PHR) results for different parts of speech are shown in Table . We report two baselines. The first (Chance) is the result of randomly classifying each word. The second (Classification) is the result of a whole image classifier which uses features from AlexNet or VGG CNN . These features were fine-tuned for this word classification task using a logistic regression loss.

As shown in Table , the MIL NOR approach improves over both baselines for all parts of speech, demonstrating that better localization can help predict words. In fact, we observe the largest improvement for nouns and adjectives, which often correspond to concrete objects in an image sub-region. Results for both classification and MIL NOR are lower for parts of speech that may be less visually informative and difficult to detect, such as adjectives (e.g., `few`, which has an AP of 2.5), pronouns (e.g., `himself`, with an AP of 5.7), and prepositions (e.g., `before`, with an AP of 1.0). In comparison words with high AP scores are typically either visually informative (`red`: AP 66.4, `her`: AP 45.6) or associated with specific objects (`polar`: AP 94.6, `stuffed`: AP 74.2). Qualitative results demonstrating word localization are shown in Figures and .

## Caption Generation

```latex
\begin{table*}\centering\small
\caption{Caption generation performance for seven variants of our system on the Microsoft COCO dataset.   We report performance on our held out test set (half of the validation set). We report Perplexity (PPLX), BLEU and METEOR, using 4 randomly selected caption references. Results from human studies of subjective performance are also shown, with error bars in parentheses. Our final System ``VGG+Score+DMSM+ft'' is ``same or better'' than human 34\% of the time.}
\label{tab:mainresult}\vspace{1.5mm}
\begin{tabular}{lcccccccc}
\toprule
System & {\sc pplx} & {\sc bleu} & {\sc meteor}
 & $\approx$human & $>$human & $\ge$human \\
\midrule
1. Unconditioned                     & 24.1 & 1.2\%  &  6.8\%  \\
2. Shuffled Human                    & --   & 1.7\%  &  7.3\%  \\
3. Baseline                          & 20.9 & 16.9\% & 18.9\% &  9.9\% ($\pm$1.5\%) & 2.4\% ($\pm$0.8\%) & 12.3\% ($\pm$1.6\%) \\
4. Baseline+Score                    & 20.2 & 20.1\% & 20.5\% & 16.9\% ($\pm$2.0\%) & 3.9\% ($\pm$1.0\%) & 20.8\% ($\pm$2.2\%) \\
5. Baseline+Score+{\small DMSM}      & 20.2 & 21.1\% & 20.7\% & 18.7\% ($\pm$2.1\%) & 4.6\% ($\pm$1.1\%) & 23.3\% ($\pm$2.3\%) \\
6. Baseline+Score+{\small DMSM}+ft   & 19.2 & 23.3\% & 22.2\% & -- & -- & --\\
7. VGG+Score+ft                      & 18.1 & 23.6\% & 22.8\% & -- & -- & --\\
8. VGG+Score+{\small DMSM}+ft        & 18.1 & 25.7\% & 23.6\% & 26.2\% ($\pm$2.1\%) & 7.8\% ($\pm$1.3\%) & {\bf 34.0\%} ($\pm$2.5\%)\\
\midrule
Human-written captions               & --   & 19.3\% & 24.1\% \\
\bottomrule
\end{tabular}
\vspace{-1em}
\end{table*}
```

We next describe our caption generation results, beginning with a short discussion of evaluation metrics.

**Metrics:** The sentence generation process is measured using both automatic metrics and human studies. We use three different automatic metrics: PPLX, BLEU , and METEOR . PPLX (perplexity) measures the uncertainty of the language model, corresponding to how many bits on average would be needed to encode each word given the language model. A lower PPLX indicates a better score. BLEU is widely used in machine translation and measures the fraction of N-grams (up to 4-gram) that are in common between a hypothesis and a reference or set of references; here we compare against 4 randomly selected references. METEOR measures unigram precision and recall, extending exact word matches to include similar words based on WordNet synonyms and stemmed tokens. We additionally report performance on the metrics made available from the MSCOCO captioning challenge,[^8] which includes scores for BLEU-1 through BLEU-4, METEOR, CIDEr , and ROUGE-L .

All of these automatic metrics are known to only roughly correlate with human judgment . We therefore include human evaluation to further explore the quality of our models. Each task presents a human (Mechanical Turk worker) with an image and two captions: one is automatically generated, and the other is a human caption. The human is asked to select which caption better describes the image, or to choose a "same" option when they are of equal quality. In each experiment, 250 humans were asked to compare 20 caption pairs each, and 5 humans judged each caption pair. We used Crowdflower, which automatically filters out spammers. The ordering of the captions was randomized to avoid bias, and we included four check-cases where the answer was known and obvious; workers who missed any of these were excluded. The final judgment is the majority vote of the judgment of the 5 humans. In ties, one-half of a count is distributed to the two best answers. We also compute errors bars on the human results by taking 1000 bootstrap resamples of the majority vote outcome (with ties), then reporting the difference between the mean and the 5th or 95th percentile (whichever is farther from the mean).

**Generation results:** Table summarizes our results on the Microsoft COCO dataset. We provide several baselines for experimental comparison, including two baselines that measure the complexity of the dataset: Unconditioned, which generates sentences by sampling an $N$-gram LM without knowledge of the visual word detectors; and Shuffled Human, which randomly picks another human generated caption from another image. Both the BLEU and METEOR scores are very low for these approaches, demonstrating the variation and complexity of the Microsoft COCO dataset.

We provide results on seven variants of our end-to-end approach: Baseline is based on visual features from AlexNet and uses the ME LM with all the discrete features as described in Table . Baseline+Score adds the feature for the word detector score into the ME LM. Both of these versions use the same set of sentence features (excluding the DMSM score) described in Section  when re-ranking the captions using MERT. Baseline+Score+DMSM uses the same ME LM as Baseline+Score, but adds the DMSM score as a feature for re-ranking. Baseline+Score+DMSM+ft adds finetuning. VGG+Score+ft and VGG+Score+DMSM+ft are analogous to Baseline+Score and Baseline+Score+DMSM but use finetuned VGG features. Note: the AlexNet baselines without finetuning are from an early version of our system which used object proposals from instead of dense scanning.

As shown in Table , the PPLX of the ME LM with and without the word detector score feature is roughly the same. But, BLEU and METEOR improve with addition of the word detector scores in the ME LM. Performance improves further with addition of the DMSM scores in re-ranking. Surprisingly, the BLEU scores are actually above those produced by human generated captions (25.69% vs. 19.32%). Improvements in performance using the DMSM scores with the VGG model are statistically significant as measured by 4-gram overlap and METEOR per-image (Wilcoxon signed-rank test, p $<$ .001).

We also evaluated an approach (not shown) with whole-image classification rather than MIL. We found this approach to under-perform relative to MIL in the same setting (for example, using the VGG+Score+DMSM+ft setting, PPLX=18.9, BLEU=21.9%, METEOR=21.4%). This suggests that integrating information about words associated to image regions with MIL leads to improved performance over image classification alone.

The VGG+Score+DMSM approach produces captions that are judged to be of the same or better quality than human-written descriptions 34% of the time, which is a significant improvement over the Baseline results. Qualitative results are shown in Figure , and many more are available on the project website.

|  | cider | bleu-4 | bleu-1 | rouge-l | meteor |  |  |  |  |  |  |  |  |  |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| **\[5\]** | .912 (.854) | .291 (.217) | .695 (.663) | .519 (.484) | .247 (.252) |  |  |  |  |  |  |  |  |  |
| **\[40\]** | .925 (.910) | .567 (.471) | .880 (.880) | .662 (.626) | .331 (.335) |  |  |  |  |  |  |  |  |  |

Official COCO evaluation server results on test set (40,775 images). First row show results using 5 reference captions, second row, 40 references. Human results reported in parentheses.

**COCO evaluation server results:** We further generated the captions for the images in the actual COCO test set consisting of 40,775 images (human captions for these images are not available publicly), and evaluated them on the COCO evaluation server. These results are summarized in Table . Our system gives a BLEU-4 score of 29.1%, and equals or surpasses human performance on 12 of the 14 metrics reported – the only system to do so. These results are also state-of-the-art on all 14 reported metrics among the four other results available publicly at the time of writing this paper. In particular, our system is the only one exceeding human CIDEr scores, which has been specifically proposed for evaluating image captioning systems .

To enable direct comparison with previous work on automatic captioning, we also test on the PASCAL sentence dataset , using the 847 images tested for both the Midge and Baby Talk systems. We show significantly improved results over the Midge system, as measured by both BLEU and METEOR (2.0% vs. 17.6% BLEU and 9.2% vs. 19.2% METEOR).[^9] To give a basic sense of the progress quickly being made in this field, Figure shows output from the system on the same images.[^10]

# Conclusion

This paper presents a new system for generating novel captions from images. The system trains on images and corresponding captions, and learns to extract nouns, verbs, and adjectives from regions in the image. These detected words then guide a language model to generate text that reads well and includes the detected words. Finally, we use a global deep multimodal similarity model introduced in this paper to re-rank candidate captions

At the time of writing, our system is state-of-the-art on all 14 official metrics of the COCO image captioning task, and equal to or exceeding human performance on 12 out of the 14 official metrics. Our generated captions have been judged by humans (Mechanical Turk workers) to be equal to or better than human-written captions 34% of the time.

[^1]: H. Fang, S. Gupta, F. Iandola and R. K. Srivastava contributed equally to this work while doing internships at Microsoft Research. Current affiliations are H. Fang: University of Washington; S. Gupta and F. Iandola: University of California at Berkeley; R. K. Srivastava: IDSIA, USI-SUPSI.

[^2]: P. Dollár is currently at Facebook AI Research.

[^3]: J. Platt is currently at Google.

[^4]: <http://research.microsoft.com/image_captioning>

[^5]: We denote the CNN from as AlexNet and the 16-layer CNN from as VGG for subsequent discussion. We use the code base and models available from the Caffe Model Zoo <https://github.com/BVLC/caffe/wiki/Model-Zoo> .

[^6]: The top 15 frequent closed-class words are `a`, `on`, `of`, `the`, `in`, `with`, `and`, `is`, `to`, `an`, `at`, `are`, `next`, `that` and `it`.

[^7]: We split the COCO train/val set ito 82,729 train/20243 val/20244 test. Unless otherwise noted, test results are reported on the 20444 images from the validation set.

[^8]: <http://mscoco.org/dataset/#cap2015>

[^9]: Baby Talk generates long, multi-sentence captions, making comparison by BLEU/METEOR difficult; we thus exclude evaluation here.

[^10]: Images were selected visually, without viewing system captions.


abstract: Determining the quality of the results obtained by clustering techniques is a key issue in unsupervised machine learning. Many authors have discussed the desirable features of good clustering algorithms. However, Jon Kleinberg established an impossibility theorem for clustering. As a consequence, a wealth of studies have proposed techniques to evaluate the quality of clustering results depending on the characteristics of the clustering problem and the algorithmic technique employed to cluster data.
author:
;
bibliography: biblio.bib
title: Evaluation Metrics for Unsupervised Learning Algorithms

# Introduction

Machine learning techniques are usually classified into supervised and unsupervised techniques. Supervised machine learning starts from prior knowledge of the desired result in the form of labeled data sets, which allows to guide the training process, whereas unsupervised machine learning works directly on unlabeled data. In the absence of labels to orient the learning process, these labels must be "discovered" by the learning algorithm.
In this technical report, we discuss the desirable features of good clustering results, recall Kleinberg's impossibility theorem for clustering, and describe a taxonomy of evaluation criteria for unsupervised machine learning. We also survey many of the evaluation metrics that have been proposed in the literature. We end our report by describing the techniques that can be used to adjust the parameters of clustering algorithms, i.e. their hyperparameters.

# Formal Limitations of Clustering

From an intuitive point of view, the clustering problem has a very clear goal; namely, properly clustering a set of unlabeled data. Despite its intuitive appeal, the notion of "cluster" cannot be precisely defined, hence the wide range of clustering algorithms that have been proposed.

## Desirable Features of Clustering

Jon Kleinberg proposes three axioms that highlight the characteristics that a grouping problem should exhibit and can be considered "good", independently of the algorithm used to find the solution. These axioms are scale invariance, consistency, and wealth , which are explained in more detail below.
A grouping function is defined as a set of $S$ of $x \geq 2$ points and the distances between pairs of points. The set of points is $S=\{1,2,...,n\}$ and the distance between points is given by the distance function $d(i,j)$, where $i,j \in S$. The distance function $d$ measures the dissimilarity between pairs of points. For instance, the Euclidean, Manhattan, Chebyshev, and Mahalanobis distances can be used, among many others. Alternatively, a similarity function might also be used.

### Scale Invariance

The first of Kleinberg's axioms states that $f(d)=f(\alpha \cdot d)$ for any distance function $d$ and any scaling factor $\alpha>0$.
This simple axiom indicates that a clustering algorithm should not modify its results when all distances between points are scaled by the factor determined by a constant $\alpha$.

### Richness

A clustering process is considered to be rich when every partition of $S$ is a possible result of the clustering process. If the use $Range(f)$ to denote the set of all $\Gamma$ partitions so that $f(d)=\Gamma$ for some distance function $d$, then $Range(f)$ is equal to the set of all $S$ partitions.
This means that the the clustering function must be flexible enough to produce any arbitrary partition/clustering of the input data set.

### Consistency

Let $d$ and $d'$ be two distance functions. If, for every pair $(i,j)$ belonging to the same cluster, $d(i,j) \geq d'(i,j)$, and for every pair $(i,j)$ belonging to different clusters, $d(i,j) \leq d'(i,j)$, then $f(d)=f(d')$.
A clustering process is "consistent" when the clustering results do not change if the distances within clusters decrease and/or the distances between clusters increase.

## An Impossibility Theorem for Clustering

Given the above three axioms, Kleinberg proves the following theorem: For every $n>=2$, there is no clustering function $f$ that satisfies scale invariance, richness, and consistency.
Determining a "good" clustering is not a trivial problem. It is impossible for any clustering procedure to be able to satisfy all three axioms. Practical clustering algorithms must trade-off the desirable features of clustering results.
Since the three axioms cannot hold simultaneously, clustering algorithms can be designed to violate one of the axioms while sarisfying the other two. Kleinberg illustrates this point by describing three variants of single-link clustering (an agglomerative hierarchical clustering algorithm):

-   $k$-cluster stopping condition: Stop merging clusters when we have $k$ clusters (violates the richness axiom, since the algorithm would never return a number of clusters different to $k$).

-   Distance-$r$ stopping condition: Stop merging clusters when the nearest pair of clusters are farther than $r$ (violates scale invariance given that every cluster will contain a single instance when $\alpha$ is large, whereas a single cluster will contain all data when $\alpha \rightarrow 0$).

-   Scale-$\epsilon$ stopping condition: Stop merging clusters when the nearest pair of clusters are farther than a fraction $\epsilon$ of the maximum pairwise distance $\Delta$ (scale invariance is satisfied, yet consistency is violated).

Clustering algorithms can often satisfy the properties of scale invariance and consistency by relaxing their richness (e.g. whenever the number of clusters is established beforehand). As we have seen, some algorithms can even be customized to satisfy two out of three axioms by relaxing the third one (e.g. simple linkage with different stopping criteria).

# Methods for Cluster Evaluation

Evaluating the results of a clustering algorithm is a very important part of the process of clustering data. In supervised learning,"the evaluation of the resulting classification model is an integral part of the process of developing a classification model and there are well-accepted evaluation measures and procedures" . In unsupervised learning, because of its very nature, cluster evaluation, also known as cluster validation, is not as well-developed.
In clustering problems, it is not easy to determine the quality of a clustering algorithm. This gives rise to multiple evaluation techniques. Quite often, the evaluation process includes a notorious particularity: the way the measurement is performed depends on the algorithm used to obtain the clustering results.
When analyzing clustering results, several aspects must be taken into account for the validation of the algorithm results:

-   Determining the clustering tendency in the data (i.e. whether non-random structure really exists).

-   Determining the correct number of clusters.

-   Assessing the quality of the clustering results without external information.

-   Comparing the results obtained with external information.

-   Comparing two sets of clusters to determine which one is better.

The first three issues are addressed by **internal or unsupervised validation**, because there is no use of external information. The fourth issue is resolved by **external or supervised validation**. Finally, the last issue can be addressed by both supervised and unsupervised validation techniques..
Gan et al. propose a taxonomy of evaluation techniques that comprises both internal and external validation approaches (see Figure ).

## Null Hypothesis Testing

One of the desirable characteristics of a clustering process is to show whether data exhibits some tendency to form actual clusters. From a statistical point of view, a feasible approach consists of testing whether data exhibits random behavior or not . In this context, the null hypothesis testing can be used: A null hypothesis $H_0$ is assumed to be true until evidence suggests otherwise. In this case, the null hypothesis is the randomness of data and, when the null hypothesis is rejected, we assume that the data is significantly unlikely to be random. .
One of the difficulties of null hypothesis testing in this context is determining the statistical distribution under which the randomness hypothesis can be rejected. Jain and Dubes propose three alternatives :

-   Random plot hypothesis $H_0$: All proximity matrices of order $n \times n$ are equally likely.

-   Random label hypothesis $H_0$: All permutations of labels on $n$ objects are equally likely.

-   Random position hypothesis $H_0$: All sets of $n$ locations in some region of a $d$-dimensional space are equally likely.

Statistical techniques such as Monte Carlo analysis and bootstrapping can be used to determine the clustering tendency in data .

# Internal validation

Internal validation methods make it possible to establish the quality of the clustering structure without having access to external information (i.e. they are based on the information provided by data used as input to the clustering algorithm).
In general, two types of internal validation metrics can be combined: cohesion and separation measures. Cohesion evaluates how closely the elements of the same cluster are to each other, while separation measures quantify the level of separation between clusters (see Figure ). These measures are also known as internal indices because they are computed from the input data without any external information . Internal indices are usually employed in conjunction with two clustering algorithm families: hierarchical clustering algorithms and partitional algorithms..

Internal validation is used when there is no additional information available. In most cases, the particular metrics used by the evaluation methods are the same metrics that the clustering algorithm tries to optimize, which can be counterproductive in determining the quality of a clustering algorithm and deliver unfair validation results. On the other hand, in the absence of other sources of information, these metrics allow different algorithms to be compared under the same evaluation criterion , yet care must be taken not to report biased results.
Internal evaluation methods are commonly classified according to the type of clustering algorithm they are used with. For partitional algorithms, metrics based on the proximity matrix, as well as metrics of cohesion and separation, such as the silhouette coefficient, are often used. For hierarchical algorithms, the cophenetic coefficient is the most common (see Figure ).

## Partitional Methods

Several of the measures employed by internal cluster validations methods are based on the concepts of cohesion and separation (see Figure ). In general, the internal validation value of a set of $K$ clusters can be decomposed as the sum of the validation values for each cluster :
$$general~validity=\sum_{i=1}^K w_i~validity(C_i)\label{eq1}$$

This measure of validity can be cohesion, separation, or some combination of both. Quite often, the weights that appear in the previous expression correspond to cluster size.
The individual measures of cohesion and separation are defined as follows:
$$cohesion(C_i) =\sum_{x \in C_i, y \in C_i} proximity(x,y)\label{eq2}$$

$$separation(C_i, C_j) =\sum_{x \in C_i, y \in C_j} proximity(x,y)\label{eq3}$$

Cohesion is measured within a cluster (an intra-cluster metric), whereas separation is measured between clusters (an inter-cluster measure). Both are based on a proximity function that determines how similar a pair of examples are (similarity, dissimilarity and distance functions can be used). These metrics can also be defined for prototype-based clustering techniques, where proximity is measured from data examples to cluster centroids or medoids.
It should be noted that the cohesion metric defined above is equivalent to the cluster SSE \[Sum of Squared Errors\], also known as SSW \[Sum of Squared Errors Within Cluster\], when the proximity function is the squared Euclidean distance :
$$SSE(C_i) = \sum_{x \in C_i} d(c_i,x)^2 = \frac{1}{2m_i}\sum_{x \in C_i}\sum_{y \in C_i} d(x,y)^2\label{eq4}$$

where $x$ is an example in the cluster, $c_i$ is a cluster representative (e.g. its centroid) and $m_i$ is the number of examples in the cluster $C_i$.
When using the SSE metric, small values indicate a good cluster quality. Obviously, this metric is minimized in those clusters that were built from SSE-optimization-based algorithms such as k-means, but is clearly suboptimal for clusters detected using other techniques, such as density-based algorithms (e.g. DBSCAN) .
Likewise, we can maximize the distance between clusters using a separation metric. This approach leads to the between group sum of squares, or SSB :
$$SSB=\sum_{i=1}^K~m_i~ d(c_i,c)^2 = \frac{1}{2K} \sum_{i=1}^K \sum_{j=1}^K \frac{m}{K} d(c_i,c_j)^2\label{eq5}$$

where $c_i$ is the mean of the $i^{th}$ cluster and $c$ is the overall mean . Unlike the SSE metric, a good cluster quality is given by the high SSB values. As before, SSB is biased in favor of algorithms based on maximizing the separation distances among cluster centroids..
As mentioned above, a clustering is considered to be good when it has a high separation between clusters and a high cohesion within clusters . Instead of dealing with separate metrics for cohesion and separation, there are several metrics that try to quantify the level of separation and cohesion in a single measure :

-   The Calisnki-Harabasz coefficient, CH, also known as the variance ratio criterion, is a measure based on the internal dispersion of clusters and the dispersion between clusters. We would choose the number of clusters that maximizes the CH value for $M$ clusters :

    $$CH=\frac{\frac{SSB_M}{(M-1)}}{\frac{SSE_M}{(M)}}\label{eq6}$$

-   The Dunn index is the ratio of the smallest distance between data from different clusters and the largest distance between clusters. Again, this ratio should be maximized : $$D=\min_{1<i<k}\left\{\min_{1<j<k,i\neq j}\left\{\frac{\delta\left(C_i,C_j \right)}{\max_{1<l<k}\left \{ \Delta \left ( C_l \right ) \right \}}\right\}\right\} \label{eq7}$$

    $$\Delta(C_i)=\max_{x,y\in c_i}\{ d ( x , y)\} \label{eq8}$$

    $$\delta(C_i,C_j)=\min_{x\in C_i,y \in C_j}\{ d(x,y) \} \label{eq9}$$

-   The Xie-Beni score was designed for fuzzy clustering, but it can applied to hard clustering. As the previous coefficients, it is a ratio whose numerator estimates the level of compaction of the data within the same cluster and whose denominator estimates the level of separation of the data from different clusters :

    $$XB=\frac{\sum_{i=1}^{N}\sum_{k=1}^{M}u_{ik}^{2}\||x_i - C_k \||^2}{N_{t\neq s}\min\{\||C_t - C_s\||^2\}} \label{eq10}$$

-   The Ball-Hall index is a dispersion measure based on the quadratic distances of the cluster points with respect to their centroid :

    $$BH=\frac{SSE_M}{M} \label{eq11}$$

-   The Hartigan index is based on the logarithmic relationship between the sum of squares within the cluster and the sum of squares between clusters .

    $$H=\log \left ( \frac{SSB_M}{SSE_M} \right ) \label{eq12}$$

-   The Xu coefficient takes into account the dimensionality $D$ of the data, the number $N$ of data examples, and the sum of squared errors $SSE_M$ form $M$ clusters :

    $$Xu=D\log_2\left ( \sqrt{\frac{SSE_M}{DN^2}} \right )+\log M \label{eq13}$$

-   The silhouette coefficient is the most common way to combine the metrics of cohesion and separation in a single measure. Computing the silhouette coefficient at a particular point consists of the following three steps :

    1.  For each example, the average distance $a(i)$ to all the examples in the same cluster is computed:

        $$a(i)=\frac{1}{\left | C_a \right |}\sum_{j \in C_a , i \neq j}d\left ( i,j \right )$$

    2.  For each example, the minimum average distance $b(i)$ between the example and the examples contained in each cluster not containing the analyzed example:

        $$b(i)= \min_{C_b \neq C_a} \frac{1}{\left | C_b \right |}\sum_{j \in C_b}d\left ( i,j \right )$$

    3.  For each example, the silhouette coefficient is determined by the following expression:

        $$s(i)=\frac{b(i) - a(i)} {\max\{a(i),b(i)\}}$$

    The silhouette coefficient is defined in the interval $[-1,1]$ for each example in our data set. The global silhouette coefficient is just the average of the particular silhouette coefficients for each example:

    $$S = \frac{1}{n}\sum_{i=1}^{n}s(i)$$
    Unlike other combined measures, the silhouette coefficient provides us a simple framework for qualification. Positive values indicate a high separation between clusters. Negative values are an indication that the clusters are mixed with each other (i.e. an indication of overlapping clusters). When the silhouette coefficient is zero, it is an an indication that the data are uniformly distributed throughout the Euclidean space .
    Unfortunately, one of the main drawbacks of the silhouette coefficient is its high computational complexity, $O(dn^2)$, which makes it impractical when dealing with huge data sets .

Despite their widespread use, cohesion and separation metrics are not the only validation method available for partitional clustering techniques. In fact, cohesion and separation metrics do not perform well when it comes to analyzing results obtained by algorithms based on density analysis.
Given the proximity (or similarity) matrix of a data set and the clustering obtained by a clustering algorithm, we can compare the actual proximity matrix to an ideal version of the proximity matrix based on the provided clustering. Reordering rows and columns so that all examples of the same cluster appear together, the ideal proximity matrix has a block diagonal structure. High correlation between the actual and ideal proximity matrices indicates that examples in the same cluster are close to each other, although it might not be a good measure for density-based clusters .
Unfortunately, the mere construction of the whole proximity matrix is computationally expensive, $O(n^2)$, and this validation method cannot be used without sampling for large data sets.

## Hierarchical Methods

The clustering validation methods discussed in the previous section were devised for partitional clustering algorithms. Several internal validation techniques have also been proposed and tested with hierarchical clustering algorithms. As you can expect, these evaluation metrics obtain better results when using hierarchical algorithms such as the single link agglomerative clustering algorithm, SLINK .

### Cophenetic Correlation Coefficient (CPCC)

The cophenetic distance between two examples is the proximity at which an agglomerative hierarchical clustering algorithm puts the examples in the same cluster for the first time . Looking at the associated dendrogram, it corresponds to the height at which the branches corresponding to the two examples are merged.
The cophenetic correlation coefficient (CPCC) is a metric used to evaluate the results of a hierarchical clustering algorithm . This correlation coefficient was proposed by Sokal and Rohlf in 1962 as the correlation between the entries of the cophenetic matrix $P_c$, containing cophenetic distances, and the proximity matrix $P$, containing similarities.
The cophenetic matrix $P_c$ defined for pairs of examples $P_c(i,j)$ as the level of proximity between the examples $(i, j)$ in the dendrogram (i.e. the level of proximity at which both examples are assigned to the same cluster). The cophenetic correlation coefficient is then defined as

$$CPCC=\frac{\sum_{i<j}(d_{ij}-\bar{d})(d_{ij}^{*}-\bar{d^{*}})}{\sqrt{\sum_{i<j}(d_{ij}-\bar{d})^2\sum_{i<j}(d_{ij}^{*}-\bar{d^{*}})^2}}$$
where $d_{ij}$ is the distance between the example pair $(i, j)$ and $d_{ij}^{*}$ is their cophenetic distance. The correlation coefficient also includes the average $\bar{d}$ of the distances in the proximity matrix and the average $\bar{d^{*}}$ of the cophenetic distances in the cophenetic matrix, which can be computed as follows:

$$\bar{d} = \frac{ \sum_{i<j}d_{ij} }{ 2\left(n^2-n\right) }$$

$$\bar{d^{*}} = \sqrt{\frac{\sum_{i<j}\left(d_{ij}-d_{ij}^{*}\right)^2}{\sum_{i<j}(d_{ij}^{*})^2}}$$
The cophenetic correlation coefficient, as the silhouette coefficient and any other correlation coefficient, is a value in the interval $[-1,1]$. High CPCC values indicate a high level of similarity between the two matrices , an indication that the clustering algorithm has been able to identify the underlying structure of its input data.

### Hubert Statistic

The Hubert statistic is similar to the cophenetic correlation coefficient. First, concordance are discordance are defined for pairs of examples.
A pair $(i,j)$ is concordant when $\left(\left(v_{p_i}<v_{c_i}\right) \& \left(v_{p_j}<v_{c_j}\right)\right)$ or $\left(\left(v_{p_i}>v_{c_i}\right) \& \left(v_{p_j}>v_{c_j}\right)\right)$. Likewise, a pair $(i,j)$ is said to be discordant when $\left(\left(v_{p_i}<v_{c_i}\right) \& \left(v_{p_j}>v_{c_j}\right)\right)$ or $\left(\left(v_{p_i}>v_{c_i}\right) \& \left(v_{p_j}<v_{c_j}\right)\right)$. Therefore, a pair is neither concordant nor discordant if $v_{p_i}=v_{c_i}$ or $v_{p_j}=v_{c_j}$.
Let $S_{+}$ and $S_{-}$ be the number of concordant and discordant pairs, respectively. Then, the Hubert coefficient is defined as :

$$\gamma = \frac{S_{+} - S_{-}}{S_{+} + S_{-}}\
$$

As the cophenetic coefficient, the Hubert statistic is between -1 and 1. Like CPCC, it has been mainly used to compare the results of two hierarchical clustering algorithms. A higher Hubert $\gamma$ value corresponds to a better clustering of data.

# External Validation

External validation methods can be associated to a supervised learning problem. External validation proceeds by incorporating additional information in the clustering validation process, i.e. external class labels for the training examples. Since unsupervised learning techniques are primarily used when such information is not available, external validation methods are not used on most clustering problems. However, they can still be applied when external information is available and also when you generate synthetic data from a real data set.
Like internal validation methods, it is also possible to classify external metrics depending on the algorithmic approach of the clustering technique used to solve a particular clustering problem. A more rational classification of external validation methods is shown in Figure . According to this taxonomy, different external validation metrics can be used to compare two sets of clusters, the first one obtained by the clustering algorithm under evaluation and the second one provided by an external source.

We want to compare the result of a clustering algorithm $C=\{C_1,C_2,...,C_m\}$ to a potentially different partition of data $P=\{P_1,P_2,...,P_s\}$, which might represent the expert knowledge of the analyst (his experience or intuition), prior knowledge of the data in the form of class labels, the results obtained by another clustering algorithm, or simply a grouping considered to be "correct" .
In order to carry out this analysis, a contingency matrix must be built to evaluate the clusters detected by the algorithm. This contingency matrix contains four terms:

-   $TP$: The number of data pairs found in the same cluster, both in $C$ and in $P$.

-   $FP$: The number of data pairs found in the same cluster in $C$ but in different clusters in $P$.

-   $FN$: The number of data pairs found in different clusters in $C$ but in the same cluster in $P$.

-   $TN$: The number of data pairs found in different clusters, both in $C$ and in $P$.

From these four indicators, we can easily obtain:

-   The number of pairs found in the same cluster in $C$: $m_1=TP+FP$.

-   The number of pairs found in the same cluster in $P$: $m_2=TP+FN$.

Obviously, the total number of pairs must be

$$M = TP+FP+FN+TN = \frac{n \left( n-1 \right) }{2}$$

## Matching Sets

The first family of external validation methods that can be used to compare two partitions of data consists of those method that identify the relationship between each cluster detected in $C$ and its natural correspondence to the classes in the reference result defined by $P$.
Several measures can be defined to measure the similarity between the clusters in $C$, obtained by the clustering algorithm, and the clusters if $P$, corresponding to our prior (external) knowledge :

-   **Precision** counts the true positives, how many examples are properly classified within the same cluster : $$Pr=\frac{TP}{TP+FP}=\frac{TP}{P}=\frac{p_{ij}}{p_i}$$

-   **Recall** evaluates the percentage of elements that are properly included in the same cluster: $$R=\frac{TP}{TP+FN}=\frac{p_{ij}}{p_j}$$

-   The **F-measure** combines precision and recall in a single metric, their weighted harmonic mean: $$F_{\alpha}=\frac{1+\alpha}{\frac{1}{Pr}+\frac{\alpha}{R}}$$
    Quite often, precision and recall are evenly combined with an unweighted harmonic mean ($\alpha=1$): $$F=\frac{2*Pr*R}{Pr+R} =\sum_{j}p_j \max_i \left[\frac{2*Pr*R}{Pr+R}\right]$$

-   **Purity** evaluates whether each cluster contains only examples from the same class: $$U = \sum_{i} p_i \left( \max_j \frac{p_{ij}}{p_i} \right)$$

In the expressions above, $p_i=n_i/n$, $p_j=n_j/n$, and $p_{ij}=n_{ij}/n$, where $n_{ij}$ is the number of examples belonging to the class $i$ found in the cluster $j$ and $n_i$ ($n_j$) is the number of examples in the cluster $i$ ($j$).

## Peer-to-peer Correlation

A second family of measures for external validation are based on the correlation between pairs, i.e. they seek to measure the similarity between two partitions under equal conditions, such as the result of a grouping process for the same set, but by means of two different methods $C$ and $P$. It is assumed that the examples that are in the same cluster in $C$ should be in the same class in $P$, and vice versa .
Some metrics based on measuring the correlation between pairs are the following:

-   The **Jaccard coefficient** assesses the similarity of the detected clusters $C$ to the provided partition $P$: $$\begin{aligned}
            J & = \frac{TP}{TP+FP+FN} \\
            J & =\frac{\sum_{ij}\binom{n_{ij}}{2}}{ \sum_i\binom{n_i}{2}+\sum_{j}\binom{n_j}{2}-\sum_{ij}\binom{n_{ij}}{2}}

    \end{aligned}$$

-   The **Rand coefficient** is similar to the Jaccard coefficient, yet it is measured against the total data set (equivalent to accuracy in a supervised machine learning setting): $$\begin{aligned}
            Rand & = \frac{TP+TN}{M} \\
            Rand & = \frac{ \binom{n}{2}-\sum_i\binom{n_i}{2}+\sum_{j}\binom{n_j}{2}-\sum_{ij}\binom{n_{ij}}{2} }{\binom{n}{2}}

    \end{aligned}$$

-   The **Folkes and Mallows coefficient** computes the similarity between the clusters found by the algorithm with respect to the independent markers:

    $$\begin{aligned}
            FM & = \sqrt{\frac{TP}{TP+FP}*\frac{TP}{TP+FN}} \\
            FM & = =\frac{\sum_{ij}\binom{n_{ij}}{2}}{\sqrt{\sum_i\binom{n_i}{2}*\sum_{j}\binom{n_j}{2}}}

    \end{aligned}$$

-   We can also resort to the **Hubert statistical coefficient** in this context:

    $$\begin{aligned}
        \Gamma & = \frac{1}{M}\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}X_{ij}Y_{ij} \\
        \Gamma & = \frac{ \binom{n}{2}\sum_{ij}\binom{n_{ij}}{2}-\sum_{i}\binom{n_i}{2}\sum_{j}\binom{n_j}{2} } { \sqrt{\sum_{i}\binom{n_i}{2}\sum_{j} \binom{n_j}{2} \left[\binom{n}{2}-\sum_{i}\binom{n_i}{2}\right] \left[\binom{n}{2}-\sum_{j}\binom{n_j}{2}\right]} }

    \end{aligned}$$

As before, $n_{ij}$ is the number of examples belonging to the class $i$ found in the cluster $j$, whereas $n_j$ ($n_j$) is the number of examples in the cluster $i$ ($j$).

## Measures Based on Information Theory

A third family of external cluster validation metrics is based on Information Theory concepts, such as the existing uncertainty in the prediction of the natural classes provided by the partition $P$. This family includes basic measures such as entropy and mutual information, as well as their respective normalized variants.

-   **Entropy** is a reciprocal measure of purity that allows us to measure the degree of disorder in the clustering results: $$H = -\sum_{i}p_i\left(\sum_{j}\frac{p_{ij}}{p_i}\log{\frac{p_{ij}}{p_i}}\right)$$

-   **Mutual information** allows us to measure the the reduction in uncertainty about the clustering results given knowledge of the prior partition: $$MI = \sum_{i}\sum_{j} p_{ij} \log\frac{p_{ij}}{p_ip_j}$$

As always, $p_{ij}=n_{ij}/n$, $p_i=n_i/n$, and $p_j=n_j/n$.

# Hyperparameter Tuning

Internal and external validation metrics are used once the clustering algorithm has been applied to the available data set. However, the clustering algorithm itself has its own parameters. Adjusting those parameters, also known as hyperparameters in the machine learning literature, can help us obtain very different clustering results.
When using unsupervised machine learning techniques, several issues affect their effectiveness. Even though external validation metrics can help us evaluate whether the obtained clusters closely match the underlying categories in the training data, which the clustering algorithm tries to identify without externally-provided class labels, those metrics cannot address other issues such as the right number of clusters for our current data set. For instance, in the case of hierarchical clustering techniques, we are certainly interested in determining the best level at which we can cut our dendrogram.
Any clustering algorithms has a set of parameters $P_{alg}$, which might include the number of clusters $n_c$ or not. Hyperparameter tuning tries to determine, for the different possible values of the parameters in $P_{alg}$, which set of parameter values is the most suitable for our particular clustering problem.
We could proceed in the following way :

-   When the algorithm does not include the number of clusters $n_c$ among its parameters ($n_c \notin P_{alg}$), we run the algorithm with different values for its parameters so that we can determine their largest range for which $n_c$ remains constant. Later, we choose as parameter values the values in the middle of this range.

-   When the algorithm parameters $P_{alg}$ include the desired number of clusters $n_c$ ($n_c \in P_{alg}$), we run the algorithm for a range of values for $n_c$ between $n_{cmin}$ and $n_{cmax}$. For each value of $n_c$, we run the algorithm multiple times using different sets of values (i.e. starting from different initial conditions) and choose the value that optimizes our desired validation metric, which might be internal or external depending on our particular clustering problem.

When we just want to determine the "right" number of clusters, $n_c$, plotting the validation results for different values of $n_c$ can sometimes show a relevant change in the validation metric, commonly referred to as a "knee" or "elbow" .
However, in practice, the number of parameters in $P_{alg}$ might be large, so we cannot test every possible value combination of parameter values in a systematic way. Hyperparameter tuning can then be seen as a combinatorial optimization problem. Fortunately, we can resort to automated tuning strategies that facilitate our search . Among the tuning strategies at our disposal, we could include the following ones :

-   **Grid search** is based on a systematic exploration of the hyperparameter space, on which we define a grid and test for a parameter combination in each cell of such grid. If we have $p$ parameters, the systematic exploration of a $p$-dimensional space might require an exponential number of parameter configurations, often unfeasible in practice.

-   **Random search** chooses parameter configurations at random. Even though our search is not exhaustive, when using random search, we hope that some parameter combinations will lead us to promising regions in our search space. The rationale behind random search is that, quite often, local changes in the parameter values do not produce significant changes in the algorithm output, so that a systematic exploration might not be really necessary, even when feasible (usually, it is not feasible anyway).

-   **Smart search** techniques try to optimize the problem of searching for hyperparameter values. Different strategies can be implemented, such as Bayesian optimization using Gaussian processes and evolutionary optimization using genetic algorithms or evolution strategies.

# Conclusion

Determining the quality of the results provided by a clustering algorithm is not an easy problem. Kleinberg defined three properties any clustering algorithm should try to satisfy (the axioms of scale invariance, richness, and consistency) and proved an impossibility theorem that shows that no clustering algorithm can simultaneously satisfy all of them.
A wide range of metrics have been proposed in the literature to quantify the quality of clustering results:

-   Internal validation metrics do not require external information. These metrics focus on measuring cluster cohesion and separation, on the statistical analysis of the proximity matrix, or on the study of the dendrogram generated by hierarchical clustering algorithms.

-   External validation metrics resort to externally-provided information to evaluate the quality of the clustering results. A large number of external validation metrics are at our disposal, ranging from matching sets to peer-to-peer correlation and information theoretical indices.

External validation metrics are also useful when comparing the results provided by different clustering algorithms (or the same algorithm with different sets of parameter values).
Finding the best configuration for the parameters of an algorithm is known as hyperparameter tuning. This process is often necessary, for instance, to determine the optimal number of clusters for a particular clustering problem.