abstract: Automatic synthesis of realistic images from text would be interesting and useful, but current AI systems are still far from this goal. However, in recent years generic and powerful recurrent neural network architectures have been developed to learn discriminative text feature representations. Meanwhile, deep convolutional generative adversarial networks (GANs) have begun to generate highly compelling images of specific categories, such as faces, album covers, and room interiors. In this work, we develop a novel deep architecture and GAN formulation to effectively bridge these advances in text and image modeling, translating visual concepts from characters to pixels. We demonstrate the capability of our model to generate plausible images of birds and flowers from detailed text descriptions.
bibliography: references.bib

# Introduction

In this work we are interested in translating text in the form of single-sentence human-written descriptions directly into image pixels. For example, "this small bird has a short, pointy orange beak and white belly" or "the petals of this flower are pink and the anther are yellow". The problem of generating images from visual descriptions gained interest in the research community, but it is far from being solved.

Traditionally this type of detailed visual information about an object has been captured in attribute representations - distinguishing characteristics the object category encoded into a vector , in particular to enable zero-shot visual recognition , and recently for conditional image generation .

While the discriminative power and strong generalization properties of attribute representations are attractive, attributes are also cumbersome to obtain as they may require domain-specific knowledge. In comparison, natural language offers a general and flexible interface for describing objects in any space of visual categories. Ideally, we could have the generality of text descriptions with the discriminative power of attributes.

Recently, deep convolutional and recurrent networks for text have yielded highly discriminative and generalizable (in the zero-shot learning sense) text representations learned automatically from words and characters . These approaches exceed the previous state-of-the-art using attributes for zero-shot visual recognition on the Caltech-UCSD birds database , and also are capable of zero-shot caption-based retrieval. Motivated by these works, we aim to learn a mapping directly from words and characters to image pixels.

To solve this challenging problem requires solving two sub-problems: first, learn a text feature representation that captures the important visual details; and second, use these features to synthesize a compelling image that a human might mistake for real. Fortunately, deep learning has enabled enormous progress in both subproblems - natural language representation and image synthesis - in the previous several years, and we build on this for our current task.

However, one difficult remaining issue not solved by deep learning alone is that the distribution of images conditioned on a text description is highly multimodal, in the sense that there are very many plausible configurations of pixels that correctly illustrate the description. The reverse direction (image to text) also suffers from this problem but learning is made practical by the fact that the word or character sequence can be decomposed sequentially according to the chain rule; i.e. one trains the model to predict the next token conditioned on the image and all previous tokens, which is a more well-defined prediction problem.

This conditional multi-modality is thus a very natural application for generative adversarial networks , in which the generator network is optimized to fool the adversarially-trained discriminator into predicting that synthetic images are real. By conditioning both generator and discriminator on side information (also studied by  and ), we can naturally model this phenomenon since the discriminator network acts as a "smart" adaptive loss function.

Our main contribution in this work is to develop a simple and effective GAN architecture and training strategy that enables compelling text to image synthesis of bird and flower images from human-written descriptions. We mainly use the Caltech-UCSD Birds dataset and the Oxford-102 Flowers dataset along with five text descriptions per image we collected as our evaluation setting. Our model is trained on a subset of training categories, and we demonstrate its performance both on the training set categories and on the testing set, i.e. "zero-shot" text to image synthesis. In addition to birds and flowers, we apply our model to more general images and text descriptions in the MS COCO dataset .

# Related work

Key challenges in multimodal learning include learning a shared representation across modalities, and to predict missing data (e.g. by retrieval or synthesis) in one modality conditioned on another.   trained a stacked multimodal autoencoder on audio and video signals and were able to learn a shared modality-invariant representation.   developed a deep Boltzmann machine and jointly modeled images and text tags.   proposed a multimodal conditional prediction framework (hallucinating one modality given the other) and provided theoretical justification.

Many researchers have recently exploited the capability of deep convolutional *decoder* networks to generate realistic images.   trained a deconvolutional network (several layers of convolution and upsampling) to generate 3D chair renderings conditioned on a set of graphics codes indicating shape, position and lighting.   added an encoder network as well as actions to this approach. They trained a recurrent convolutional encoder-decoder that rotated 3D chair models and human faces conditioned on action sequences of rotations.   encode transformations from analogy pairs, and use a convolutional decoder to predict visual analogies on shapes, video game characters and 3D cars.

Generative adversarial networks  have also benefited from convolutional decoder networks, for the generator network module.   used a Laplacian pyramid of adversarial generator and discriminators to synthesize images at multiple resolutions. This work generated compelling high-resolution images and could also condition on class labels for controllable generation.   used a standard convolutional decoder, but developed a highly effective and stable architecture incorporating batch normalization to achieve striking image synthesis results.

The main distinction of our work from the conditional GANs described above is that our model conditions on *text descriptions* instead of class labels. To our knowledge it is the first end-to-end differentiable architecture from the character level to pixel level. Furthermore, we introduce a manifold interpolation regularizer for the GAN generator that significantly improves the quality of generated samples, including on held out zero shot categories on CUB.

The bulk of previous work on multimodal learning from images and text uses retrieval as the target task, i.e. fetch relevant images given a text query or vice versa. However, in the past year, there has been a breakthrough in using recurrent neural network decoders to generate text descriptions conditioned on images . These typically condition a Long Short-Term Memory  on the top-layer features of a deep convolutional network to generate captions using the MS COCO  and other captioned image datasets.   incorporated a recurrent visual attention mechanism for improved results.

Other tasks besides conditional generation have been considered in recent work.   generate answers to questions about the visual content of images. This approach was extended to incorporate an explicit knowledge base . applied sequence models to both text (in the form of books) and movies to perform a joint alignment.

In contemporary work  generated images from text captions, using a variational recurrent autoencoder with attention to paint the image in multiple steps, similar to DRAW . Impressively, the model can perform reasonable synthesis of completely novel (unlikely for a human to write) text such as "a stop sign is flying in blue skies", suggesting that it does not simply memorize. While the results are encouraging, the problem is highly challenging and the generated images are not yet realistic, i.e., mistakeable for real. Our model can in many cases generate visually-plausible $64 \times 64$ images conditioned on text, and is also distinct in that our entire model is a GAN, rather only using GAN for post-processing.

Building on ideas from these many previous works, we develop a simple and effective approach for text-based image synthesis using a character-level text encoder and class-conditional GAN. We propose a novel architecture and learning strategy that leads to compelling visual results. We focus on the case of fine-grained image datasets, for which we use the recently collected descriptions for Caltech-UCSD Birds and Oxford Flowers with 5 human-generated captions per image . We train and test on class-disjoint sets, so that test performance can give a strong indication of generalization ability which we also demonstrate on MS COCO images with multiple objects and various backgrounds.

# Background

In this section we briefly describe several previous works that our method is built upon.

## Generative adversarial networks

Generative adversarial networks (GANs) consist of a generator $G$ and a discriminator $D$ that compete in a two-player minimax game: The discriminator tries to distinguish real training data from synthetic images, and the generator tries to fool the discriminator. Concretely, $D$ and $G$ play the following game on V(D,G): $$\begin{aligned}
\underset{G}{\text{ min }} \underset{D}{\text{ max }} V( D, G) =& \text{ }\mathbb{E}_{ x \sim p_{data}(x)}[\log D(x)] + \\
& \text{ }\mathbb{E}_{ x \sim p_z(z)}[\log(1-D(G(z)))] \nonumber
\end{aligned}$$ prove that this minimax game has a global optimium precisely when $p_g = p_{data}$, and that under mild conditions (e.g. $G$ and $D$ have enough capacity) $p_g$ converges to $p_{data}$. In practice, in the start of training samples from $D$ are extremely poor and rejected by $D$ with high confidence. It has been found to work better in practice for the generator to maximize $\log(D(G(z)))$ instead of minimizing $\log(1-D(G(z)))$.

## Deep symmetric structured joint embedding

To obtain a visually-discriminative vector representation of text descriptions, we follow the approach of  by using deep convolutional and recurrent text encoders that learn a correspondence function with images. The text classifier induced by the learned correspondence function $f_t$ is trained by optimizing the following structured loss: $$\begin{aligned}
\label{eq:objective}
\dfrac{1}{N}\sum_{n=1}^{N}\Delta(y_{n},f_v(v_n)) + \Delta(y_{n},f_t(t_n))
\end{aligned}$$ where $\{(v_{n},t_{n},y_{n}): n = 1, ..., N\}$ is the training data set, $\Delta$ is the 0-1 loss, $v_n$ are the images, $t_n$ are the corresponding text descriptions, and $y_n$ are the class labels. Classifiers $f_v$ and $f_t$ are parametrized as follows: $$\begin{aligned}
\label{eq:classifiers}
f_v(v) = \underset{y \in \mathcal{Y}}{\text{arg max }} \mathbb{E}_{t \sim \mathcal{T}(y)}[\phi(v)^T\varphi(t))]\\
f_t(t) = \underset{y \in \mathcal{Y}}{\text{arg max }} \mathbb{E}_{v \sim \mathcal{V}(y)}[\phi(v)^T\varphi(t))]
\end{aligned}$$ where $\phi$ is the image encoder (e.g. a deep convolutional neural network), $\varphi$ is the text encoder (e.g. a character-level CNN or LSTM), $\mathcal{T}(y)$ is the set of text descriptions of class $y$ and likewise $\mathcal{V}(y)$ for images. The intuition here is that a text encoding should have a higher compatibility score with images of the correspondong class compared to any other class and vice-versa.

To train the model a surrogate objective related to is minimized (see  for details). The resulting gradients are backpropagated through $\varphi$ to learn a discriminative text encoder.   found that different text encoders worked better for CUB versus Flowers, but for full generality and robustness to typos and large vocabulary, in this work we always used a hybrid character-level convolutional-recurrent network.

# Method

Our approach is to train a deep convolutional generative adversarial network (DC-GAN) conditioned on text features encoded by a hybrid character-level convolutional-recurrent neural network. Both the generator network $G$ and the discriminator network $D$ perform feed-forward inference conditioned on the text feature.

## Network architecture

We use the following notation. The generator network is denoted $G : \mathbb{R}^{Z} \times \mathbb{R}^{T} \rightarrow \mathbb{R}^{D}$, the discriminator as $D : \mathbb{R}^{D} \times \mathbb{R}^{T} \rightarrow \{0,1\}$, where $T$ is the dimension of the text description embedding, $D$ is the dimension of the image, and $Z$ is the dimension of the noise input to $G$. We illustrate our network architecture in .

In the generator $G$, first we sample from the noise prior $z \in \mathbb{R}^{Z} \sim \mathcal{N}(0,1)$ and we encode the text query $t$ using text encoder $\varphi$. The description embedding $\varphi(t)$ is first compressed using a fully-connected layer to a small dimension (in practice we used 128) followed by leaky-ReLU and then concatenated to the noise vector $z$. Following this, inference proceeds as in a normal deconvolutional network: we feed-forward it through the generator $G$; a synthetic image $\hat{x}$ is generated via $\hat{x} \leftarrow G(z, \varphi(t))$. Image generation corresponds to feed-forward inference in the generator $G$ conditioned on query text and a noise sample.

In the discriminator $D$, we perform several layers of stride-2 convolution with spatial batch normalization  followed by leaky ReLU. We again reduce the dimensionality of the description embedding $\varphi(t)$ in a (separate) fully-connected layer followed by rectification. When the spatial dimension of the discriminator is $4 \times 4$, we replicate the description embedding spatially and perform a depth concatenation. We then perform a $1 \times 1$ convolution followed by rectification and a $4 \times 4$ convolution to compute the final score from $D$. Batch normalization is performed on all convolutional layers.

## Matching-aware discriminator (GAN-CLS)

The most straightforward way to train a conditional GAN is to view (text, image) pairs as joint observations and train the discriminator to judge pairs as real or fake. This type of conditioning is naive in the sense that the discriminator has no explicit notion of whether real training images match the text embedding context.

However, as discussed also by , the dynamics of learning may be different from the non-conditional case. In the beginning of training, the discriminator ignores the conditioning information and easily rejects samples from $G$ because they do not look plausible. Once $G$ has learned to generate plausible images, it must also learn to align them with the conditioning information, and likewise $D$ must learn to evaluate whether samples from $G$ meet this conditioning constraint.

In naive GAN, the discriminator observes two kinds of inputs: real images with matching text, and synthetic images with arbitrary text. Therefore, it must implicitly separate two sources of error: unrealistic images (for *any* text), and realistic images of the wrong class that mismatch the conditioning information. Based on the intuition that this may complicate learning dynamics, we modified the GAN training algorithm to separate these error sources. In addition to the real / fake inputs to the discriminator during training, we add a third type of input consisting of real images with mismatched text, which the discriminator must learn to score as fake. By learning to optimize image / text matching in addition to the image realism, the discriminator can provide an additional signal to the generator.

Algorithm  summarizes the training procedure. After encoding the text, image and noise (lines 3-5) we generate the fake image ($\hat{x}$, line 6). $s_{r}$ indicates the score of associating a real image and its corresponding sentence (line 7), $s_{w}$ measures the score of associating a real image with an arbitrary sentence (line 8), and $s_{f}$ is the score of associating a fake image with its corresponding text (line 9). Note that we use $\partial \mathcal{L}_{D} / \partial D$ to indicate the gradient of $D$'s objective with respect to its parameters, and likewise for $G$. Lines 11 and 13 are meant to indicate taking a gradient step to update network parameters.

## Learning with manifold interpolation (GAN-INT)

Deep networks have been shown to learn representations in which interpolations between embedding pairs tend to be near the data manifold . Motivated by this property, we can generate a large amount of additional text embeddings by simply interpolating between embeddings of training set captions. Critically, these interpolated text embeddings need not correspond to any actual human-written text, so there is no additional labeling cost. This can be viewed as adding an additional term to the generator objective to minimize: $$\begin{aligned}
\mathbb{E}_{t_1,t_2 \sim p_{data}}[ \log (1 - D(G(z, \beta t_1 + (1-\beta) t_2 ))) ]
\end{aligned}$$ where $z$ is drawn from the noise distribution and $\beta$ interpolates between text embeddings $t_1$ and $t_2$. In practice we found that fixing $\beta = 0.5$ works well.

Because the interpolated embeddings are synthetic, the discriminator $D$ does not have "real" corresponding image and text pairs to train on. However, $D$ learns to predict whether image and text pairs match or not. Thus, if $D$ does a good job at this, then by satisfying $D$ on interpolated text embeddings $G$ can learn to fill in gaps on the data manifold in between training points. Note that $t_1$ and $t_2$ may come from different images and even different categories.[^1]

## Inverting the generator for style transfer

If the text encoding $\varphi(t)$ captures the image content (e.g. flower shape and colors), then in order to generate a realistic image the noise sample $z$ should capture style factors such as background color and pose. With a trained GAN, one may wish to transfer the style of a query image onto the content of a particular text description. To achieve this, one can train a convolutional network to invert $G$ to regress from samples $\hat{x} \leftarrow G(z,\varphi(t))$ back onto $z$. We used a simple squared loss to train the style encoder: $$\begin{aligned}
\mathcal{L}_{style} = \mathbb{E}_{t, z \sim \mathcal{N}(0,1)} || z - S(G(z,\varphi(t))) ||^{2}_{2}
\end{aligned}$$ where $S$ is the style encoder network. With a trained generator and style encoder, style transfer from a query image $x$ onto text $t$ proceeds as follows: $$\begin{aligned}
s \leftarrow S(x)\text{,   }
\hat{x} \leftarrow G(s, \varphi(t))\nonumber
\end{aligned}$$ where $\hat{x}$ is the result image and $s$ is the predicted style.

# Experiments

In this section we first present results on the CUB dataset of bird images and the Oxford-102 dataset of flower images. CUB has 11,788 images of birds belonging to one of 200 different categories. The Oxford-102 contains 8,189 images of flowers from 102 different categories.

As in  and , we split these into class-disjoint training and test sets. CUB has 150 train+val classes and 50 test classes, while Oxford-102 has 82 train+val and 20 test classes. For both datasets, we used 5 captions per image. During mini-batch selection for training we randomly pick an image view (e.g. crop, flip) of the image and one of the captions.

For text features, we first pre-train a deep convolutional-recurrent text encoder on structured joint embedding of text captions with 1,024-dimensional GoogLeNet image embedings  as described in . For both Oxford-102 and CUB we used a hybrid of character-level ConvNet with a recurrent neural network (char-CNN-RNN) as described in . Note, however that pre-training the text encoder is not a requirement of our method and we include some end-to-end results in the supplement. The reason for pre-training the text encoder was to increase the speed of training the other components for faster experimentation. We also provide some qualitative results obtained with MS COCO images of the validation set to show the generalizability of our approach.

We used the same GAN architecture for all datasets. The training image size was set to $64 \times 64 \times 3$. The text encoder produced $1,024$-dimensional embeddings that were projected to $128$ dimensions in both the generator and discriminator before depth concatenation into convolutional feature maps.

As indicated in Algorithm , we take alternating steps of updating the generator and the discriminator network. We used the same base learning rate of $0.0002$, and used the ADAM solver  with momentum $0.5$. The generator noise was sampled from a $100$-dimensional unit normal distribution. We used a minibatch size of $64$ and trained for $600$ epochs. Our implementation was built on top of `dcgan.torch`[^2].

## Qualitative results

We compare the GAN baseline, our GAN-CLS with image-text matching discriminator (), GAN-INT learned with text manifold interpolation () and GAN-INT-CLS which combines both.

Results on CUB can be seen in . GAN and GAN-CLS get some color information right, but the images do not look real. However, GAN-INT and GAN-INT-CLS show plausible images that usually match all or at least part of the caption. We include additional analysis on the robustness of each GAN variant on the CUB dataset in the supplement.

Results on the Oxford-102 Flowers dataset can be seen in . In this case, all four methods can generate plausible flower images that match the description. The basic GAN tends to have the most variety in flower morphology (i.e. one can see very different petal types if this part is left unspecified by the caption), while other methods tend to generate more class-consistent images. We speculate that it is easier to generate flowers, perhaps because birds have stronger structural regularities across species that make it easier for $D$ to spot a fake bird than to spot a fake flower.

Many additional results with GAN-INT and GAN-INT-CLS as well as GAN-E2E (our end-to-end GAN-INT-CLS without pre-training the text encoder $\varphi(t)$) for both CUB and Oxford-102 can be found in the supplement.

## Disentangling style and content

In this section we investigate the extent to which our model can separate style and content. By content, we mean the visual attributes of the bird itself, such as shape, size and color of each body part. By style, we mean all of the other factors of variation in the image such as background color and the pose orientation of the bird.

The text embedding mainly covers content information and typically nothing about style, e.g. captions do not mention the background or the bird pose. Therefore, in order to generate realistic images then GAN must learn to use noise sample $z$ to account for style variations.

To quantify the degree of disentangling on CUB we set up two prediction tasks with noise $z$ as the input: pose verification and background color verification. For each task, we first constructed similar and dissimilar pairs of images and then computed the predicted style vectors by feeding the image into a style encoder (trained to invert the input and output of generator). If GAN has disentangled style using $z$ from image content, the similarity between images of the same style (e.g. similar pose) should be higher than that of different styles (e.g. different pose).

To recover $z$, we inverted the each generator network as described in . To construct pairs for verification, we grouped images into 100 clusters using K-means where images from the same cluster share the same style. For background color, we clustered images by the average color (RGB channels) of the background; for bird pose, we clustered images by 6 keypoint coordinates (beak, belly, breast, crown, forehead, and tail).

For evaluation, we compute the actual predicted style variables by feeding pairs of images style encoders for GAN, GAN-CLS, GAN-INT and GAN-INT-CLS. We verify the score using cosine similarity and report the AU-ROC (averaging over 5 folds). As a baseline, we also compute cosine similarity between text features from our text encoder.

We present results on . As expected, captions alone are not informative for style prediction. Moreover, consistent with the qualitative results, we found that models incorporating interpolation regularizer (GAN-INT, GAN-INT-CLS) perform the best for this task.

## Pose and background style transfer

We demonstrate that GAN-INT-CLS with trained style encoder () can perform style transfer from an unseen query image onto a text description. shows that images generated using the inferred styles can accurately capture the pose information. In several cases the style transfer preserves detailed background information such as a tree branch upon which the bird is perched.

Disentangling the style by GAN-INT-CLS is interesting because it suggests a simple way of generalization. This way we can combine previously seen content (e.g. text) and previously seen styles, but in novel pairings so as to generate plausible images very different from any seen image during training. Another way to generalize is to use attributes that were previously seen (e.g. blue wings, yellow belly) as in the generated parakeet-like bird in the bottom row of . This way of generalization takes advantage of text representations capturing multiple visual aspects.

## Sentence interpolation

demonstrates the learned text manifold by interpolation (Left). Although there is no ground-truth text for the intervening points, the generated images appear plausible. Since we keep the noise distribution the same, the only changing factor within each row is the text embedding that we use. Note that interpolations can accurately reflect color information, such as a bird changing from blue to red while the pose and background are invariant.

As well as interpolating between two text encodings, we show results on (Right) with noise interpolation. Here, we sample two random noise vectors. By keeping the text encoding fixed, we interpolate between these two noise vectors and generate bird images with a smooth transition between two styles by keeping the content fixed.

## Beyond birds and flowers

We trained a GAN-CLS on MS-COCO to show the generalization capability of our approach on a general set of images that contain multiple objects and variable backgrounds. We use the same text encoder architecture, same GAN architecture and same hyperparameters (learning rate, minibatch size and number of epochs) as in CUB and Oxford-102. The only difference in training the text encoder is that COCO does not have a single object category per class. However, we can still learn an instance level (rather than category level) image and text matching function, as in .

Samples and ground truth captions and their corresponding images are shown on . A common property of all the results is the sharpness of the samples, similar to other GAN-based image synthesis models. We also observe diversity in the samples by simply drawing multiple noise vectors and using the same fixed text encoding.

From a distance the results are encouraging, but upon close inspection it is clear that the generated scenes are not usually coherent; for example the human-like blobs in the baseball scenes lack clearly articulated parts. In future work, it may be interesting to incorporate hierarchical structure into the image synthesis model in order to better handle complex multi-object scenes.

A qualitative comparison with AlignDRAW  can be found in the supplement. GAN-CLS generates sharper and higher-resolution samples that roughly correspond to the query, but AlignDRAW samples more noticably reflect single-word changes in the selected queries from that work. Incorporating temporal structure into the GAN-CLS generator network could potentially improve its ability to capture these text variations.

# Conclusions

In this work we developed a simple and effective model for generating images based on detailed visual descriptions. We demonstrated that the model can synthesize many plausible visual interpretations of a given text caption. Our manifold interpolation regularizer substantially improved the text to image synthesis on CUB. We showed disentangling of style and content, and bird pose and background transfer from query images onto text descriptions. Finally we demonstrated the generalizability of our approach to generating images with multiple objects and variable backgrounds with our results on MS-COCO dataset. In future work, we aim to further scale up the model to higher resolution images and add more types of text.

# Acknowledgments

This work was supported in part by NSF CAREER IIS-1453651, ONR N00014-13-1-0762 and NSF CMMI-1266184.

[^1]: In our experiments, we used fine-grained categories (e.g. birds are similar enough to other birds, flowers to other flowers, etc.), and interpolating across categories did not pose a problem.

[^2]: <https://github.com/soumith/dcgan.torch>


abstract: Named entity recognition (ner) is the very first step in the linguistic processing of any new domain. It is currently a common process in BioNLP on English clinical text. However, it is still in its infancy in other major languages, as it is the case for Spanish. Presented under the umbrella of the PharmaCoNER shared task, this paper describes a very simple method for the annotation and normalization of pharmacological, chemical and, ultimately, biomedical named entities in clinical cases. The system developed for the shared task is based on limited knowledge, collected, structured and munged in a way that clearly outperforms scores obtained by similar dictionary-based systems for English in the past. Along with this recovering of the knowledge-based methods for ner in subdomains, the paper also highlights the key contribution of *resource-based* systems in the validation and consolidation of both the annotation guidelines and the human annotation practices. In this sense, some of the authors discoverings on the overall quality of human annotated datasets question the above-mentioned 'official' results obtained by this system, that ranked second (0.91 F1-score) and first (0.916 F1-score), respectively, in the two PharmaCoNER subtasks.
author: Fernando Sánchez León
*unaffiliated*
`firstname.lastname@example.com`
Ana González Ledesma
*unaffiliated*
`email@example.com`
bibliography: PharmaCoNER-FSL-AGL_final.bib
title: Annotating and normalizing biomedical nes with limited knowledge[^1]

# Introduction

Named Entity Recognition (ner) is considered a necessary first step in the linguistic processing of any new domain, as it facilitates the development of applications showing co-occurrences of domain entities, cause-effect relations among them, and, eventually, it opens the (still to be reached) possibility of understanding full text content. On the other hand, Biomedical literature and, more specifically, clinical texts, show a number of features as regards ner that pose a challenge to NLP researchers : (1) the clinical discourse is characterized by being conceptually very dense; (2) the number of different classes for nes is greater than traditional classes used with, for instance, newswire text; (3) they show a high formal variability for nes (actually, it is rare to find entities in their "canonical form"); and, (4) this text type contains a great number of ortho-typographic errors, due mainly to time constraints when drafted.

Many ways to approach ner for biomedical literature have been proposed, but they roughly fall into three main categories: rule-based, dictionary-based (sometimes called knowledge-based) and machine-learning based solutions. Traditionally, the first two approaches have been the choice before the availability of Human Annotated Datasets (had), albeit rule-based approaches require (usually hand-crafted) rules to identify terms in the text, while dictionary-based approaches tend to miss medical terms not mentioned in the system dictionary . Nonetheless, with the creation and distribution of had as well as the development and success of supervised machine learning methods, a plethora of data-driven approaches have emerged —from Hidden Markov Models (HMMs) , Support Vector Machines (SVMs)  and Conditional Random Fields (CRFs) , to, more recently, those founded on neural networks . This fact has had an impact on knowledge-based methods, demoting them to a second plane. Besides, this situation has been favoured by claims on the uselessness of gazetteers for ner in, for example, Genomic Medicine (GM), as it was suggested by :

> One of the findings of the first BioCreative shared task was the demonstration of the long-suspected fact that gazetteers are typically of little use in GM.

Although one might think that this view could strictly refer to the subdomain of GM and to the past —BioCreative I was a shared task held back in 2004—, we can still find similar claims today, not only referred to rule-based and dictionary-based methods, but also to stochastic ones .
In this paper, in spite of previous statements, we present a system that uses rule-based and dictionary-based methods combined (in a way we prefer to call *resource-based*). Our final goals in the paper are two-fold: on the one hand, to describe our system, developed for the PharmaCoNER shared task[^2], dealing with the annotation of some of the nes in health records (namely, pharmacological, chemical and biomedical entities) using a revisited version of rule- and dictionary-based approaches; and, on the other hand, to give pause for thought about the quality of datasets (and, thus, the fairness) with which systems of this type are evaluated, and to highlight the key role of resource-based systems in the validation and consolidation of both the annotation guidelines and the human annotation practices.
In section , we describe our initial resources and explain how they were built, and try to address the issues posed by features (1) and (2) above. Section  depicts the core of our system and the methods we have devised to deal with text features (3) and (4). Results obtained in PharmaCoNER by our system are presented in section . Section  details some of our errors, but, most importantly, focusses on the errors and inconsistencies found in the evaluation dataset, given that they may shed doubts on the scores obtained by any system in the competition. Finally, we present some concluding remarks in section .

# Resource building

As it is common in resource-based system development, special effort has been devoted to the creation of the set of resources used by the system. These are mainly two —a flat subset of the snomed ct medical ontology[^3], and the library and a part of the contextual regexp grammars developed by for a previous competition on abbreviation resolution in clinical texts written in Spanish. The process of creation and/or adaptation of these resources is described in this section.

## SNOMED CT

Although the competition proposes two different scenarios, in fact, both are guided by the snomed ct ontology —for subtask 1, entities must be identified with offsets and mapped to a predefined set of four classes (`PROTEINAS`, `NORMALIZABLES`, `NO_NORMALIZABLES` and `UNCLEAR`); for subtask 2, a list of all snomed ct ids (sctid) for entities occurring in the text must be given, which has been called *concept indexing* by the shared task organizers[^4]. Moreover, PharmaCoNER organizers decided to promote snomed ct substance ids over product, procedure or other possible interpretations also available in this medical ontology for a given entity. This selection must be done even if the context clearly refers to a different concept, according to the annotation guidelines[^5] (henceforth, *AnnotGuide*) and the praxis. Finally, `PROTEINAS` is ranked as the first choice for substances in this category.

These previous decisions alone on the part of the organizers greatly simplify the task at hand, making it possible to build (carefully compiled) subsets of the entities to be annotated. This is a great advantage over open domain ner, where (like in GM) the texts may contain an infinite (and very creative indeed) number of nes. For clinical cases, although the ne density is greater, there exist highly structured terminological resources for the domain. Moreover, the set of classes to use in the annotation exercise for subtask 1 has been dramatically cut down by the organizers.

With the above-mentioned initial constraints in mind, we have painstakingly collected, from the whole set of snomed ct terms, instances of entities as classified by the human annotators in the datasets released by the organizers and, when browsing the snomed ct web version, we have tried to use the ontological hierarchical relations to pull a complete class down from snomed ct. This way, we have gathered 80 classes —from lipids to proteins to peptides or peptide hormones, from plasminogen activators to dyes to drugs or medicaments—, that have been arranged in a ranked way so as to mimic human annotators choices[^6]. The number of entities so collected (henceforth, 'primary entities') is 51,309.

## Contextual regexp grammars

Some of the entities to be annotated, specially those in abbreviated form, are ambiguous without a context. This is the case, for instance, of *PCR*, whose expanded forms are (among other meanings; we use only English expanded forms) 'reactive protein c', 'polymerase chain reaction', 'cardiorespiratory arrest'. In order to deal with these cases, we use a contextual regexp rule system with a lean and simple rule formalism previously developed . As an exemplification, we include one rule to deal with one of the cases of the preceeding ambiguity:

    b:[il::bioquímica|en sangre|hemoglobina|
       hemograma|leucocit|parásito|plaqueta|
       prote.na|recuento|urea] - [PCR] - >
    [m=proteína]

A rule has a left hand side (LHS) and a right hand side (RHS). There is a focus in the LHS (`PCR`, within dashes) and a left and right context (that may be empty). When the left context includes a `b:` (like in this case), it indicates either left or right context. The words in the context can take other qualifiers —in this case, the matching will be case insensitive (`i` to the left of `bioquímica`) and local (`l`), which means the disjunction of words and/or stems can be found in a distance of 40 characters (this can be modifified by the user). Hence, the rule applies, selecting the `proteína` expansion (in RHS) of `PCR` if any of the words/stems specified as local context (40 chars maximum) is matched either to the left or right of the focus term (which is usually an abbreviation).

With no tweaking at all for the datasets in PharmaCoNER competition, the system annotates correctly 18 out of 20 occurrences of *PCR* in the test dataset (a precision of 0.9)[^7].

This component of the system is important because, only when the previous abbreviation is expanded as the first string (that of a protein name), it must be annotated, according to the *AnnotGuide*. The same ambiguity happens with *Cr*, which may mean 'creatinine' or 'chrome'[^8]. These expansions are both `NORMALIZABLES`, but, obviously, their sctid is different.

The system currently uses 104 context rules, only for abbreviations and acronyms in the clinical cases. These rules, contrary to what is commonly referred in the biomedical processing literature , do not require a special domain knowledge (none of the authors do have it) and can be written, most of the times, in a very straightforward way in the formalism briefly described above.

# Development

In general, dictionary-based methods rely on strict string matching over a fixed set of lexical entries from the domain. This is clearly insufficient to deal with non-canonical linguistic forms of nes as used in clinical texts. For this reason, we have devised two different solutions to this shortcoming.

In the first place, we have munged a great number of our primary entities, in a way similar to that described in for gazetteers used for protected information anonymization in clinical texts. We basically transform canonical forms in other possible textual forms observed when working with biomedical texts. With such transformations, a system module converts a salt compound like *clorhidrato de ciclopentolato* into *ciclopentolato clorhidrato*, or simply the `PP` *de potasio* into its corresponding adjective *potásico*. Other, more complex conversions include the treatment of antibodies —for instance *anticuerpo contra especie de Leishmania* becomes *ac. Leishmania*, among other variants—, or pairs of antibiotics normally prescribed together —which have a unique sctid and whose order we handle just as the 'glueing' characters. Note, incidentally, that, while the input to this pre-processing step is always a string, the output can be a regular expression, that is linked to a sctid. Plural forms are also generated through this module, that uses 45 transformations (not all equally productive). Using these transformation rules, we produce 139,150 'secondary entities', many of them regexps. As a final (simple) example of this, consider the entity *antígeno CD13*: after applying one of the previous string-to-regexp transformations, it is converted to:

    (?:antígeno )?CD[- ]?13

With the previous regexp, the system is able to identify (and string-normalize) six different textual realizations of the same unique snomed ct term. There are more complex rules that, thus, produce many more potential strings. The important thing with this strategy is that through the generative power of these predictably-created regexps from snomed ct entities the system is able to improve its recall and overcome the limitations of traditional dictionary-based approaches.
Secondly, to tackle with careless drafting of clinical reports, a Levenshtein edit distance library[^9] is used on the whole background dataset. The process is run once, using our secondary entities as lexicon[^10] and a general vocabulary lexicon to rule out common words in the candidate search process. We have used distances in the range 1-3 (depending on string length) for sequences up to 3 words long[^11]. The output of this process, which links forms with spelling errors with canonical ones and, thus, to sctids, can be inspected prior to its inclusion in the system lexicon, if so desired.

## Annotation process

As such, the annotation process is very simple. The program reads the input byte stream trying to identify known entities by means of a huge regexp built through the pre-processing of the available resources. If the candidate entity is ambiguous and (at least) one contextual rule exists for it, it is applied. For the rest of the nes, the system assigns them the class and sctid found in our ranked in-memory lexicon. As already mentioned in passing, the system does not tokenize text prior to ner, a processing order that we consider the right choice for highly entity-dense texts. The data structures built during pre-processing are efficiently stored on disk for subsequent runs, so the pre-processing is redone only when resources are edited.

# Results

According to the organizers, and taking into account the ha of the tiny subset from the background dataset released to the participants[^12], the system obtained the scores presented in table , ranking as second best system for subtask1 and best system for subtask2 .[^13].

|               | Precision |  Recall | F1-score |
|:--------------|----------:|--------:|---------:|
| **Subtask 1** |   0.90625 | 0.91314 |  0.90968 |
| **Subtask 2** |   0.91108 | 0.92083 |  0.91593 |

Results for PharmaCoNER test dataset (both subtasks)

Our results are consistent with our poor understanding of the classes for subtask 1. Having a null knowledge of Pharmacology, Biomedicine or even Chemistry, assigning classes (as requested for subtask 1) to entities is very hard, while providing a sctid (subtask 2) seems an easier goal. We will explain the point with an example entity —*ácido hialurónico* ('hyaluronic acid'). Using the ontological structure of snomed ct, one can find the following parent relations (just in English):
*hyaluronic acid* `is-a` *mucopolysaccharide* `is-a` *protein*
The authors have, in this case, promoted the `PROTEINAS` annotation for this entity, disregarding its interpretation as a replacement agent and overlooking a recommendation on polysaccharides in the *AnnotGuide*. Fortunately, all its interpretations share a unique sctid. The same may be true for
*haemosiderin* `is-a` *protein*
which is considered `NORMALIZABLE` in the test dataset. Similar cases are responsible for the lower performance on subtask 1 with respect to the more complex subtask 2.
In spite of these human classification errors, our system scores outperform those obtained by PharmacoNER Tagger[^14] , a simpler system using a binary classification and a very different organization of the dataset with a smaller fragment for test (10% of the data as opposed to 25% for the official competition). In fact, our system improves their F1-score (89.06) by 1.3 points when compared with our results for the more complex PharmaCoNER subtask 1.

# Discussion

In this section, we perform error analysis for our system run on the test dataset. We will address both recall and precision errors, but mainly concentrate on the latter type, and on a thorough revision of mismatches between system and human annotations.
In general, error analysis is favoured by knowledge-based methods, since it is through the understanding of the underlying reasons for an error that the system could be improved. Moreover, and differently to what happens with the current wave of artificial neural network methods, the whole annotation process —its guidelines for human annotators, the collection and appropriate structuring of resources, the adequate means to assign tags to certain entities but not to other, similar or even pertaining to the same class— must be clearly understood by the designer/developer/data architect of such systems. As a natural consequence of this attempt to mimick a task defined by humans to be performed, in the first place, also by humans, some inconsistencies, asystematic or missing assignments can be discovered, and this information is a valuable treasure not only for system developers but also for task organizers, guideline editors and future annotation campaigns, not to mention for the exactness of program evaluation results.

Most of the error types made by the system (i.e., by the authors) in class assignment for subtask 1 have already been discussed. In the same vein, as regards subtask 2, a great number of errors come from the selection of the 'product containing substance' reading from snomed ct rather to the 'substance' itself. This is due to inexperience of the authors on the domain and the wrong consideration of context when tagging entities —the latter being clearly obviated in the *AnnotGuide*.

In the following paragraphs, some of the most relevant inconsistencies found when performing error analysis of our system are highlighted. The list is necessarily incomplete due to space constraints, and it is geared towards the explanation of our possible errors.

## Inconsistency in the ****

Among some of the paradoxical examples in the *AnnotGuide* it stands out the double explicit consideration of *gen* ('gene'), when occurs alone in context, as both an entity to be tagged (positive rule P2 of the *AnnotGuide*) and a noun not to be tagged (negative rule N2). This inconsistency (and a bit of bad luck) has produced that none of the 6 occurrences as an independent noun —not introducing an entity— is tagged in the train`+`dev (henceforth, t`+`d) while the only 2 in the same context in the test dataset have been tagged. This amounts for 2 true negatives (tns) for the evaluation script.

## Inconsistency in **** as regards ****

The *AnnotGuide* proposal for the treatment of elliptical elements is somewhat confusing. For these cases, a longest match annotation is proposed, which is difficult to replicate automatically and not easy to remember for the human annotator. In many contexts, the annotator has made the right choice —for instance, in *receptores de estrógeno y de progesterona*— whereas in others do not —*$|$anticuerpos anticardiolipina$|$ $|$IgG$|$ e $|$IgM$|$*, with '$|$' marking the edges of the annotations. The last example occurs twice in the test dataset. Hence, the disagreement counts as 6 tns and 2 false positives (fps)[^15].

On the other hand, there is a clear reference to food materials and nutrition in the *AnnotGuide*, where they are included in the class of substances. However, none of the following entities is tagged in the test dataset: *azúcar* (which is mandatory according to *AnnotGuide* and was tagged in t`+`d; 1 fp); *almidón de maíz* (also mandatory in *AnnotGuide*; 1 fp); and *Loprofín*, *Aglutella*, *Aproten* (hypoproteic nutrition products, 3 fps in total)[^16].

There is an explicit indication in the *AnnotGuide* to annotate salts, with the example *iron salts*. However, in the context *sales de litio* ('lithium salts'), only the chemical element has been tagged (1 fp[^17]).

There exist other differing-span mismatches between human and automatic annotation. These include *anticuerpos anticitoplasma de neutrófilo*, where the ha considers the first two words only (in one of the occurrences, 1 fp); in the text fragment *b2 microglobulina, CEA y CA 19,9 normales*, *CA 19,9* is the correct span for the last entity (and not *CA*, 1 fp); *A.S.T* is the span selected (for *A.S.T.*, 1 fp); finally, in the context *lgM anticore* only *lgM* has been tagged (1 fp).

Other prominent mismatch between had and *AnnotGuide* is that of *DNA*, which is explicitly included in the *AnnotGuide* (sects. P2 and O1). It accounts for 2 fps.
But perhaps one of the most common discrepancies between human and automatic annotation has to do with medicaments normally prescribed together, which have a unique sctid. Examples include *amiloride/hidroclorotiazida* (1 fp); and *betametasona + calcipotriol* (1 fp) in the test set. This situation was also observed in the t`+`d corpus fragment (*tenofovir + emtricitabina*, *carbonato cálcico /colecalciferol*, *lopinavir/ritonavir*).

## Inconsistency in **** on the test set as regards t`+`d sets

Some inconsistencies between dataset annotations have turned the authors crazy: *NPT* (acronym for 'total parenteral nutrition, TPN') is tagged in the train`+`dev dataset 15 out of 21 times it occurs[^18]. The common sense of frequency in the ha of texts has led us to tag it in the background set. Unluckily, neither *NPT* nor its expansion have been tagged in the test dataset. This has also been the behaviour in ha for 'parenteral nutrition' and 'enteral nutrition' (and their corresponding acronyms) in test dataset, since these entities have not been tagged. We asked the organizers about this and other entities for which we had doubts, either because the *AnnotGuide* didn't cover their cases or because the ha didn't match the recommendations in the *AnnotGuide*. Woefully, communication with the organizers has not been very fluent on this respect. All in all, this bad decision on the part of the authors amounts for 6 fps (more than 7.5% of our fps according to evaluation script).

For other cases, decisions that may be clearly induced from the tagging of train`+`dev datasets, have not been applied in the test corpus fragment. These include *cadenas ligeras* (5 times in t`+`d, 1 fp in test); *enzimas hepáticas* (tagged systematically in t`+`d, 1 fp); *p53* (also tagged in t`+`d, 1 fp).

Another entity that stands out is *hidratos de carbono* ('carbohydrates'). It is tagged twice in the t`+`d dataset, occurring 4 times in the set (once as *HC*). However, although the form *carbohidratos* has been annotated twice in the test set, *hidratos de carbono* has been not (1 fp).

Moreover, *suero* ('Sodium chloride solution' or 'serum') deserves its own comment. Both entity references are tagged in the train`+`dev datasets (although with the latter meaning it is tagged only 4 out of 12 occurrences). We decided to tag it due to its relevance. In the test dataset, it occurs 5 times with the blood material meaning, but it has only been tagged twice as such (one of them being an error, since it refers to the former meaning). Our system tagged all occurrences, but tagged also one of the instances with the former meaning as serum (3 fps).

Finally, there are some inconsistencies within the same dataset. For example, nutricional agent *Kabiven* is tagged as both `NORMALIZABLES` (with sctid) and `NO_NORMALIZABLES` in the very same text. The same happens with another nutritional complement, *Cernebit*, this time in two different files. The perfusion solution *Isoplasmal G* (with a typo in the datasets —*Isoplasmar G*) is tagged as `NORMALIZABLES` and `UNCLEAR`. These examples reveal a vague understanding (or definition) of criteria as regards fluids and nutrition, as we pointed out at the beginning of this section.

## Asystematic/incomplete annotation

Some of the entities occurring in the test dataset have not always been tagged. This is the case for *celulosa* (annotated only once but used twice, 1 fp); *vimentina* (same situation as previous, 1 fp); *LDH* (tagged 20 times in t`+`d but not in one of the files, 1 fp); *cimetidina* (1 fp); *reactantes de fase aguda* (2 fps; 2 other occurrences were tagged); *anticuerpos antinucleares* (human annotators missed 1, considered fp).

## Incorrect sctids

On our refinement work with the system, some incorrect sctids have emerged. These errors impact on subtask 2 (some also on subtask 1). A large sample of them is enumerated below.

*ARP* ('actividad de renina plasmática', 'plasma renin activity', PRA) cannot be linked to sctid for *renina*, which happens twice. In the context 'perfil de antigenos \[sic\] extraíbles del núcleo (ENA)', *ENA* has been tagged with sctid of the antibody (1 fp). In one of the files, *tioflavina* is linked to sctid of *tioflavina T*, but it could be *tioflavina S*. Thus, it should be `NO_NORMALIZABLE`. *Harvoni* is ChEBI:85082 and not `<null>` (1 fp). *AcIgM contra CMV* has a wrong sctid (1 fp). *HBsAg* has no sctid in the test set; it should be 22290004 ('Hepatitis B surface antigen') (1 fp).
There are other incorrect annotations, due to inadvertent human errors, like *biotina* tagged as `PROTEINAS` or *VEB* ('Epstein-Barr virus') being annotated when it is not a substance. Among these mismatches between ha and system annotation, the most remarkable is the case of synonyms in active principles. For instance, the brand name drug *Dekapine* has been linked to 'ácido valproico' in the former case and to 'valproato sódico' in the latter. These terms are synonymous[^19], but sadly they don't share sctid. Hence, this case also counts as a fp.
A gold standard dataset for any task is very hard to develop, so a continuous editing of it is a must[^20]. In this discussion, we have focused on false positives (fps) according to the script used for system evaluation, with the main purpose of *understanding* the domain knowledge encoded in the linguistic conventions (lexical/terminological items and constructions) used by health professionals, but also the decisions underlying both the *AnnotGuide* and the ha practice.

In this journey to system improvement and authors enlightenment, some inconsistencies, errors, omissions have come up, as it has been reflected in this section, so both the guidelines for and the practice of annotation can also be improved in future use scenarios of the clinical case corpus built and maintained by the shared task organizers.

Our conclusion on this state of affairs is that some of the inconsistencies spotted in this section show that there were not a rational approach to the annotation of certain entities contained in the datasets (apart from other errors and/or oversights), and, hence, the upper bound of any tagging system is far below the ideal 1.0 F1-score. To this respect, in very many cases, the authors have made the wrong choice, but in others they were guided by analogy or common sense. Maybe a selection founded on probability measures estimated on training material could have obtained better results with this specific test dataset. However, in the end, this cannot be considered as an indication of a better system performance, since, as it has been shown, the test dataset used still needs more refinement work to be used as the right dataset for automatic annotation evaluation.

# Conclusions

With this resource-based system developed for the PharmaCoNER shared task on ner of pharmacological, chemical and biomedical entities, we have demonstrated that, having a very limited knowledge of the domain, and, thus, making wrong choices many times in the creation of resources for the tasks at hand, but being more flexible with the matching mechanisms, a simple-design system can outperform a ner tagger for biomedical entities based on state-of-the-art artificial neural network technology. Thus, knowledge-based methods stand on their own merits in task resolution.

But, perhaps most importantly, the other key point brought to light in this contribution is that a resource-based approach also favours a more critical stance on the dataset(s) used to evaluate system performance. With these methods, system development can go hand in hand with dataset refinement in a virtuous circle that let us think that maybe next time we are planning to add a new gazetteer or word embedding to our system in order to try to improve system performance, we should first look at our data and, like King Midas, turn our Human Annotated Dataset into a true Gold Standard Dataset.

# Acknowledgements

We thank three anonymous reviewers of our manuscript for their careful reading and their many insightful comments and suggestions. We have made our best in providing a revised version of the manuscript that reflects their suggestions. Any remaining errors are our own responsability.

[^1]: This paper should have been published in the *Proceedings of the 5th Workshop on BioNLP Shared Tasks*. Unfortunately, due to their complete lack of funding, the authors could not afford the registration fees, a mandatory expense for a contribution to be published in the aforementioned proceedings.

[^2]: <http://temu.bsc.es/pharmaconer/>

[^3]: From <https://browser.ihtsdotools.org/>.

[^4]: In the train`+`dev datasets, only 17 of the `PROTEINAS` ('proteins') and `NORMALIZABLES` ('standardizable') entities have an id not in the snomed ct ontology. Besides, just 40 out of 5,615 annotations —not taking into account the class `UNCLEAR`, which is not considered for the system evaluation— are tagged as `NO_NORMALIZABLES` ('non standardizable'), many of them due to the fact that they include elliptical constructions.

[^5]: <https://bit.ly/2qxofgd>, p. 4.

[^6]: Note that we have gathered the complete set of medical terms included in snomed ct, but, for the purpose of this shared task, we only use a subset of it.

[^7]: Note that 2 of the *PCR* occurrences in the train`+`dev datasets have been incorrectly mapped to the protein interpretation (file `S1130-63432014000100012-1`, 2 times).

[^8]: Again, one of the occurrences of *Cr* has been incorrectly mapped to the former extended form (file `S0212-16112012000500042-1`).

[^9]: We use `Text::Levensthtein::Flexible` library, from `Perl` ecosystem. One of the anonymous reviewers has shed doubts about the use of `Perl` as a language for "NLP and text-mining nowadays". In this respect, we are not committed with a given programming language more than we are with our native language —and we have submitted our paper in English, a foreign language for us. The system could have been implemented in any other programming language more popular "nowadays", provided that we were as *proficient* in it as we are in `Perl` and the language used were as efficient in string and regexp handling and in I/O operations as `Perl` is. In this regard, the most popular language nowadays —`Python`— is 2 to 10 times slower for these particular features. `Perl` is even faster for regexp processing than `Python PyPy` —see, for instance, <https://github.com/mariomka/regex-benchmark>. Idiomatic `Perl` is even faster. Finally, `Perl` has a long tradition in biology and medicine text processing.

[^10]: With enumeration of strings from non-infinite-loop regexps.

[^11]: These words are not isolated from the byte stream, and the process uses textual anchors to delimit them as word candidates. Consequently, no proper tokenization is performed.

[^12]: When compared with the rest of the tasks in BioNLP-OST 2019, the time given to PharmaCoNER participants to submit their system runs is 4 times longer than the mean —longer time that is unnecessary if system is mature enough. On the other hand, the dataset released for evaluation purposes is more than 4 times larger than the mean. As a consequence, participating groups have to annotate full domain corpora rather than just test dataset(s). A shorter submission period and a smaller test dataset would be preferable, and besides fairer, in future calls.

[^13]: The authors have been unable to obtain these results with the official script, downloaded from <https://github.com/PlanTL-SANIDAD/PharmaCoNER-CODALAB-Evaluation-Script>. In their execution of the evaluation script, system results are better (?).

[^14]: The tagger authors, some of them also organizers of shared task, have changed the casing of the name for the program.

[^15]: When we indicate this kind of information, mostly using only fps, it must be understood that the system made the choice(s) that the authors judge as correct, although disagreeing with ha and/or *AnnotGuide*.

[^16]: On nutrition replacements, see also section .

[^17]: Note, in passing, that these span errors account for 1 tn also for the evaluation scripts.

[^18]: However, at least one expanded variant of it —*nutrición parenteral*, 'parenteral nutrition'— is never tagged.

[^19]: Although, 'valproato sódico' is the name used in the leaflet, as it can be seen in the Spanish Medicament Agency, AEMPS, web page (https://cima.aemps.es/cima/dochtml/p/
    48828/P_48828.html: last consulted on 16.07.2019).

[^20]: Besides, when the dataset is being used in a shared task, this refinement process should be available to participants while the task is open.


abstract: Surveillance is very essential for the safety of power substation. The detection of whether wearing safety helmets or not for perambulatory workers is the key component of overall intelligent surveillance system in power substation. In this paper, a novel and practical safety helmet detection framework based on computer vision, machine learning and image processing is proposed. In order to ascertain motion objects in power substation, the ViBe background modelling algorithm is employed. Moreover, based on the result of motion objects segmentation, real-time human classification framework C4 is applied to locate pedestrian in power substation accurately and quickly. Finally, according to the result of pedestrian detection, the safety helmet wearing detection is implemented using the head location, the color space transformation and the color feature discrimination. Extensive compelling experimental results in power substation illustrate the efficiency and effectiveness of the proposed framework.
author:
title: Automatic Safety Helmet Wearing Detection

# Introduction

As is well known, a surveillance system is considerable significant for power substation safety. Over the past decades, some artificial intelligent techniques like computer vision and machine learning with growing progress has been widely applied in intelligent surveillance in power substation . It can not only avoid time consuming labour intensive task, but also point out the power equipment fault and worker illegal operation in time and accurately against accidents.

Wang *et al.* designed a method to identify the status of the isolation switch and breaker in substation, which employed SIFT feature matching, Hough transform and KNN algorithm . Reddy *et al.* proposed a scheme for condition monitoring of insulators. Based on the ROI region obtained from K-Means algorithm, this scheme used discrete orthogonal S-transform in conjunction with adaptive neuro-fuzzy inference system to ascertain the condition of the insulators . Chen *et al.* presented a effective automatic detection and state recognition method for disconnecting switch, who makes full use of some prior knowledge about disconnecting switch and combines two important features of the fixed-contact . Liu *et al.* developed an image-based state recognition approach: they propose extraction of texture features using a Gabor transformation, then the state of the isolator is classified by the SVM .

Above-mentioned researches mainly focus on the power equipment fault detection and state recognition. Aside from equipment safety, intelligent surveillance system still need to monitor operator safety work. The real time safety helmet wearing detection for perambulatory workers, as a most common safe operation situation in power substation, is a considerable important task related to worker safe. Thus it is necessary to develop a system for automatic detection of safety helmet wearing in power substation. Unfortunately, the related work is little and mostly had been made in the detection of motorcyclists with or without helmet. Waranusast *et al.* utilized moving objects extracting and K-Nearest-Neighbor (KNN) classifier to develop a system which can automatically detect motorcycle riders and determine that they are wearing safety helmets or not . In , Silva *et al.* applied the circular Hough transform and the Histogram of Oriented Gradients descriptor to extract feature, utilized the Multi-layer perceptron classifier to ascertain motorcyclist without helmet. There are little researches about detection of safety helmet wearing in power substation. In , the Kalman filtering and Cam-shift algorithm are used to track pedestrians and determine motion objects. Meanwhile, the color information of safety helmets is used to detect safety helmets wearing.

The main purpose of this paper is to develop an innovative and practical safety helmet wearing detection system for perambulatory workers in power substation, based on computer vision, machine learning and image processing. The proposed detection framework focuses on design and implementation of detection of perambulatory workers with or without safety helmet in power substation, simultaneously is capable of ascertaining what color of the safety helmet. Toward this objective, the ViBe background modelling is employed to track motion objects. Furthermore, The C4 pedestrian detection algorithm is applied to determine motion objects whether human or not. Finally, the specific contribution made here is that utilizing the head location, the color space transformation and the color feature discrimination to detect whether wearing safety helmets or not for workers in power substation. Noticeably, the proposed framework is not only able to accurately realize the detection of whether wearing safety helmets or not for workers theoretically, but is also well appropriate of the practical application of intelligent surveillance in power substation on account of less dependence on hardware and more quick running speed.

The rest of this paper is organized as follows. Section II provides a overall framework of safety helmet wearing detection system. The background modeling, pedestrian detection and color feature discrimination method is detailed in Section III. Experiments and analyses are presented in Section IV. Finally, conclusions and future work are summarized in Section V.

# Safety Helmet Wearing Detection System Framework

Fig.  shows the schematic of safety helmet wearing detection system investigated in this paper. The system is divided in three steps: a) Moving objects segmentation; b) Pedestrians classification; c) Safety helmets detection.

## Moving Objects Segmentation

The purpose of moving objects segmentation in power substation is that obtaining moving pixels as quick as possible to reduce time of searching region of interest. On account of the video camera fixed in power substation, the background information can be used as valuable prior knowledge to extract moving targets. The intuitive idea is that comparing per frame with background image, and then calculating the different distance in pixels to determination moving objects. Here, we using a fast background modeling algorithm named visual background extractor (ViBe).

## Pedestrians classification

Based on the results of moving objects segmentation, extracting image features and classifying pedestrians in power substation will become the most crucial task for safety helmet wearing detection. The more precisely pedestrians is classified, the accurately safety helmet is detected. Thus it is necessary to select an excellent feature and an efficient classifier. The strategy explored in this part is that using C4 pedestrian detection algorithm . The only difference of C4 algorithm in this paper is that C4 pedestrians classification is executed on the surrounding of motion objects rather than all pixels of raw frame.

## Safety helmets detection

The final step is safety helmet detection. After the pedestrian classification, the head region need to be determined. Meanwhile, the color space transformation and color feature discrimination are implemented to ascertain whether wearing safety helmets or not for perambulatory workers. The image processing techniques including HSV transformation and adaptive threshold selection method are employed in this paper.

# Method Principles of System Framework

## ViBe:a universal background subtraction algorithm

ViBe is a pixel model used to estimate the background. The implementation of this algorithm is organized as three procedures: 1) model initialization 2) foreground segmentation 3) model update.

### Model Initialization

Model initialization, as the first step of ViBe algorithm, uses only the first frame of video sequences. It is an attractive merit that differs from other background model initialization depending on known certain frame numbers of the video. The details will be described as follows.

Suppose that ${N_t}(x)$ is a set of $N$ neighborhood sample values at time $t$ of the pixel $x$, ${n_{ti}}(x)$ is the component of this sample set. The first frame sample set can be represented by: $$\label{eq1}
{N_0}(x) = \{ {n_{01}}(x),{n_{02}}(x),...,{n_{0N}}(x)\}, i = 1,2,...,N$$

For each pixel of the first frame, select randomly one value from ${N_0}(x)$ as the initial value of pixel $x$. This model initialization can be denoted as: $$\label{eq2}
{B_0}(x) = \{ v|v \in {N_0}(x)\}$$ where ${B_0}(x)$ is the initial background model of the pixel $x$, $v$ represents value of pixel.

### Foreground Segmentation

Assume that ${S_R}({n_{ti}}(x))$ is a sphere of radius $R$ centered on ${n_{ti}}(x)$, which can be computed by comparing the ${n_{ti}}(x)$ to its closets values among the sample set ${N_t}(x)$. And then calculating the cardinality, defined $\#$, of the set intersection of this sphere and the sample set ${N_t}(x)$, which can be written as: $$\label{eq3}
\# \{ {S_R}({n_{ti}}(x)) \cap {N_t}(x)\}$$ Further, denoting $\# \min$ as a fixed threshold for background modeling. The pixel value is classified as foreground if the $\# \min$ is under the cardinality.

### Model Updating

After the foreground detection, ViBe algorithm adopts a conservative update scheme. Firstly, background pixel has a possibility of ${1 \mathord{\left/{\vphantom {1 \beta }} \right. \kern-\nulldelimiterspace} \beta }$ to be updated. Moreover, there is only a certain ${1 \mathord{\left/{\vphantom {1 \beta }} \right. \kern-\nulldelimiterspace} \beta }$ possibility for the update of current frame instead of other method with updating per frame. Finally, allowing each pixel to diffuse between neighboring pixels, randomly select its own sample set for updating.

## C4:a real-time human detection framework

C4 is a real-time and accurate human detector based on CENTRIST which focuses on contour cues for its detection. The overview of feature extraction and human detection chain is shown in Fig. . This method shows that contour information is particularly crucial for pedestrian detection. Therefore, it uses Sobel edge detection and Census Transformation (CT) to encode contours of pedestrians. After that, C4 algorithm integrates the position information into feature vector by dividing the image patch into blocks and combining adjacent blocks into cells. At last, The cascade classifier of linear SVM and histogram intersection kernel (HIK) SVM is used to determine human or non-human. The more details are illustrated as follows.

### Feature Extraction

As is shown in Fig. , to find an excellent feature of human objects from the motion foreground, we compute the Sobel gradient of each pixel at first. And then an 8 bit value, named Census Transform, is calculated to encode the signs of comparison. As illustrated in Eq. , by comparing a pixel value with its eight neighboring pixels, the bit is set to 1 if the center pixel is bigger than one of its neighbors or it is set to 0. The 8 bits can be collected in any order and converted to a CT value, which range from 0 to 255.

$$\label{eq4}
\begin{array}{*{20}{c}}
{32}&{64}&{96}\\
{32}&{64}&{96}\\
{32}&{32}&{96}
\end{array} \Rightarrow \begin{array}{*{20}{c}}
1&1&0\\
1&{}&0\\
1&1&0
\end{array} \Rightarrow {(11010110)_2} \Rightarrow CT = 214$$

In order to introduce spatial distribution information for CENTRIST feature vector, C4 algorithm partitions the image patch into $9 \times 6$ blocks and regards any adjacent $2 \times 2$ blocks as a super-block. The extracting unit is a super-block, thus there are 24 super-blocks. The final feature vector is $256 \times 24 = 6144$ dimensions.

### Pedestrian Classification

The detector is trained in a bootstrap way. In the training phase, we prepare a set of positive training image patches ${\cal P}$ and a set of non-human negative images ${\cal N}$. The first step is that randomly selecting a part of negative patches from ${\cal N}$ to form a subset of negative samples ${{\cal N}_1}$. Using ${\cal P}$ and ${{\cal N}_1}$, we train a SVM classifier ${H_1}$. Then, ${H_1}$ is employed to search hard examples, and these hard examples are added into ${{\cal N}_1}$ to form a new negative set ${H_2}$. Using ${\cal P}$ and ${{\cal N}_2}$, a SVM classifier ${H_2}$ is trained. It is a loop process until set times is over, the final trained linear SVM ${H_f}$ will be attained.

Here, the pedestrian classifier is a cascade classifier consisting of a linear SVM and a HIK SVM. Linear classifier, as the first classifier, achieves a rough determination of human with quick testing speed. While the HIK SVM classifier plays a more important role in accurate detection human on CENTRIST.

## Color feature discrimination

Assume that the pedestrian classification is well fulfilled using C4, the safety helmet detection can make full use of the location information of pedestrian. As shown in Fig. , the top region of one-fifth in the bounding box is our interest of region for head location. This value is chosen empirically. In order to use the most important color information of helmet to determine human whether wearing helmet or not, color space transformation and color feature discrimination are executed. Since the image in HSV color space is more adaptive to color segmentation, so we convert RGB to HSV. It is worth noting that setting fix threshold for Hue and Saturation channels can segment various colors, and then discriminating human with or without safety helmet. Whereas we only set the Hue channel threshold and not define the Saturation channel threshold. Instead, we use OSTU method on the channel of Saturation to obtain automatically threshold and segment color.

# Results And Discussions

## Experimental Setup

To evaluate the proposed safety helmet detection framework, extensive experiments were conducted on surveillance video sequences of power substation, which contain single pedestrian and multiple pedestrians under challenging backgrounds including electrical wire, wire pole, railing and etc. We capture ten videos with different scenes under fixed view and run our detection algorithm. Noticeably, proposed method is only adapted to detect whether wearing safety helmet or not for perambulatory workers, because safety patrol is one of the most common situations in power substation. Therefore, ten videos are both perambulatory situations.

Here, we have labeled data into wearing safety helmet and no wearing safety helmet manually. The accuracy of pedestrians classification is defined as $Ac{c_{pd}} = \frac{T}{{T + F}}$, where $T$ represents the number of correct classification results for pedestrians, $F$ represents the number of false classification results for pedestrians. Besides, based on the accurate pedestrian classification, Receiver Operating Characteristic (ROC) curve and Precision Recall (PR) curve of safety helmet wearing detection system are drew to illustrate the excellent performance of proposed method.

## Experimental Results and Analyses

Background modelling is the first step of safety helmet wearing detection. As is shown in Fig.  $(a)$ and $(b)$, the motion objects on scenes consisting of single pedestrian and multiple pedestrians respectively are segmented. The white pixels are motion foregrounds, the black pixels are static background instead. The locomotor objects detection results were satisfactory.

In terms of pedestrian detection, we use the *INRIA* person dataset and pedestrian data collected from power substation to train classifier. Fig.  $(a)$ and $(b)$ show the classification results of pedestrian on scenes including single pedestrian and multiple pedestrians respectively. The performance of the classifier is excellent on power substation with complex background.

Finally, the results of safety helmet wearing detection were presented in Fig.  $(a)$ and $(b)$. Single pedestrian safety helmet can be detected pretty well, but multiple pedestrians safety helmets maybe not detected sometimes. In our opinion, the main reason of failed examination is that the target is far from the surveillance camera. C4 pedestrian classification can not work on the too little target. Meanwhile, C4 features have limit description capability for pedestrians, which also can lead to failed examination. Besides, there are two reasons of false detection. One is the errors of pedestrian detection. such as the too big or too small bounding box, which can cause the error location of head. The other is that the color feature of safety helmet is sensitive to illumination changes. So here are much improving space.

Aside from qualitative analysis, quantitative analysis is needful to evaluate the performance of proposed method. The mean accuracy of pedestrian classification $Ac{c_{pd}}$ in ten videos is around 84.2$\%$. Fig.  and Fig.  respectively show the ROC curve and PR curve of different methods on ten videos in power substation. We can see that the performance of ViBe algorithm in conjunction with C4 classifier and color feature discrimination (CFD) is better than ViBe algorithm combining with HOG feature extraction, support vector machine (SVM) and color feature discrimination (CFD). The AUC index of ROC curve is used to determine the quality of different classifiers. The AUC of method using HOG feature extraction and SVM classifier is 89.20$\%$, and the AUC of our method is 94.13$\%$. It improves around 5$\%$.

In a word, no matter qualitative analysis or quantitative analysis, the proposed method has achieved accurate detection of perambulatory workers whether wearing safety helmet or not to a certain extent.

## Discussion

This work is meaningful for safety surveillance because the patrol and examine is most common in power substation and the determination of wearing safety helmet is crucial for life safe of perambulatory workers. In order to liberate labors, computer vision and machine learning algorithm is employed to complete this detection task. The ViBe background algorithm shrinks searching range for perambulatory workers. C4 algorithm implements pedestrians classification quickly. Color feature discrimination (CFD) ascertain safety helmet wearing situations. The overall framework is efficient and effective for this task.

However, there are some challenging problems. For example, Method of C4 pedestrian classification in conjunction with color feature discrimination with fixed parameters does not work very well for varying environments such as weather changes, which needs to modify parameters in different scenes.

# Conclusion

In this paper we have developed a novel and practical safety helmet wearing detection system to determine perambulatory workers to whether wear safety helmet or not. ViBe background modelling algorithm has realized excellent motion object detection. Furthermore, C4 pedestrians classification algorithm has implemented the worker location quickly. The crucial step of proposed safety helmet detection framework is color feature discrimination on HSV color space, which has achieved a great performance on common situations in power substation. Extensive experimental results has illustrated the effectiveness and efficiency of this safety helmet wearing detection system.

The ongoing and future work will focus on safety helmet wearing detection on some scenes including multiple pedestrians and complex background. The more accurate feature design and more robust detection method explore will be executed in future.

# Acknowledgment

This work was partly supported by National Natural Science Foundation of China under Grants 61271432 and 61421004, and partly supported by the State Grid Corporation of China Technology Projects(52053015000G).


abstract: This document is prepared using LaTeX2e and shows the desired format and appearance of a manuscript prepared for the Proceedings of the SPIE.[^1] It contains general formatting instructions and hints about how to use LaTeX. The LaTeX source file that produced this document, article.tex (Version 3.4), provides a template, used in conjunction with spie.cls (Version 3.4). These files are available on the Internet at <https://www.overleaf.com>. The font used throughout is the LaTeX default font, Computer Modern Roman, which is equivalent to the Times Roman font available on many systems.
author: Anna A. Author; Barry B. Author
bibliography: report.bib
title: SPIE Proceedings: Style template and guidelines for authors

# INTRODUCTION

Begin the Introduction below the Keywords. The manuscript should not have headers, footers, or page numbers. It should be in a one-column format. References are often noted in the text and cited at the end of the paper.

| Article title                 | 16 pt., bold, centered            |
|:------------------------------|:----------------------------------|
| Author names and affiliations | 12 pt., normal, centered          |
| Keywords                      | 10 pt., normal, left justified    |
| Abstract Title                | 11 pt., bold, centered            |
| Abstract body text            | 10 pt., normal, justified         |
| Section heading               | 11 pt., bold, centered (all caps) |
| Subsection heading            | 11 pt., bold, left justified      |
| Sub-subsection heading        | 10 pt., bold, left justified      |
| Normal text                   | 10 pt., normal, justified         |
| Figure and table captions     |   9 pt., normal                   |
| Footnote                      |   9 pt., normal                   |
| Reference Heading             | 11 pt., bold, centered            |
| Reference Listing             | 10 pt., normal, justified         |

Fonts sizes to be used for various parts of the manuscript. Table captions should be centered above the table. When the caption is too long to fit on one line, it should be justified to the right and left margins of the body of the text.

| Margin             | A4               | Letter          |
|:-------------------|:-----------------|:----------------|
| Top margin         | 2.54 cm          | 1.0 in.         |
| Bottom margin      | 4.94 cm          | 1.25 in.        |
| Left, right margin | 1.925 cm         | .875 in.        |
| Printable area     | 17.15 x 22.23 cm | 6.75 x 8.75 in. |

Margins and print area specifications.

LaTeX margins are related to the document's paper size. The paper size is by default set to USA letter paper. To format a document for A4 paper, the first line of this LaTeX source file should be changed to `\documentclass[a4paper]{spie}`.

Authors are encouraged to follow the principles of sound technical writing, as described in Refs.  and , for example. Many aspects of technical writing are addressed in the *AIP Style Manual*, published by the American Institute of Physics. It is available on line at <https://publishing.aip.org/authors>. A spelling checker is helpful for finding misspelled words.

An author may use this LaTeX source file as a template by substituting his/her own text in each field. This document is not meant to be a complete guide on how to use LaTeX. For that, please see the list of references at <http://latex-project.org/guides/> and for an online introduction to LaTeX please see .

# FORMATTING OF MANUSCRIPT COMPONENTS

This section describes the normal structure of a manuscript and how each part should be handled. The appropriate vertical spacing between various parts of this document is achieved in LaTeX through the proper use of defined constructs, such as `\section{}`. In LaTeX, paragraphs are separated by blank lines in the source file.

At times it may be desired, for formatting reasons, to break a line without starting a new paragraph. This situation may occur, for example, when formatting the article title, author information, or section headings. Line breaks are inserted in LaTeX by entering `\\` or `\linebreak` in the LaTeX source file at the desired location.

## Title and Author Information

The article title appears centered at the top of the first page. The title font is 16 point, bold. The rules for capitalizing the title are the same as for sentences; only the first word, proper nouns, and acronyms should be capitalized. Avoid using acronyms in the title. Keep in mind that people outside your area of expertise might read your article. At the first occurrence of an acronym, spell it out, followed by the acronym in parentheses, e.g., noise power spectrum (NPS).

The author list is in 12-pt. regular, centered. Omit titles and degrees such as Dr., Prof., Ph.D., etc. The list of affiliations follows the author list. Each author's affiliation should be clearly noted. Superscripts may be used to identify the correspondence between the authors and their respective affiliations. Further author information, such as e-mail address, complete postal address, and web-site location, may be provided in a footnote by using `\authorinfo{}`, as demonstrated above.

## Abstract and Keywords

The title and author information is immediately followed by the Abstract. The Abstract should concisely summarize the key findings of the paper. It should consist of a single paragraph containing no more than 250 words. The Abstract does not have a section number. A list of up to eight keywords should immediately follow the Abstract after a blank line. These keywords will be included in a searchable database at SPIE.

## Body of Paper

The body of the paper consists of numbered sections that present the main findings. These sections should be organized to best present the material. See Sec.  for formatting instructions.

## Appendices

Auxiliary material that is best left out of the main body of the paper, for example, derivations of equations, proofs of theorems, and details of algorithms, may be included in appendices. Appendices are enumerated with uppercase Latin letters in alphabetic order, and appear just before the Acknowledgments and References. Appendix  contains more about formatting equations and theorems.

## Acknowledgments

In the Acknowledgments section, appearing just before the References, the authors may credit others for their guidance or help. Also, funding sources may be stated. The Acknowledgments section does not have a section number.

## References

SPIE is able to display the references section of your paper in the SPIE Digital Library, complete with links to referenced journal articles, proceedings papers, and books, when available. This added feature will bring more readers to your paper and improve the usefulness of the SPIE Digital Library for all researchers. The References section does not have a section number. The references are numbered in the order in which they are cited. Examples of the format to be followed are given at the end of this document.

The reference list at the end of this document is created using BibTeX, which looks through the file report.bib for the entries cited in the LaTeX source file. The format of the reference list is determined by the bibliography style file spiebib.bst, as specified in the `\bibliographystyle{spiebib}` command. Alternatively, the references may be directly formatted in the LaTeX source file.

For books, the listing includes the list of authors, book title, publisher, city, page or chapter numbers, and year of publication. A reference to a journal article includes the author list, title of the article (in quotes), journal name (in italics, properly abbreviated), volume number (in bold), inclusive page numbers, and year. By convention, article titles are capitalized as described in Sec. . A reference to a proceedings paper or a chapter in an edited book includes the author list, title of the article (in quotes), volume or series title (in italics), volume number (in bold), if applicable, inclusive page numbers, publisher, city, and year. References to an article in the SPIE Proceedings may include the conference name (in italics), as shown in Ref. . For websites the listing includes the list of authors, title of the article (in quotes), website name, article date, website address either enclosed in chevron symbols ('$<$' and '$>$'), underlined or linked, and the date the website was accessed.

If you use this formatting, your references will link your manuscript to other research papers that are in the CrossRef system. Exact punctuation is required for the automated linking to be successful.

Citations to the references are made using superscript numerals, as demonstrated in the above paragraph. One may also directly refer to a reference within the text, e.g., "as shown in Ref.  ..."

## Footnotes

Footnotes[^2] may be used to provide auxiliary information that doesn't need to appear in the text, e.g., to explain measurement units. They should be used sparingly, however.

Only nine footnote symbols are available in LaTeX. If you have more than nine footnotes, you will need to restart the sequence using the command `\footnote[1]{Your footnote text goes here.}`. If you don't, LaTeX will provide the error message Counter too large., followed by the offending footnote command.

# SECTION FORMATTING

Section headings are centered and formatted completely in uppercase 11-point bold font. Sections should be numbered sequentially, starting with the first section after the Abstract. The heading starts with the section number, followed by a period. In LaTeX, a new section is created with the `\section{}` command, which automatically numbers the sections.

Paragraphs that immediately follow a section heading are leading paragraphs and should not be indented, according to standard publishing style. The same goes for leading paragraphs of subsections and sub-subsections. Subsequent paragraphs are standard paragraphs, with 14-pt. (5 mm) indentation. An extra half-line space should be inserted between paragraphs. In LaTeX, this spacing is specified by the parameter `\parskip`, which is set in spie.cls. Indentation of the first line of a paragraph may be avoided by starting it with `\noindent`.

## Subsection Attributes

The subsection heading is left justified and set in 11-point, bold font. Capitalization rules are the same as those for book titles. The first word of a subsection heading is capitalized. The remaining words are also capitalized, except for minor words with fewer than four letters, such as articles (a, an, and the), short prepositions (of, at, by, for, in, etc.), and short conjunctions (and, or, as, but, etc.). Subsection numbers consist of the section number, followed by a period, and the subsection number within that section.

### Sub-subsection attributes

The sub-subsection heading is left justified and its font is 10 point, bold. Capitalize as for sentences. The first word of a sub-subsection heading is capitalized. The rest of the heading is not capitalized, except for acronyms and proper names.

# FIGURES AND TABLES

Figures are numbered in the order of their first citation. They should appear in numerical order and on or after the same page as their first reference in the text. Alternatively, all figures may be placed at the end of the manuscript, that is, after the Reference section. It is preferable to have figures appear at the top or bottom of the page. Figures, along with their captions, should be separated from the main text by at least 0.2 in. or 5 mm.

Figure captions are centered below the figure or graph. Figure captions start with the figure number in 9-point bold font, followed by a period; the text is in 9-point normal font; for example, "Figure 3. Original image...". See Fig.  for an example of a figure caption. When the caption is too long to fit on one line, it should be justified to the right and left margins of the body of the text.

Tables are handled identically to figures, except that their captions appear above the table.

# MULTIMEDIA FIGURES - VIDEO AND AUDIO FILES

Video and audio files can be included for publication. See Tab.  for the specifications for the mulitimedia files. Use a screenshot or another .jpg illustration for placement in the text. Use the file name to begin the caption. The text of the caption must end with the text "http://dx.doi.org/doi.number.goes.here" which tells the SPIE editor where to insert the hyperlink in the digital version of the manuscript.

Here is a sample illustration and caption for a multimedia file:

| Item | Video | Audio |
|:---|:---|:---|
| File name | Video1, video2... | Audio1, audio2... |
| Number of files | 0-10 | 0-10 |
| Size of each file | 5 MB | 5 MB |
| File types accepted | .mpeg, .mov (Quicktime), .wmv (Windows Media Player) | .wav, .mp3 |

Information on video and audio files that must accompany a manuscript submission.

# MISCELLANEOUS FORMATTING DETAILS

It is often useful to refer back (or forward) to other sections in the article. Such references are made by section number. When a section reference starts a sentence, Section is spelled out; otherwise use its abbreviation, for example, "In Sec. 2 we showed..." or "Section 2.1 contained a description...". References to figures, tables, and theorems are handled the same way.

## Formatting Equations

Equations may appear in line with the text, if they are simple, short, and not of major importance; e.g., $\beta = b/r$. Important equations appear on their own line. Such equations are centered. For example, "The expression for the field of view is $$\label{eq:fov}
2 a = \frac{(b + 1)}{3c} \, ,$$ where $a$ is the ..." Principal equations are numbered, with the equation number placed within parentheses and right justified.

Equations are considered to be part of a sentence and should be punctuated accordingly. In the above example, a comma follows the equation because the next line is a subordinate clause. If the equation ends the sentence, a period should follow the equation. The line following an equation should not be indented unless it is meant to start a new paragraph. Indentation after an equation is avoided in LaTeX by not leaving a blank line between the equation and the subsequent text.

References to equations include the equation number in parentheses, for example, "Equation () shows ..." or "Combining Eqs. (2) and (3), we obtain..." Using a tilde in the LaTeX source file between two characters avoids unwanted line breaks.

## Formatting Theorems

To include theorems in a formal way, the theorem identification should appear in a 10-point, bold font, left justified and followed by a period. The text of the theorem continues on the same line in normal, 10-point font. For example,

**Theorem 1.** For any unbiased estimator...

Formal statements of lemmas and algorithms receive a similar treatment.

This unnumbered section is used to identify those who have aided the authors in understanding or accomplishing the work presented and to acknowledge sources of funding.

[^1]: The basic format was developed in 1995 by Rick Hermann (SPIE) and Ken Hanson (Los Alamos National Lab.).

[^2]: Footnotes are indicated as superscript symbols to avoid confusion with citations.