abstract: Image paragraph generation is the task of producing a coherent story (usually a paragraph) that describes the visual content of an image. The problem nevertheless is not trivial especially when there are multiple descriptive and diverse gists to be considered for paragraph generation, which often happens in real images. A valid question is how to encapsulate such gists/topics that are worthy of mention from an image, and then describe the image from one topic to another but holistically with a coherent structure. In this paper, we present a new design — Convolutional Auto-Encoding (CAE) that purely employs convolutional and deconvolutional auto-encoding framework for topic modeling on the region-level features of an image. Furthermore, we propose an architecture, namely CAE plus Long Short-Term Memory (dubbed as CAE-LSTM), that novelly integrates the learnt topics in support of paragraph generation. Technically, CAE-LSTM capitalizes on a two-level LSTM-based paragraph generation framework with attention mechanism. The paragraph-level LSTM captures the inter-sentence dependency in a paragraph, while sentence-level LSTM is to generate one sentence which is conditioned on each learnt topic. Extensive experiments are conducted on Stanford image paragraph dataset, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, CAE-LSTM increases CIDEr performance from 20.93% to 25.15%.
author: Jing Wang$^1$; Yingwei Pan$^2$; Ting Yao$^{2}$; Jinhui Tang$^1$[^1] Tao Mei$^2$
$^1$School of Computer Science and Engineering, Nanjing University of Science and Technology, China
$^2$JD AI Research, Beijing, China
{jwang, firstname.lastname@example.com, {panyw.ustc, email@example.com, firstname.lastname@example.com
bibliography: ijcai19.bib
title: Convolutional Auto-encoding of Sentence Topics
  for Image Paragraph Generation[^2]

# Introduction

The recent advances in Convolutional Neural Networks (CNN) have successfully pushed the limits and improved the state-of-the-art technologies of image understanding. As such, it has become achievable to recognize an image with a pre-defined set of categories. In a further step to describe an image in a natural-language utterance, image captioning has expanded the understanding from individual labels to a sentence to reflect the visual content in the image. Nevertheless, considering that a single sentence certainly has a upper-bound of descriptive capacity and thus fails to recapitulate every details in the image, the task of image paragraph generation is recently introduced in . The ultimate goal is to generate a coherent paragraph that describes an image in a finer manner.

The difficulty of image paragraph generation originates from two aspects: 1) how to distill the gists/topics that are worthy of mention from an image? 2) how to describe each topic in one sentence while ensuring the coherence across sentences to form a paragraph? One straightforward way in to alleviate the first issue relies on Recurrent Neural Networks (RNN), e.g., Long Short-Term Memory (LSTM) network. The region-level features are encoded into a global vector via mean pooling, which is subsequently input into LSTM to decode the topics, as shown in Figure (a). In this case, the inherent structure among all regions is not fully exploited, making it difficult to encapsulate the topics in the image. Furthermore, there is no clear evidence that the outputs of LSTM could characterize the topics well. Instead, we novelly devise Convolutional Auto-Encoding (CAE) structure to model the topics of an image in this paper as illustrated in Figure (b). CAE, on one hand, abstracts the topics in the encoder by employing convolutions over the region-level features, and on the other, steers the deconvolutional decoder through reconstruction from topics to features. As such, the learnt topics are potentially more representative and contain the information needed. To address the second issue, we remould a two-level LSTM-based paragraph generation framework, in which paragraph-level LSTM models the dependency holistically across all the sentences in a paragraph and sentence-level LSTM generates words in sequence conditioning on each learnt topic.

By consolidating the idea of modeling sentence topics for image paragraph generation, we present a new Convolutional Auto-Encoding plus Long Short-Term Memory (CAE-LSTM) architecture, as shown in Figure . Specifically, Faster R-CNN is firstly implemented to detect a set of salient image regions. We purely perform convolutions over all the region-level features in the encoder to distill the knowledge and extract the topics in the image. The learnt topics are ensured to capture holistic and representative information through achieving high reconstruction quality by the deconvolutional decoder. After that, we fuse all the region-level features via mean pooling as image representation, which are fed into a two-level LSTM networks for paragraph generation. The paragraph-level LSTM typically explores the dependency recursively throughout the procedure of paragraph generation and outputs a new initial state at a time for sentence-level LSTM with the paragraph history. As such, the generation of one sentence is affected by semantic context from the previous sentences when producing the paragraph. The sentence-level LSTM generates the sentence conditioning on each learnt topic, one word at each time step. Please also note that we uniquely design the metric of Coverage which is to encourage global coverage of objects in the sentence and consider such metric as a reward in the self-critical training strategy. The whole CAE-LSTM framework is jointly optimized.

The main contribution of this work is the proposal of modeling gists/topics in an image to boost image paragraph generation. This also leads to the elegant views of how to distill the knowledge and abstract the topics in an image, and how to nicely integrate the learnt topics into sentence generation and ensure inter-sentence dependency in a paragraph, which are problems not yet fully understood in the literature.

# Related Work

#### Image Captioning.

The dominant paradigm in modern image captioning is sequence learning methods which utilize CNN plus RNN model to generate free-form and content-relevant captions. proposes an end-to-end neural network architecture by utilizing LSTM to generate sentence for an image. Later, by amending the discrepancy between training and inference distributions for sequence modeling, presents a self-critical sequence training strategy to further enhance image captioning. Most recently, a combined bottom-up and top-down attention mechanism is proposed for image captioning, which enables attention at the level of objects and salient image regions.

#### Image Paragraph Generation.

The task of image paragraph generation has received increasing attention recently, which describes an image with a long, descriptive, and coherent paragraph. is one of the early works that leverages a hierarchical RNN for generating paragraph with the input of region features. In particular, a sentence RNN recursively generates sentence topic vectors conditioned on the global vector (i.e., the mean pooling of region-level features) and a word RNN is subsequently adopted to decode each topic into output sentence. extends the hierarchical RNN by involving multi-level adversarial discriminators for paragraph generation. The paragraph generator is thus enforced to produce realistic paragraphs with smooth logical transition between sentence topics. Furthermore, augments the hierarchical RNN with coherence vectors, global topic vectors, and a formulation of Variational Auto-Encoders to further model the inherent ambiguity of associating paragraphs with images.

#### Summary.

In short, our approach focuses on the latter scenario, which produces a coherent paragraph to depict an image. Unlike most of the aforementioned methods that recursively decode gists/topics for paragraph generation via LSTM conditioned on the global vector, our work contributes by exploiting a convolutional and deconvolutional auto-encoding module for gists/topics modeling over region-level features. Such design not only abstracts the topics in the convolutional encoder by exploiting inherent structure among all regions, but also steers the deconvolutional decoder through reconstruction to enforce the distilled topics more representative and informative. Moreover, a reward of Coverage is adopted in the self-critical training strategy to encourage global coverage of objects in the paragraph.

# Approach

We devise our Convolutional Auto-Encoding plus Long Short-Term Memory (CAE-LSTM) architecture to facilitate image paragraph generation by modeling gists/topics in an image. CAE-LSTM firstly utilizes a convolutional encoder to encapsulate region-level features into the topics, which are endowed with the holistic and representative information through achieving high reconstruction quality by deconvolutional decoder. The distilled topics are further integrated into a two-level LSTM-based paragraph generator, enabling the inter-sentence dependency modeling in a paragraph via paragraph-level LSTM and topic-oriented sentence generation through sentence-level LSTM. The overall training of CAE-LSTM is performed by exploring the reconstruction loss to pursue high-quality reconstruction from topics to region-level features, and the sequence-level reward (e.g., CIDEr) plus coverage reward in self-critical training to encourage the maximum coverage of objects in the paragraph. An overview of our model is depicted in Figure .

## Notation

Suppose we have an image ${I}$ to be described by a paragraph $\mathcal{P}$, where $\mathcal{P} = \{s_k\}^K_{k=1}$ consisting of $K$ sentences. Each sentence $s_k = \{w_{(k,t)}\}^{T_k}_{t=1}$ is composed of $T_k$ words and each word $w_{(k,t)}$ is represented as $D_s$-dimensional textual feature $\textbf{w}_{(k,t)}$. Faster R-CNN is firstly leveraged to produce a set of detected objects $\mathcal{V}=\{{r_m}\}^{M}_{m=1}$ with $M$ image regions in ${I}$. ${{\bf{v}}^0_m}\in {{\mathbb{R}}^{D_0}}$ denotes the $D_0$-dimensional feature of each image region $r_m$.

## CAE for Topic Modeling

The most typical way to distill the topics from an image is to encode region-level features into a global vector via mean pooling and a LSTM-based decoder is subsequently utilized to recursively output topics conditioning on the global vector, while the inherent structure among all regions is not fully exploited. Furthermore, there is no clear evidence that the outputs of LSTM could characterize the topics well. Here we devise a Convolutional Auto-Encoding (CAE) module for topic modeling. The spirit behind follows the philosophy that the holistical topics abstraction over all region-level features via convolutional encoder and the reconstruction from topics to features via deconvolutional decoder can enforce the learnt topics to be more representative and informative.

#### Convolutional Encoder.

Given the set of detected regions $\mathcal{V}=\{{r_m}\}^{M}_{m=1}$, a convolutional encoder is leveraged to encapsulate all the image regions into $K$ topic vectors, which is purely applied with convolutions. Specifically, each region ${{\bf{v}}^0_m}$ is firstly embedded into a $D_1$-dimensional region-level feature ${{\bf{v}}_m}$ through a linear layer. A region feature map ${\textbf{V}}\in{\mathbb{R}^{M \times D_1 \times 1}}$ is thus constructed by concatenating all region-level features (i.e., ${{\bf{v}}_m}$ is the $m$-th column of ${\textbf{V}}$), which will be set as the input of convolutional encoder. Here $M$, $D_1$, and $1$ denote the width, height, and the number of channels for the input feature map ${\textbf{V}}$, respectively. Next, we utilize a convolutional layer ($conv$) to encode the feature map ${\textbf{V}}$ into the topic feature map $\textbf{V}^s$ consisting of topic vectors: $$\label{eq:01}
\textbf{V}^s=conv (\textbf{V})\in{\mathbb{R}^{1 \times D_2 \times K}}.$$ For the convolutional layer $conv$, the size of the convolutional filter is set as $M \times C_1$, the stride size is $C_2$, and the filter number is $K$. Here each vector along channel dimension within the topic feature map $\textbf{V}^s$ can be regarded as the $k$-th distilled topic vector $\textbf{v}_{k}^s\in{\mathbb{R}^{D_2}}$.

#### Deconvolutional Decoder.

For deconvolutional decoder, we perform deconvolution (i.e., the conjugate operation of convolution) to decode the distilled topic feature map, $\textbf{V}^s$, back to the region feature map. In particular, given the distilled topic feature map $\textbf{V}^s$, a deconvolutional layer ($deconv$) is adopted to reproduce the region feature map $\tilde{\textbf{V}}$: $$\label{eq:02}
\tilde{\textbf{V}}=deconv (\textbf{V}^s)\in{\mathbb{R}^{M \times D_1 \times 1}}.$$ The filter size and the stride size in $deconv$ are set the same as the convolutional layer $conv$. Here the $m$-th column of the reconstructed region feature map $\tilde{\textbf{V}}$ corresponds to the reconstructed region-level feature for $m$-th image region.

#### Reconstruction Loss.

To measure the quality of reconstruction for our CAE module, a global reconstruction loss is defined as the L$_1$ distance between the input region feature map $\textbf{V}$ and reconstructed region feature map $\tilde{\textbf{V}}$: $$\label{eq:03}
L_{rec}(\tilde{\textbf{V}},\textbf{V}) = {\parallel\tilde{\textbf{V}}-\textbf{V} \parallel}_1.$$ We also experimented with L$_2$ distance as the reconstruction loss but that did not make a major difference. By minimizing the reconstruction loss, the distilled topic vectors are enforced to capture holistic and representative information from all regions through achieving higher reconstruction quality.

Note that besides acting as the gists to guide the sentence generation in the followed two-level LSTM networks, the distilled topic vectors can also be utilized to determine the number of generated sentences. Specifically, each topic vector is additionally injected into a linear layer to obtain a distribution over two states {CONTINUE=0, STOP=1}, which determines whether the sentence is the last one in the paragraph.

## Two-level LSTM-based Paragraph Generator

Inspired by the two-layer LSTM structure with top-down attention mechanism in , we remould a two-level LSTM-based paragraph generator consisting of a paragraph-level LSTM for inter-sentence dependency modeling and a sentence-level LSTM for sentence generation conditioning on each distilled topic, as illustrated in Figure .

To be specific, at each time step $t$ for generating the $k$-th sentence in a paragraph, the input vector $\textbf{x}_{(k,t)}^p$ of paragraph-level LSTM is defined as the concatenation of the previous output of sentence LSTM $\textbf{h}_{(k,t-1)}^s\in{\mathbb{R}^{H}}$, the mean-pooled image feature $\bar{\textbf{v}} = \frac{1}{M}\sum_{m=1}^M \textbf{v}_m$, and the embedding of previously generated word $\textbf{w}_{(k,t-1)}$: $$\small
\label{eq:1}
\textbf{x}_{(k,t)}^p=[\textbf{h}_{(k,t-1)}^s, \bar{\textbf{v}}, \textbf{W}_s\textbf{w}_{(k,t-1)}],$$ where $\textbf{W}_s$ is transformation matrix for input word. Such input collects the maximum contextual information for paragraph-level LSTM to model inter-sentence dependency.

Next, given the output $\textbf{h}_{(k,t)}^p\in{\mathbb{R}^{H}}$ of paragraph-level LSTM and the corresponding topic vector $\textbf{v}_{k}^s\in{\mathbb{R}^{D_2}}$, a normalized attention distribution over all regions is measured as $$\small
\label{eq:2}
\begin{array}{l}
    a^m_{(k,t)} = \textbf{W}_{att} [tanh(\textbf{W}_v\textbf{v}_m+\textbf{W}_h\textbf{h}_{(k,t)}^p+\textbf{W}_t\textbf{v}^s_k)],\\
    \bm{\alpha}_{(k,t)} = softmax(\textbf{a}_{(k,t)}),
\end{array}$$ where $a^m_{(k,t)}$ is the $m$-th element of $\textbf{a}_{(k,t)}$, $\textbf{W}_{att}\in{\mathbb{R}^{1 \times D_3}}$, $\textbf{W}_v\in{\mathbb{R}^{D_3 \times D_1}}$, $\textbf{W}_h\in{\mathbb{R}^{D_3 \times H}}$, and $\textbf{W}_t\in{\mathbb{R}^{D_3 \times D_2}}$ are transformation matrices, respectively. Here the $m$-th element ${\alpha}^m_{(k,t)}$ of the attention distribution $\bm{\alpha}_{(k,t)}$ denotes the attention probability of $\textbf{v}_m$. The attended image feature $\hat{\textbf{v}}_{(k,t)} = \sum_{m=1}^{M}\alpha^m_{(k,t)}\textbf{v}_m$ is thus calculated by aggregating all region-level features weighted by attention.

We further set the concatenation of the attended image feature $\hat{\textbf{v}}_{(k,t)}$, the corresponding topic vector $\textbf{v}_{k}^s$ and the output $\textbf{h}_{(k,t)}^p$ of paragraph-level LSTM, as the input $\textbf{x}_{(k,t)}^s$ of sentence-level LSTM for topic-oriented sentence generation: $$\small
\label{eq:5}
\textbf{x}_{(k,t)}^s=[\hat{\textbf{v}}_{(k,t)}, \textbf{v}^s_k, \textbf{h}_{(k,t)}^p].$$ The output $\textbf{h}_{(k,t)}^s$ of sentence-level LSTM is fed into a softmax layer to predict the next word $w_{(k,t)}$.

Note that the initial state of sentence-level LSTM is set to zero at every beginning of sentence generation. The state of paragraph-level LSTM is initialized with zero only at the beginning of the entire paragraph generation since paragraph-level LSTM should explore inter-sentence dependency recursively throughout whole procedure of paragraph generation.

## Self-critical Training with Coverage Reward

To further boost our paragraph generation model by amending the discrepancy between training and inference, we adopt the self-critical training strategy to optimize two-level LSTM-based paragraph generator. Despite having high quantitative scores, qualitative analysis shows that the captioning models solely optimized with sequence-level loss/reward are often limited to describing very generic information of objects, or rely on prior information and correlations from training examples, and resulting frequently in undesired effects such as object hallucination . As a result, besides the sequence-level reward (e.g., CIDEr), we additionally include a coverage reward in self-critical training strategy to encourage the global coverage of objects in the paragraph and thus improve the paragraph.

Specifically, we select the top-1$k$ frequent objects from the vocabulary in the training data as the high-frequency objects. The coverage reward is defined as the coverage ratio of high-frequency objects for the generated paragraph relative to ground-truth paragraph: $R^c =\frac{|Q_g\cap Q_{gt}|}{|Q_{gt}|}$, where $|\cdot|$ denotes the number of elements in a set. Here $Q_g$ and $Q_{gt}$ represent the set of high-frequency objects mentioned in the generated paragraph and ground-truth paragraph respectively. Accordingly, the final reward in self-critical training strategy is measured as the combination of CIDEr reward ($R^d$) and coverage reward: $R = \beta R^c + R^d$, where $\beta$ is the tradeoff parameter.

# Experiments

## Datasets and Experimental Settings

#### Dataset.

We conducted the experiments and evaluated our CAE-LSTM on Stanford image paragraph dataset (Stanford) , a benchmark in the field of image paragraph generation. The dataset contains 19,551 images and there is one human-annotated paragraph per image. On average, each paragraph has 67.5 words and each sentence consists of 11.91 words. In our experiments, we follow the widely used settings in and take 14,575 images for training, 2,487 for validation and 2,489 for testing.

#### Compared Methods.

We compare the proposed method with the following state-of-the-art methods: (1) **Image-Flat** is a standard image captioning model which directly decodes an image into a paragraph word by word, via a single LSTM. (2) **Regions-Hierarchical** adopts a hierarchical LSTM to generate a paragraph, sentence by sentence. (3) **RTT-GAN** integrates sentence attention and word attention into the hierarchical LSTM, coupled with adversarial training strategy. (4) **CapG-RevG** leverages coherence vectors/global topic vectors to generate coherent paragraphs and maintains the diversity of the paragraphs by a variational auto-encoder formulation. (5) **CAE-LSTM** is the proposal in this paper. Moreover, one degraded version of CAE-LSTM, i.e., **LSTM-ATT**, is devised to model topics via LSTM (instead of CAE) and adopt the same two-level LSTM architecture with attention for paragraph generation, which is trained without self-critical training strategy.

#### Settings.

For each image, we apply Faster R-CNN to detect objects within this image and select top $M=50$ regions with highest detection confidences to represent the image. Each region is represented as the 4,096-dimensional output of fc7 layer after RoI pooling from the conv5<u> </u>3 feature map of Faster R-CNN (backbone: VGG16 ). The Faster R-CNN is pre-trained over Visual Genome , similar to . To build the vocabulary, all the words from the training set are converted to lowercase and those which appear less than 4 times are omitted. Each word is represented as "one-hot" vector (binary index vector in a vocabulary). The maximum sentence number $K$ is 6 and the maximum word number in a sentence is 20 (padded where necessary). For our CAE, the convolutional filter size in the convolutional layer is set as $C_1=26$ with stride $C_2=2$. The dimensions of the embedded region-level feature and distilled topic vector are set as $D_1=1,024$ and $D_2=500$. For the two-level LSTM networks, the dimension of hidden state in each LSTM is $H=1,000$. The dimension of the hidden layer for measuring attention distribution is $D_3=512$. Three popular metrics are adopted: METEOR , CIDEr , and BLEU-4 . We compute all metrics with the released code from Microsoft COCO Evaluation Server .

#### Implementation Details.

We follow the two-phrase training recipe in to train our CAE-LSTM. For the first phrase, we set the learning rate as $1\times10^{-4}$ and the training of CAE-LSTM is thus performed by jointly leveraging the reconstruction loss in CAE and the cross entropy loss in two-level LSTM paragraph generator. Here we evaluate the model at each epoch on the validation set and select the model with the best CIDEr score as an initialization for the next training phrase. For the second phrase of self-critical training, the learning rate is set as $5\times10^{-6}$ and CAE-LSTM is further optimized with the combination of CIDEr reward and coverage reward. During inference, we adopt inference constraint in to penalize trigram repetition. The tradeoff parameter $\beta$ is set as 8 according to the validation performance. Note that Batch normalization and dropout (dropout rate: $0.5$) are applied in our experiments.

One issue of training our CAE-LSTM is to determine the order of the regions in the concatenation of region-level features as the input of CAE module. In the experiments, the region orders are determined according to the objectiveness score of each image region. We additionally studied the region orders by sorting all the regions with regard to the confidence score of bounding box, or simply using fixed random order. The CIDEr score constantly fluctuates within 0.008 when using different orders of regions for concatenation in our CAE-LSTM. The results indicate that the performance is not sensitive to the selection of region order.

```latex
\scalebox{1}{
\begin{tabular}{l|c c c}
\hline
Method                                             & C & M & B-4 \\ \hline
Image-Flat \cite{karpathy2015deep}            & 11.06  & 12.82   & 7.71   \\
Regions-Hierarchical \cite{krause2017hierarchical}                & 13.52  & 15.95  & 8.69   \\
RTT-GAN \cite{liang2017recurrent}                    & 20.36   & 18.39  & 9.21   \\
CapG-RevG \cite{chatterjee2018diverse} & 20.93   & 18.62   & 9.43   \\ \hline
LSTM-ATT                        & 20.17  & 17.72  & 8.72   \\
CAE-LSTM                           & \textbf{25.15}  & \textbf{18.82}  & \textbf{9.67}   \\\hline
\end{tabular}}
```

## Performance Comparison and Analysis

#### Quantitative Analysis.

The performances of different models on the Stanford dataset are shown in Table . Overall, the results across three evaluation metrics consistently indicate that our proposed CAE-LSTM achieves better performances against other state-of-the-art techniques including non-attention models (Image-Flat, Regions-Hierarchical, and CapG-RevG) and attention-based approaches (LSTM-ATT and RTT-GAN). Specifically, the CIDEr and METEOR scores of our CAE-LSTM can achieve 25.15% and 18.82%, which makes the relative improvement over the best competitor CapG-RevG by 20.2% and 1.1%, respectively. As expected, by additionally modeling topics/gists in an image via LSTM, Regions-Hierarchical exhibits better performance than Image-Flat which leaves inter-sentence dependency unexploited. Moreover, LSTM-ATT leads to a performance boost over Regions-Hierarchical, which directly encodes an image as a global representation by performing mean pooling over all region-level features. The results basically indicate the advantage of region-level attention mechanism in the two-level LSTM networks by learning to focus on the image regions that are most indicative to infer the next word. Most specifically, RTT-GAN and CapG-RevG by modeling reality and diversity of paragraphs with Generative Adversarial Networks and Variational Auto-Encoders, improves LSTM-ATT. However, the performances of RTT-GAN and CapG-RevG are still lower than our CAE-LSTM which exploits the inherent structure among all image regions for topic modeling in a convolutional and deconvolutional auto-encoding framework.

#### Qualitative Analysis.

Figure shows several paragraph examples generated by LSTM-ATT, CAE-LSTM and one human-annotated Ground Truth (GT) paragraph. From these exemplar results, it is easy to see that all of these paragraph generation models can produce somewhat relevant paragraphs, while our proposed CAE-LSTM can generate coherent and accurate paragraphs by learning to distill the gists/topics from an image via a convolutional auto-encoding module to guide paragraph generation. For example, compared to LSTM-ATT which repeatedly generates the phrase "holding a tennis racket" in the $2$-th and $3$-th sentences for the first image, the produced paragraph for our CAE-LSTM is more coherent to describe the diverse and accurate topics in the image. The results again verify that the convolutional auto-encoding module in our CAE-LSTM could characterize the topics better than LSTM. Moreover, by additionally measuring the coverage of objects as a reward in the self-critical training strategy, our CAE-LSTM is encouraged to produce paragraphs which cover more objects in images, leading to more descriptive paragraph with objects, e.g., "shirt" and "shorts."

#### Ablation study.

Here we study how each design in our CAE-LSTM influences the overall performance. Convolutional Encoding (**CE**) replaces the LSTM for topic modeling in LSTM-ATT with the convolutional encoder in our devised CAE module. Reconstruction Loss (**RL**) enforces the distilled topics to capture holistic and representative information via reconstruction from topics to region-level features. Self-critical training with Coverage (**SC**) further encourages global coverage of objects with a coverage reward in self-critical training. Figure shows the performance improvement by considering one more design for paragraph generation in our CAE-LSTM, over the performance of LSTM-ATT across different metrics. The results across different metrics consistently indicate that the topic modeling via convolutional encoding in CE leads to a performance boost compared to LSTM-ATT which capitalizes on LSTM to explore dependency among topics. Furthermore, by integrating reconstruction loss into the learning of topics, CE+RL exhibits better performance than CE. The results demonstrate the advantage of reconstruction from topics to region-level features in our CAE. By additionally exploiting coverage reward in self-critical training, CE+RL+SC further boosts up the performances.

#### Human Evaluation.

To better understand how satisfactory are the paragraphs generated from different methods, we conducted Turing Test to evaluate our CAE-LSTM against the baseline LSTM-ATT. In particular, six evaluators are invited and a subset of 1,000 images is randomly selected from testing set for human evaluation. We show all the evaluators once only one paragraph generated by different approach or human annotation and they are asked: can you determine whether the given paragraph has been generated by a system or by a human being? From evaluators' responses, we calculate the percentage of paragraphs that pass the Turing Test. The results of Turing Test for Human, CAE-LSTM, and LSTM-ATT are 88.5%, 39.8%, and 14.7%, which clearly show that our CAE-LSTM outperforms LSTM-ATT.

# Conclusion

We have present Convolutional Auto-Encoding plus Long Short-Term Memory (CAE-LSTM), which explores the modeling of sentence topics to boost image paragraph generation. Particularly, we study the problem from the viewpoint of topic distillation in an auto-encoding manner. To verify our claim, we have devised a purely convolutional structure, in which an encoder abstracts topics by employing convolutions over region-level image features and a deconvolutional decoder is to validate the topics through high-quality reconstruction from topics to features. A two-level LSTM-based paragraph generator is then remould, on one hand, to model dependency across sentences in a paragraph, and on the other, to generate the sentence conditioning on each learnt topic. Experiments conducted on Stanford image paragraph dataset verify our proposal and analysis. Performance improvements are observed when comparing to state-of-the-art image paragraph generation techniques.

# Acknowledgements

This work was partially supported by the National Natural Science Foundation of China (Grant No. 61732007 and U1611461).

[^1]: Corresponding Author.

[^2]: This work was performed at JD AI Research.


abstract: We present a scheme to embed molecular anions in a gas of ultracold rubidium atoms as a route towards the preparation of cold molecular ions by collisional cooling with ultracold atoms. Associative detachment as an important loss process in collisions between OH$^-$ molecules and rubidium atoms is studied. The density distribution of trapped negative ions in the multipole radiofrequency trap is measured by photodetachment tomography, which allows us to derive absolute rate coefficients for the process. We define a regime where translational and internal cooling of molecular ions embedded into the ultracold atomic cloud can be achieved.
author: J. Deiglmayr; A. Göritz; T. Best; M. Weidemüller; R. Wester
date: 2024-09-30
title: Reactive collisions of trapped anions with ultracold atoms

Cold molecules have stimulated a lot of research in the last decade since they offer new opportunities for ultrahigh precision spectroscopy, for studying ultracold gases with anisotropic interactions, and for ultracold chemistry driven by quantum mechanics . This has created a new research field that has focussed on the formation, preparation and control of cold molecules with translational, rotational, vibrational and hyperfine energies corresponding to temperatures below 1 Kelvin. Most of these studies are based on specialized preparation techniques to achieve molecular temperatures far below liquid helium temperature, based on the molecular Stark shift, magnetic moment, or the association of laser-cooled atoms.

For molecular ions the situation is advantageous, because collisional cooling is a very general method to cool ions . Translational temperatures down to few millikelvin are provided by sympathetic cooling of molecular ions with laser-cooled atomic ions trapped jointly in a radiofrequency (rf) ion trap. However, ion-ion collisions do not lead to internal quantum state cooling due to the long-range nature of the interaction. Therefore complex and molecule-specific optical pumping or state-selected photoionisation schemes have to be used for rotational state control . Alternatively, molecular ions can be cooled in collisions with cold neutral atoms, which feature much closer encounters and thereby allow for efficient cooling of rotational and vibrational states in inelastic collisions. This general cooling of all molecular degrees of freedom can be applied to molecular ions and ionic clusters of almost arbitrary size and complexity. However, it limits experiments to temperatures above about 4 K due to the limitations imposed by standard cryostats.

In this letter we present a hybrid ion-atom trap for the study of interactions of trapped molecular ions with ultracold atoms. We discuss the prospects of this trap for collisional cooling of complex molecular ions to temperatures below 1 Kelvin. In contrast to several recent experiments that study the interaction of single atomic cations with ultracold atoms , we do not employ a Paul trap, but an octupole ion trap. This strongly suppresses disturbances of ion-atom collisions due to micro-motion throughout the extended volume of the atom trap . As a spectroscopic diagnostics of the ions' density distribution, which can be related to their translational temperature, we employ single-photon photodetachment spectroscopy . This is a general method that can be applied to all negative ions and, while species- and state-selective, does not depend on the existence of a closed cycling transition as it is the case for fluorescence or absorption imaging.

Certain conditions have to be fulfilled for efficient collisional cooling of an ion by a neutral. First, as in collisions between neutrals, the system has to exhibit a large ratio of elastic or rotationally inelastic collisions to reactive collisions, which lead to a loss of the original ion. A prominent channel of such reactive collisions, which hinder cooling, is charge transfer from the ion to the neutral coolant, as observed in recent experiments involving cationic atoms . A recent detailed investigation of collision processes in the OH$^-$+ Rb system revealed very promising conditions for collisional cooling of internal and external degrees of freedom of the molecular anion by ultracold rubidium . Furthermore charge transfer from OH$^-$ to rubidium is energetically forbidden as the electron affinity of OH greatly exceeds the one of Rb (1.83eV  versus 0.486  respectively). There remains an important loss mechanism, namely associative detachment (AD) $$\textrm{OH}^- + \textrm{Rb} \rightarrow \textrm{RbOH} + \textrm{e}^- \label{eq:defAD}$$ which is exothermal by 1.4 eV . To our knowledge the cross section for this process has not been investigated yet, however early studies of associative detachment involving OH$^-$ showed significant rate constants close to the capture limit . Here we measure the rate coefficients for the reactive losses of the molecular ion OH$^-$ due to collisions with ultracold rubidium and use this result to define a regime where translational and internal cooling of molecular ions embedded into the ultracold atomic cloud can be achieved.

The setup combines an octupole rf trap for ions with a magneto-optical trap (MOT) for rubidium atoms. A schematic sketch of the trap is shown in figure . The rf trap consists of eight parallel wires arranged equidistantly on a circle with a diameter of 6 mm to which rf is applied. This rf field creates an effective potential along the radial dimension $r$ which is almost flat in the center and exhibits steep walls proportional to $r^6$ . Confinement along the trap axis is provided by two hollow end caps. A simulation of the ion density at room temperature in this trapping potential with typical parameters for rf amplitude (150 V) and end cap voltage (20 V) predicts in axial direction a Gaussian distribution with a full-width at half maximum (FWHM) of 4.0 mm and in radial direction an almost uniform distribution with a FWHM of 2.0 mm. The volume occupied by laser-cooled rubidium atoms (typical FWHM 1.0 mm) is thus fully embedded in the ion cloud. Furthermore the largest value of the effective potential in which collisions between ions and neutrals can occur corresponds to few Kelvin. This significantly limits the maximum amount of energy transferred from the electric field to the ions kinetic energy during a collision event and thus limits rf heating . It also suppresses losses due to collisions under non-adiabatic conditions in high rf amplitudes .

Ions are created from neutral precursors in a supersonic expansion through a plasma discharge, mass selected, accelerated towards the trapping region, and eventually loaded into the rf trap along the trap axis through the hollow entrance cap. During loading neon gas is fed into the trapping chamber through a pulsed valve which facilitates initial loading and thermalization of the ion cloud. During storage a background pressure of typically $5\cdot10^{-9}$ mbar is maintained. For detection the ions are accelerated through the exit endcap towards a channeltron detector. A time-of-flight tube of 500 mm length between trap and detector allows for distinguishing ions with different ratios of $q/m$.

In order to combine the ion trap with laser cooled atoms, the rf trap is mounted between two coils (free distance 50mm) which create the magnetic field gradient for the MOT. The light for laser cooling of rubidium atoms (collimated waist 1 cm, total peak light intensity 38mW/cm$^2$) passes in the z-direction (perpendicular to the ion trap axis) through the center of the magnetic coils. In the orthogonal plane the beams cross the axis of the rf trap under an angle of 30$\,^{\circ}$ each. Rubidium atoms are provided by two rubidium dispensers mounted at a distance of roughly 5 cm from the trap center where the line-of-sight from the hot dispensers to the trap center is blocked by a wire. The size and position of the MOT is monitored by recording the fluorescence of the trapped atoms via two CCD-cameras under an angle of 90$\,^{\circ}$. The magnification of the imaging system is obtained by observing the light scattered from the wires of the rf trap with well known spacings. The number of trapped rubidium samples is then calculated from the measured dimensions of the atom cloud assuming a typical atom peak density of $2\cdot10^{10}$ atoms/cm$^3$.

The spatial distribution of ions in the trap is determined via photodetachment tomography . In contrast to established imaging techniques for gases of neutral particles, such as absorption imaging or fluorescence detection, photodetachment is a single-photon effect which does not rely on closed cycling transitions and can thus be applied to any trapped anion. Also photodetachment close to threshold can be used to probe the internal state distribution of trapped anions . For photodetachment of OH$^-$ we use light from a free running diode laser at 660 nm, well above the threshold at 678 nm, focused with a cylindrical lens to create a line focus with axial width of 92 $\mu$m and almost homogeneous intensity distribution over the ion trap in vertical direction. This line focus is then scanned along the trap axis. The photodetachment process is observed via the light-induced trap loss, which is proportional to the column density of ions along the beam propagation.

Measurements of the ion distribution along the trap axis are shown in fig.  for OH$^-$ ions stored together with Rb atoms for storage times between 2 and 10s. The widths of these distributions measure the translational temperature of the ions. Specifically, as the axial confinement is well described by a harmonic potential, the ion temperature scales with the square of the axial width. The absolute kinetic temperature is estimated from a simulation of the effective trapping potential and the resulting ion density distribution to be $470\pm200$K, where the accuracy is mostly limited by counting statistics and by the precision of the electric field simulation. It can be seen that irrespective of the interaction time with the Rb atoms, no change in translational temperature is observed. We estimate our method to be sensitive to relative changes exceeding 20%. With the given Rb atom number, OH$^-$ ion distribution, and the Langevin rate-coefficient for OH$^-$-Rb collisions we estimate a collision rate on the order of 8 s$^{-1}$, which should suffice for a significant cooling of the trapped ions within about a second. The absence of cooling is attributed to competing thermalization, most likely due to collisions with the residual background gas. While the latter is about two orders of magnitude lower in density, it overlaps the ions throughout the trap whereas the Rb atoms fill only a small volume in the trap center. In particular resonant proton transfer with H$_2$O molecules, known to be a significant fraction of the residual gas due to its use in the ion source, would lead to room temperature OH$^-$ in just a single collision. The rate for this process is estimated to be about 0.4 s$^{-1}$ . Following from simple kinematic arguments, elastic OH$^-$-Rb collisions should lead to an exponential decay of the ion temperature with a time constant of 3 s, thus on the same time scale as the heating due to collisions with background gas.

Strong losses of OH$^-$ ions from the trap are induced by the presence of ultracold rubidium atoms. Figure  shows the number of trapped OH$^-$ ions during trapping. When the cooling laser is tuned to the blue of the cycling transition, and thus no rubidium atoms are trapped, we observe a loss of OH$^-$ ions with a time constant of $47\pm3$ s. When the cooling laser is detuned to the red of the cooling transition by 2.2$\Gamma$, about $4\cdot10^{7}$ atoms are trapped in the center of the ion cloud and induce an ion loss with a rate of $0.329\pm0.018$ 1/s. Due to the large depth of the ion trap and the low temperature of the rubidium atoms the kinetic energy transfer in elastic collisions can not lead to significant ion losses. Thus the observed losses can be attributed dominantly to inelastic collisions between OH$^-$ ions and ultracold $^{85}$Rb atoms. As pointed out in the introduction, the process of associative detachment (eq. ) is energetically allowed. Another possible reaction channel is $\textrm{OH}^-+\textrm{Rb}^*\rightarrow \textrm{OH}^-+\textrm{Rb}+E_\textrm{kin}$, a collision between a rubidium atom in the excited state and OH$^-$, potentially releasing 1.6 eV of energy which would suffice to eject the ion out of the trap. At a given time roughly half of the atoms in the MOT are in the excited state, thus both processes might contribute. In order to clarify the influence of excited rubidium atoms in the MOT we vary the cooling laser detuning and thus the excited state fraction from 25% to 44%. The observed loss rates shown in figure  exhibit a stronger correlation with the absolute number of atoms than with the number of excited atoms, which suggests a dominant contribution from AD in the ground state.

The rate constant for the observed inelastic losses is derived from the independent determination of ion and atom distributions. As shown above we can assume that the rubidium sample is fully embedded in the ion cloud and that the ion density is roughly constant over the MOT region (a detailed numerical simulation confirmed this assumption to be valid with 10% of the final rate constant). We took care to perform the loss measurements under conditions where a correlation between number of ions and ion density due to Coulomb repulsion can be excluded. The rate constant for inelastic losses $K_\textrm{inel}$ is then given by $K_\textrm{inel} = \frac{R_\textrm{inel}}{ N_{\textrm{Rb}} \rho_\textrm{{0,ion}}}$ where $R_\textrm{inel}$ is the measured rate of inelastic losses, $N_{\textrm{Rb}}$ is the total number of rubidium atoms and $\rho_\textrm{{0,ion}}$ is the peak value of the normalized ion distribution. $N_{\textrm{Rb}}$ is determined as described above from the analysis of fluorescence images, $\rho_\textrm{{0,ion}}$ is calculated from a fit of the trapping potential, which was derived from a numerical simulation, to the experimentally measured widths in axial and radial directions. Performing this analysis for the measurement shown in figure  we find an inelastic rate coefficient of $(2^{+2}_{-1})\cdot10^{-10}$  where the uncertainty is mostly caused by a systematic uncertainty in the determination of the atom number. This rate coefficient holds for a relative temperature of the OH$^-$-Rb system of $400\pm200$K, based on the estimated kinetic temperature of OH$^-$ and the negligible kinetic temperature of the Rb atoms. This loss rate is significantly smaller than the rate coefficient of $4.3\cdot10^{-9}$  given by Langevin theory for collisions between OH$^-$ and Rb 5S$_{1/2}$ atoms. As AD leads to losses but not to a heating of the remaining ensemble, cooling through elastic collisions should still be possible if the interaction time between ion and neutrals can be limited, *e. g.* by shuttering the MOT after a certain interaction time.

From our observations we derive the following protocol for collisional cooling of anions by ultracold atoms in a rf multipole trap. First, as cooling requires a sufficient number of elastic collisions before inelastic losses can occur, negative ions with a sufficiently small associative detachment rate, suitably an order of magnitude smaller than for OH$^-$+Rb, are required. AD will occur for all anions for which an ion pair state with the cation of the neutral coolant exists and the ion pair can approach close enough to overcome the ionization potential of the coolant atom and couple to the electron continuum. For example, hydrated OH$^-$(H$_2$O)$_\textrm{n}$ clusters have been found to generally show significantly reduced AD rates  which is explained by the stronger screening of the negative charge by the surrounding water molecules. Also for more delocalized electron wave functions in other polyatomic molecular ions, such as benzene anions or SF$_6^-$, we expect lower AD rates than in the present study. Second, residual collision rates have to be limited far below 0.4 s$^{-1}$ and the rf amplitude has to be reduced following the cooling process to further suppress heating. Third, the interaction between the ions and the atoms has to be limited to a time comparable to the time scale for AD to avoid excessive losses. Furthermore, electronically excited atoms should be removed from the trap using a dark spontaneous force optical trap  to exclude loss processes due to these excited atoms. Following these steps, collisional cooling of a large class of molecular ions to temperatures far below liquid helium temperature seems to become feasible.


abstract: In this paper we present a method for algorithmic melody generation using a generative adversarial network without recurrent components. Music generation has been successfully done using recurrent neural networks, where the model learns sequence information that can help create authentic sounding melodies. Here, we use DCGAN architecture with dilated convolutions and towers to capture sequential information as spatial image information, and learn long-range dependencies in fixed-length melody forms such as Irish traditional reel.
author: Antonina Kolokolova, Mitchell Billard, Robert Bishop, Moustafa Elsisy & Zachary Northcott
Department of Computer Science
Memorial University of Newfoundland
St. John's, NL, Canada
`{kol,mlb238,r.bishop,mmatelsisy,email@example.com` Laura Graves & Vineel Nagisetty
Department of Electrical and Computer Engineering
University of Waterloo
Waterloo, ON, Canada
`{laura.graves,email@example.com` Heather Patey
Fiddler and Folk-dancer
St. John's, NL, Canada
`email@example.com`
bibliography: gans.bib
title: GANs & REELS
  Creating Irish Music Using a Generative Adversarial Network

# Introduction

Algorithmic music composition is almost as old as computers themselves, dating back to the 1957 "Illiac suite" . Since then, automated music composition evolved with technology, progressing from the first rule-and-randomness based methods to the sophisticated tools made possible by modern-day machine learning (see and for detailed surveys on history and state of the art of algorithmic music composition).

One of the first machine learning (ML) approaches to music generation was , who used the common notion of entropy as a measurement to build what they termed a multiple viewpoint system. Standard feedforward neural networks have difficulties with sequence based information such as music. Predicting the next note of a piece, when only based on the current note, does not account for long-range context or structure (such as key and musical sections) which help give coherence to compositions. As music is traditionally represented as sequences of notes, recurrent neural networks are a natural tool for music (especially melody) generation, and multiple groups used RNNs fairly successfully for a variety of types of music. used a sequential model for composition in 1989, and used the adapted LSTM structure to successfully generate music that had both short-term musical structure and contained the higher-level context and structure needed. Subsequently, there have been a number of RNN-based melody generators . Other approaches such as MidiNet by , though not RNNs, also leveraged the sequential representation of music.

Using an RNN architecture provides a lot of flexibility when generating music, as an RNN has the ability to generate pieces of varying length. However, in some styles of music this is not as desired. This is true of traditional Irish music - and especially their jigs and reels. These pieces have a more rigid format where the varying length can prevent capturing the interplay between the phrases of the piece.Finding jigs and reels to train on was made easy by an excellent database of Irish traditional melodies in ABC notation (a text based format), publicly available at TheSession . Several RNN-based generators were trained on the melodies from TheSession, most notably Sturm et al. , as well as .

It is natural to view music, and in particular melodies, as sequential data. However, to better represent long-term dependencies it can be useful to present music as a two-dimensional form, where related parts and occurrences of long patterns end up aligned. This benefit is especially apparent in forms of music where a piece consists of a well-defined, fixed-length components, such as reels in Irish music. These components are often variations on the same theme, with specific rules on where repeats vs. changes should be introduced. Aligning them allows us to use vertical spatial proximity to capture these dependencies, while still representing the local structure in the sequence by horizontal proximity.

In this project, we leverage such two-dimensional representation of melodies for non-sequential melody generation. We focus on melody generation using deep convolutional generative adversarial networks (DCGANs) without recurrent components for fixed-format music such as reels. This approach is intended to capture higher-level structures in the pieces (like sections), and better mimic interplay between smaller parts (musical motifs). More specifically, we use dilations of several semantically meaningful lengths (a bar or a phrase) to further capture the dependencies.

Dilated convolutions, introduced by , have been used in a number of applications over the last several years to capture long-range dependencies, notably in WaveNet . However, they are usually combined with some recurrent component even when used for a GAN-based generation such as in or .

Not all techniques applicable to images can be used for music, however: pooling isn't effective, as the average of two pitches can create notes which fall outside of the 12-semitone scale (which is the basis the major and minor scale as well as various modes). This is reflected in the architecture of our discriminator, with dilations and towers as the main ingredients.

# Structure of the data: traditional Irish tunes

In spite of emigration and a well-developed connection to music influences from its neighbouring cultures - especially Scotland - Irish traditional music has kept many of its elements and itself influenced many forms of music. Its influences can be found in the folk music of Newfoundland and Quebec and in Bluegrass from the Appalachian region of United States.

A significant part of the traditional Irish music is instrumental dance music, where a tune following a set format is played on a solo instrument, such as a fiddle, tin whistle, accordion or flute. The core of the piece is usually monophonic, so while harmonies and embellishments can be introduced during performances, they are considered extra and not usually encoded when a piece is transcribed. Generally the melody consists of two distinct 8-bar parts, which in turn consist of two 4-bar phrases each. Usually, phrases represent a pattern which repeats with variations from phrase to phrase, with one (most often third) phrase deviating most from the pattern, and the final phrase returning to it. When performing, each part is repeated twice; sometimes the second repeat of a part is different enough that it is transcribed separately, making the transcription from 16 to 32 bars long. The whole piece is usually performed three (or more) times, especially when accompanying a dance.

The two most common types of dances are reels and jigs: the reels have time signature of 4/4 (that is, each bar contains 4 units, where each unit is a quarter note), whereas a jig has a time signature 6/8 (that is, each bar contains 6 units, where each unit is an eighth note).

Many Irish tunes are in a major key, with D major being most common. However, a number of tunes use the Dorian and Mixolydian modes (starting with 2nd or 5th note with respect to the relative major scale, respectively). The natural minor key (Aeolian mode) is less common. Overall, it is uncommon for a tune to incorporate notes not in its key and to go outside of a 2-octave range (either of these would make it hard to play on a tin whistle).

Traditionally, musicians learned tunes by ear. However, the relative simplicity of the structure of Irish tunes has lent itself well to a text-based representation, culminating in the ABC notation developed by Chris Walshaw in the 1970s. There, the key and default length of a note are specified in a header (together with the title, time signature, etc), and the melody is then encoded by letters: A-G ,where C represents middle C, and a-g, which represents the notes an octave higher than A-G. Additionally there are modifiers which raise and lower notes by an octave, as well as charaters which represent sharps and flats. Like in the staff notation, sharps and flats which are implied by the key are not explicitly written into the piece. These letters are also appended with duration modifiers: for example, if ABC header sets the default note length as 1/8, then D represents an eighth note (which is a D), D2 represents a quarter note, and D/2 represents a sixteenth note. For readability, bar separators $|$ and spacing are often included: for example, the sequence "d2dA BAFA$|$ABdA BAFA$|$ABde f2ed$|$Beed egfe" encodes the first phrase of a reel called "Merry Blacksmith", where the key was specified to be D major and the default note length is set to 1/8.

Nowadays, ABC notation is widely used, especially for Irish folk music, with additional features for representing ornaments, repetitions, and so on. In particular, this is the default representation for tunes in databases such as TheSession which we used for this project.

## Preprocessing TheSession dataset

TheSession dataset contains over 31,000 tunes, which contain 10,000 reels in a major key. However, after removing improperly formatted tunes, tunes with features we did not want to generate (such as triplets), and tunes that had more than 16 bars we were left with 820 reels, which we used as our training data.

The samples in ABC notation were then converted into numerical vectors, with 16 numbers per bar (so 256 numbers total), corresponding to 16 possible notes (normalized midi values). This encoding did not preserve information about the duration of each note, only their (normalized) midi value. However, we found that with simple postprocessing (converting a sequence of occurrences of the same note into a note of a longer duration, starting with an occurrence on a beat) we were able to recover the tunes with not much more variation than would be introduced in performances.

We have sourced music from The Session website (https://thesession.org/), hosted by Jeremy Keith. We chose this website because it maintains a frequently updated repository of Irish music stored using ABC notation and Jeremy graciously provided us with a copy of The Session's ABC notated data and permission to use it. After cleaning and and filtering the data (for example, we only trained on 4/4 tunes and converted to D major) we were left with a training set of approximately 6,000 tunes.

One of the main ideas behind our encoding was representing the resulting vectors of 256 values as a 64x4 image, with midi values corresponding to pixel values. The resulting image had a line for each the phrase of the tune, and the vertical alignment of the corresponding notes and bars let us exploit the long-range dependencies among the phrases using their spatial proximity. Intuitively, this is akin to building a mental map of a space explored by touch: though the touch information is sequential, the resulting map can be two or three-dimensional, with significant amount of information encoded by proximity in the second and third dimensions.

For each tune we used 16 bars, with each bar divided into $16^{th}$s. By converting each note's numeric value to the range $[-1,1]$ and stacking each group of 4 bars on top of each other, we converted each written tune into a 64x4 real valued vector (4 rows, where each row is a phrase, with 4 bars per row). This representation lets the convolutions learn a simulation of sequence based data - notes following each other are represented by horizontal proximity and notes in the same position (or beat) in the related 4 bar phrases are represented by vertical proximity. In this way our GAN is able to imitate sequential learning through spatial learning, which we believe is a novel approach to sequential learning.

# GAN Architecture

Traditionally, sequential data is learned using some variation of RNNs. Simple implementations of RNNs (standalone GRU and LSTM cells) tend to struggle with capturing long-distance dependencies; a popular solution to this issue is to use bi-directional RNNs (that is, a pair of layers that run in opposite directions) with an activation mechanism (using an auxiliary trainable network to better relate elements of the output sequence to those of the input sequence) so that information for multiple sections of the sequence can be captured .

A similar effort at capturing long-distance dependencies has been applied to Convolutional Neural Networks (CNNs) too. Yu et al. have proposed *dilated convolutions* as a solution to the problem: in a dilated convolution, the convolution kernel is not a contiguous block, but rather subsections of the kernel are separated by some number of pixels from each other. As a result, the convolution operation is not limited to the immediate neighbourhood, and long-distance dependencies are made available to the kernel . While attention-based and bi-directional RNNs have proven to be quite successful at capturing long-distance dependencies, we believe that there are a few notable benefits for being able to use a DCGAN for generating sequential data, as opposed to an RNN:

-   DCGANs that incorporate dilations allow for domain-specific heuristics to be passed to the neural network, by means of the dilation rates and the convolution kernel dimensions. This allows for a whole new dimension of flexibility and fine-tuning that is not available through RNNs.

-   A DCGAN yields *both* a discriminator and a generator, as opposed to just a generator.

-   Unlike an RNN, a GAN does not require a meaningful or domain-specific seed to be passed as an input to the generator; a vector of random noise can be used instead, making it easier to generate novel outputs from the generator.

GANs have traditionally worked well with pictures, but to our knowledge existing music generation with GANs either relies on RNNs in the discriminator, as in SeqGAN or generates the melody in a sequential bar-after-bar process as in MidiNet .

## Model

### Discriminator

The discriminator starts off with a 6-tower convolution network (that is, 6 convolutions run side-by-side, as opposed to being dependent on each other). We chose to use dilated filters on 5 of the towers to capture relative information between bars and phrases (we shall refer to this as the *global context*), with the remaining layer being a contiguous 2x9 convolution to also capture the structure of the immediate neighbourhood (*local context*). Having multiple towers each learning different aspects gives us versatility and helps avoid the "tunnel vision" nature of other generative music models. This can be viewed in Figure .

All the towers use 32 convolution filters, and use zero-padding for edge pixels that might not have a sufficient neighbourhood to be covered by each filter. The outputs of the convolutions from each tower are then stacked horizontally, and passed to a second (regular) layer of convolution made of 64 filters, using a 3x3 kernel, a 2x2 stride and with the convolution being only applied to pixels that have a sufficient neighbourhood for the convolution filter. The output of this layer is then flattened, and passed to a dense layer consisting of 1024 neurons, which is in turn followed by a sigmoid neuron that generates the prediction of the discriminator. No batch normalization is applied to any of the layers on the discriminator.

### Generator

Unlike the discriminator, the generator does not make use of dilations or towers. We start off with a dense layer made of 32x2x256 (where the last dimension is the number of filters). The dense layer is then reshaped into 256 filters of size 32x2, and passed through two layers of deconvolution, each with halving the number of filters, while using a stride of 1x1 and a kernel size of 2x5. A last deconvolution layer (also 2x5 in kernel size) is used to bring down the filters into one image, with a stride of 2x2 to return to the original image dimension of 64x4; this layer also uses a tanh activation to place the output values in the same range as the training dataset. All layers in the generator use batch normalization.

## Post-Processing

After our GAN has generated music in the form of a 64x4 matrix, we map the real values to notes in ABC notation, rounding to the nearest note in the D major key. We then take each beat and merge notes with the same pitch together, so a sequence of four 16th notes C starting from a 1st, 5th, 9th or 13th note in a bar becomes a quarter note C instead. The resulting ABC notated music can then be converted into sheet music or played with midi generators.

# Comparison with other melody generators

Our approach is similar to the model by as both train on text-based musical notation and aim to learn Irish traditional instrumental music that has distinct structure. The models in Magenta are considered to be state of the art in music generation at the moment. For this reason, we compared our samples with the Folk-RNN and Magenta models - specifically the MelodyRNN model in Magenta - training using the same dataset of 820 curated reels. We generated an equivalent number of samples from each model and compared the samples. Since our goal is for the GAN to learn the global patterns, we use measures that highlight similarity in structure.

The first metric we chose was the normalized mean Fréchet distance. Fréchet distance is a measure of similarity between vectors which prioritizes the ordering and magnitude of the values in those vectors. Since the order and distribution of notes are important for a tune to be considered structurally similar to another, this metric is ideal for us. We normalize this Fréchet distance so we can view the changes between the phrases better.

A tune in a AABB format would have a lower Fréchet distance between phrases (1,2) and (3,4) but a higher distance between (2,3). A tune in the ABAB format, by contrast, would have a closer distance between (1,3) and (2,4) but not (1,2) and (3,4). Here we compute the average Fréchet distance between the phrases for a given distribution and compare them with the training data to determine how well each model has learnt the global structure. We normalize this Fréchet distance so we can view the changes between the phrases better.

The results of this comparison are found in Figure . Each group records the normalized Fréchet distance between corresponding combinations of phrases i.e the first group highlights the normalized mean Fréchet distance between phrase 1 and 2 for all the distributions. We can see that our generated samples compare favorably with the Folk-RNN model and both exhibit similar structure to the training set. It is important to note that the MelodyRNN used here is the default model without the attention mechanism. Interestingly, the distance between phrases (3-4) is lower and phrases (1,3) is higher in our samples - indicating that our tunes are more in the AABB format. Listening to the samples also confirms this findings.

Another measure to visualize similarity between distributions is the t-distributed stochastic neighbourhood embedding (t-SNE) algorithm developed by . This algorithm groups points that are similar in higher dimension closer together in low dimensions - making it useful to visualize similarity between the tunes generated by both models along with the training data. Here, we set the tunable parameter perplexity to 50. This increases the importance of global similarity between vectors.

Figure shows distributions of the training data and samples generated by our model, FolkRNN and Magenta after being trained on the same data. Since t-SNE is stochastic and can generate different visualizations for each run, we display visualization produced by two runs of t-SNE algorithm. It can be observed that most of the time (\>75%), the tunes generated by our model seem to lie within the distribution of the training data, although they only cover a subset of the training data. Both RNN models seem to mimic a slightly different distribution, where we can think of the tunes generated by Magenta as a subset of the FolkRNN ones, with FolkRNN generating by far the most diverse set of samples.

A third metric is looking at the distribution of notes in the samples. Figure shows the frequency of distribution of notes as midi key values in the samples. As music theory would suggest, the tonic of the key (in this case, the midi value corresponding to D) is the most common, with the 3rd and 5th of the key more frequent than the rest of the scale. Both our model and FolkRNN created a set of samples which mimics this distribution of the notes in the key, however Magenta seems to have a more uniform distribution of note frequencies.

The tunes generated by our model seem to compare favourably with the dataset and the Folk-RNN model, with the Magenta model generating more notes in the lower octaves. Since all the tunes in our training set have been transposed the D Major scale - the keys in the tunes generated by all three models fall in the scale.

Overall, these metrics show that music generated by our DCGAN is comparable to melodies generated by RNNs on the same symbolic data. The use of dilations allows for our model to learn global tune structure.

# Conclusion

Converting sequential data into a format which implicitly encodes temporal information as spatial information is an effective way of generating samples of such data as whole pieces. Here, we explored this approach for melody generation of fixed-length music forms, such as an Irish reel, using non-recurrent architecture for the discriminator CNN with towers and dilations, as well as a CNN for the GAN itself.

One advantage of this approach is that the model learns global and contextual information simultaneously, even with a small model. LSTMs and other approaches need a much larger model to be able to learn both the contextual neighboring note sequences and global melody structure.

In future work, we would like to introduce boosting in order to capture the structure of the distribution more faithfully, and increase the range of pieces our model can generate. Natural extensions of the model would be to introduce multiple channels to capture durations better (for example, as in ), and add polyphony (ie, using some form of piano roll representation). Another direction could be to experiment with higher-dimensional representation of the sequence data, to better capture several types of dependencies simultaneously. Additionally, it would be interesting to apply it to other kinds of fixed-length sequential data with long-range patterns.


abstract: Programs to solve so-called constraint problems are complex pieces of software which require many design decisions to be made more or less arbitrarily by the implementer. These decisions affect the performance of the finished solver significantly . Once a design decision has been made, it cannot easily be reversed, although a different decision may be more appropriate for a particular problem.
  .
  We investigate using machine learning to make these decisions automatically depending on the problem to solve with the alldifferent constraint as an example. Our system is capable of making non-trivial, multi-level decisions that improve over always making a default choice.
author: Lars Kotthoff; Ian Gent; Ian Miguel
bibliography: alldiff.bib
title: Using machine learning to make constraint solver implementation decisions

# Introduction

Constraints are a natural, powerful means of representing and reasoning about combinatorial problems that impact all of our lives. Constraint solving is applied successfully in a wide variety of disciplines such as aviation, industrial design, banking, combinatorics and the chemical and steel industries, to name but a few examples.

A *constraint satisfaction problem* (CSP ) is a set of decision variables, each with an associated domain of potential values, and a set of constraints. An assignment maps a variable to a value from its domain. Each constraint specifies allowed combinations of assignments of values to a subset of the variables. A *solution* to a CSP is an assignment to all the variables that satisfies all the constraints. Solutions are typically found for CSPs through systematic search of possible assignments to variables. During search, constraint *propagation* algorithms are used. These propagators make inferences, usually recorded as domain reductions, based on the domains of the variables constrained and the assignments that satisfy the constraints. If at any point these inferences result in any variable having an empty domain then search backtracks and a new branch is considered.

When implementing constraint solvers and modelling constraint problems, many design decision have to be made – for example how much propagation to do, what data structures to use to enable the solver to backtrack and when and how often to check if a constraint is satisfied and which values could be removed. These decisions have so far been made mostly manually. Making the "right" decision often depends on the experience of the person making it.

We approach this problem using machine learning. Given a particular problem, we want to decide *automatically* which design decisions to make. This improves over the current state of the art in two ways. First, we do not require humans to more or less arbitrarily decide on something they may not have any experience with. Second, we can change design decisions for particular problems.

We demonstrate that we can approach machine learning as a "black box" and use generic techniques to increase the performance of the learned classifiers. The result is a system which is able to dynamically decide which implementation to use by looking at an unknown problem. The decision made is in general better than simply relying on a default choice and enables us to solve constraint problems faster.

# Background

We are addressing an instance of the Algorithm Selection Problem , which, given variable performance among a set of algorithms, is to choose the best candidate for a particular problem instance. Machine learning is an established method of addressing this problem . Particularly relevant to our work are the machine learning approaches that have been taken to configure, to select among, and to tune the parameters of solvers in the related fields of mathematical programming, propositional satisfiability (SAT), and constraints.

Multi-tac  configures a constraint solver for a particular instance distribution. It makes informed choices about aspects of the solver such as the search heuristic and the level of constraint propagation. The Adaptive Constraint Engine  learns search heuristics from training instances. SATenstein  configures stochastic local search solvers for solving SAT problems.

An algorithm *portfolio* consists of a collection of algorithms, which can be selected and applied in parallel to an instance, or in some (possibly truncated) sequence. This approach has recently been used with great success in SATzilla  and CP Hydra . In earlier work Borrett *et al*  employed a sequential portfolio of constraint solvers. Guerri and Milano  use a decision-tree based technique to select among a portfolio of constraint- and integer-programming based solution methods for the bid evaluation problem. Similarly, Gent *et al*  investigate decision trees to choose whether to use lazy constraint learning  or not.

Rather than select among a number of algorithms, it is also possible to learn parameter settings for a particular algorithm. Hutter *et al*  apply this method to local search. Ansotegui *et al*  employ a genetic algorithm to tune the parameters of both local and systematic SAT solvers.

The *alldifferent* constraint requires all variables which it is imposed on to be pairwise alldifferent. For example alldiff($x_1,x_2,x_3$) enforces $x_1\neq x_2$, $x_1\neq x_3$ and $x_2\neq x_3$.

There are many different ways to implement the alldifferent constraint. The naïve version decomposes the constraint and enforces disequality on each pair of variables. More sophisticated versions (e.g. ) consider the constraint as a whole and are able to do more propagation. For example an alldifferent constraint which involves four variables with the same three possible values each cannot be satisfied, but this knowledge cannot be derived when just considering the decomposition into pairs of variables.

Even when the high-level decision of how much propagation to do has been made, a low-level decision has to be made on how to implement the constraint. For an in-depth survey of the decisions involved, see .

We make both decisions and therefore combine the selection of an algorithm (the naïve implementation or the more sophisticated one) and the tuning of algorithm parameters (which one of the more sophisticated implementations to use).

# The benchmark instances and solvers

We evaluated the performance of the different versions of the alldifferent constraint on two different sets of problem instances. The first one was used for learning classifiers, the second one only for the evaluation of the learned classifiers.

The set we used for machine learning consisted of 277 benchmark instances from 14 different problem classes. It has been chosen to include as many instances as possible whatever our expectation of which version of the alldifferent constraint will perform best.

The set to evaluate the learned classifiers consisted of 1036 instances from 2 different problem classes that were not present in the set we used for machine learning. We chose this set for evaluation because the low number of different problem classes makes it unsuitable for training.

Our sources are Lecoutre's XCSP repository  and our own stock of CSP instances. The reference constraint solver used is Minion  version 0.9 and its default implementation of the alldifferent constraint `gacalldiff`. The experiments were run with binaries compiled with g++ version 4.4.3 and Boost version 1.40.0 on machines with 8 core Intel E5430 2.66GHz, 8GB RAM running CentOS with Linux kernel 2.6.18-164.6.1.el5 64Bit.

We imposed a time limit of 3600 seconds for each instance. The total number of instances that no solver could solve solve because of a time out was 66 for the first set and 26 for the second set. We took the median CPU time of 3 runs for each problem instance.

Adapting the implementation decision to the problem instead of always choosing a standard implementation has the potential of achieving speedups of up to 2 on the first set of benchmark instances and speedups of up to 1.2 on the second set. A speedup of 2 means that the fastest version of alldifferent is twice as fast as the default version.

We ran the problems with 9 different versions of the alldifferent constraint – the naïve version which is equivalent to the binary decomposition and 8 different implementations of the more sophisticated version which does more propagation (see ). The amount of search done by the 8 versions which implement the more sophisticated algorithm was the same.

The instances, the binaries to run them, and everything else required to reproduce our results are available from the primary author on request.

# Instance attributes and their measurement

We measured 37 attributes of the problem instances. They describe a wide range of features such as constraint and variable statistics and a number of attributes based on the primal graph. The primal graph $g=\langle V,E\rangle$ has a vertex for every CSP variable, and two vertices are connected by an edge iff the two variables are in the scope of a constraint together.

Edge density

:   The number of edges in $g$ divided by the number of pairs of distinct vertices.

Clustering coefficient

:   For a vertex $v$, the set of neighbours of $v$ is $n(v)$. The edge density among the vertices $n(v)$ is calculated. The clustering coefficient is the mean average of this local edge density for all $v$ . It is intended to be a measure of the local cliqueness of the graph. This attribute has been used with machine learning for a model selection problem in constraint programming .

Normalised degree

:   The normalised degree of a vertex is its degree divided by $|V|$. The minimum, maximum, mean and median normalised degree are used.

Normalised standard deviation of degree

:   The standard deviation of vertex degree is normalised by dividing by $|V|$.

Width of ordering

:   Each of our benchmark instances has an associated variable ordering. The width of a vertex $v$ in an ordered graph is its number of *parents* (i.e. neighbours that precede $v$ in the ordering). The width of the ordering is the maximum width over all vertices . The width of the ordering normalised by the number of vertices was used.

Width of graph

:   The width of a graph is the minimum width over all possible orderings. This can be calculated in polynomial time , and is related to some tractability results. The width of the graph normalised by the number of vertices was used.

Variable domains

:   The quartiles and the mean value over the domains of all variables.

Constraint arity

:   The quartiles and the mean of the arity of all constraints (the number of variables constrained by it), normalised by the number of constraints.

Multiple shared variables

:   The proportion of pairs of constraints that share more than one variable.

Normalised mean constraints per variable

:   For each variable, we count the number of constraints on the variable. The mean average is taken, and this is normalised by dividing by the number of constraints.

Ratio of auxiliary variables to other variables

:   Auxiliary variables are introduced by decomposition of expressions in order to be able to express them in the language of the solver. We use the ratio of auxiliary variables to other variables.

Tightness

:   The tightness of a constraint is the proportion of disallowed tuples. The tightness is estimated by sampling 1000 random tuples (that are valid w.r.t. variable domains) and testing if the tuple satisfies the constraint. The tightness quartiles and the mean tightness over all constraints is used.

Proportion of symmetric variables

:   In many CSPs, the variables form equivalence classes where the number and type of constraints a variable is in are the same. For example in the CSP $x_1 \times x_2 = x_3, x_4 \times x_5 = x_6$, $x_1, x_2, x_4, x_5$ are all indistinguishable, as are $x_3$ and $x_6$. The first stage of the algorithm used by Nauty  detects this property. Given a partition of $n$ variables generated by this algorithm, we transform this into a number between 0 and 1 by taking the proportion of all pairs of variables which are in the same part of the partition.

Alldifferent statistics

:   The size of the union of all variables in an alldifferent constraint divided by the number of variables. This is a measure of how many assignments to all variables that satisfy the constraint there are. We used the quartiles and the mean over all alldifferent constraints.

In creating this set of attributes, we intended to cover a wide range of possible factors that affect the performance of different alldifferent implementations. Wherever possible, we normalised attributes that would be specific to problem instances of a particular size. This is based on the intuition that similar instances of different sizes are likely to behave similarly. Computing the features took 27 seconds per instance on average.

# Learning a problem classifier

Before we used machine learning on the set of training instances, we annotated each problem instance with the alldifferent implementation that had the best performance on it according to the following criteria. If the naïve alldifferent implementation took less CPU time than all the other ones, it was chosen, else the implementation which had the best performance in terms of search nodes per second was chosen. As all implementations except the naïve one explore the same search space, nodes per second is a more reliable measure of performance than only CPU time. If no solver was able to solve the instance, we assigned a "don't know" annotation.

We used the WEKA  machine learning software through the R interface to learn classifiers. We used almost all of the WEKA classifiers that were applicable to our problem – algorithms which generate decision rules, decision trees, Bayesian classifiers, nearest neighbour and neural networks. Our selection is broad and includes all major machine learning methodologies. The specific classifiers we used are `BayesNet`, `BFTree`, `ConjunctiveRule`, `DecisionTable`, `FT`, `HyperPipes`, `IBk`, `J48`, `J48graft`, `JRip`, `LADTree`, `MultilayerPerceptron`, `NBTree`, `OneR`, `PART`, `RandomForest`, `RandomTree`, `REPTree` and `ZeroR`, all of which are described in .

The problem of classifying problem instances here is different to normal machine learning classification problems. We do not particularly care about classifying as many instances as possible correctly; we rather care that the instances that are important to us are classified correctly. The higher the potential gain is for an instance, the more important it is to us. If, for example, the difference between making the right and the wrong decision means a difference in CPU time of 1%, we do not care whether the instance is classified correctly or not. If the difference is several orders of magnitude on the other hand, we really do want this instance to be classified correctly.

Based on this observation, we decided to measure the performance of the learned classifiers not in terms of the usual machine learning performance measures, but in terms of misclassification penalty. The misclassification penalty is the additional CPU time we require to solve a problem instance when choosing to solve it with a solver that is not the fastest one. If the selected solver was not able to solve the problem, we assumed the timeout of 3600 seconds minus the CPU time the fastest solver took to be the misclassification penalty. This only gives the lower bound, but the correct value cannot be estimated easily.

We furthermore decided to assign the maximum misclassification penalty (or the maximum possible gain) as a cost to each instance as follows. To bias the WEKA classifiers towards the instances we care about most, we used the common technique of duplicating instances . Each instance appeared in the new data set $1+\left\lceil\log_2(\mathtt{cost})\right\rceil$ times. The particular formula to determine how often each instance occurs was chosen empirically such that instances with a low cost are not disregarded completely, but instances with a high cost are much more important. Each instance will be in the data set used for training the machine learning classifiers at least once and at most 13 times for a theoretic maximum cost of 3600.

To achieve multi-level classification, each individual classifier below consists of a combination of classifiers. First we make the decision whether to use the alldifferent version equivalent to the binary decomposition or the other one, then, based on the previous decision, we decide which specific version of the alldifferent constraint to use.

Table  shows the total misclassification penalty for all classifiers with and without instance duplication on the first data set. It clearly shows that our cost model improves the performance significantly in terms of misclassification penalty for almost all classifiers.

Misclassification penalty for all classifiers with and without instances duplicated according to their cost in the training data set. All numbers are rounded.

Misclassification penalty for all classifiers with and without instances duplicated according to their cost in the training data set. All numbers are rounded.

misclass. penalty [s]

classifier
all equal
cost model

BayesNet
1494
3.9

BFTree
8.4
1.1

ConjunctiveRule
2300
1433

DecisionTable
249
1.6

FT
248
1.2

HyperPipes
867
867

IBk
109
109

J48
8.2
1.2

J48graft
8.2
1.2

JRip
283
1.3

Misclassification penalty for all classifiers with and without instances duplicated according to their cost in the training data set. All numbers are rounded.

misclass. penalty [s]

classifier
all equal
cost model

LADTree
8.4
6.5

MultilayerPerceptron
249
8.5

NBTree
9
1.3

OneR
69.5
409

PART
5.9
1

RandomForest
41.9
0.9

RandomTree
1
1

REPTree
1099
10.8

ZeroR
2304
2304

For each classifier, we did stratified $n$-fold cross-validation – the original data set is split into $n$ parts of roughly equal size. Each of the $n$ partitions is in turn used for testing. The remaining $n-1$ partitions are used for training. In the end, every instance will have been used for both training and testing in different runs . Stratified cross-validation ensures that the ratio of the different classification categories in each subset is roughly equal to the ratio in the whole set. If, for example, about 50% of all problem instances in the whole data are solved fastest with the naïve implementation, it will be about 50% of the instances in each subset as well.

There are several problems we faced when generating the classifiers. First, we do not know which one of the machine learning algorithms was suited best for our classification problem; indeed we do not know whether the features of the problem instances we measured are able to capture the factors which affect the performance of each individual implementation at all. Second, the learned classifiers could be overfitted. We could evaluate the performance of each classifier on the second set of problem instances and compare it to the performance during machine learning to assess whether it might be overfitted. Even if we were able to reliably detect overfitting this way, it is not obvious how we would change or retrain the classifier to remove the overfitting. Instead, we decided to use all classifiers – for each machine learning algorithm the $n$ different classifiers created during the $n$-fold cross-validation and the classifiers created by each different machine learning algorithm.

We decided to use three-fold cross-validation as an acceptable compromise between trying to avoid overfitting and time required to compute and run the classifiers. We combine the decisions of the individual classifiers by majority vote (bagging and boosting).

Table  shows the overall performance of our meta-classifier compared to the best and worst individual classifier for each set and several other hypothetical classifiers. Our meta-classifier outperforms a classifier which always makes the default decision even on the second set of problem instances. This set is an extreme case because just making the default choice is almost always the best choice – the misclassification penalty for the default choice classifier is extremely low given the large number of instances. Even though there is only very little room for improvement, we achieve some of it.

|  |  |  |  |  |
|:---|---:|---:|---:|---:|
|  | misclassification penalty \[s\] |  |  |  |
|  | instance set 1 |  | instance set 2 |  |
| classifier | all features | cheap features | all features | cheap features |
| oracle | 0 | 0 | 0 | 0 |
| anti-oracle | 19993 | 19993 | 47144 | 47144 |
| **default decision** | **2304** | **2304** | **223** | **223** |
| random decision | 5550 | 5550 | 564 | 564 |
| best classifier on set 1 | 0.998 | 0.994 | 131 | 220.3 |
| worst classifier on set 1 | 2304 | 2304 | 223 | 223 |
| best classifier on set 2 | 0.998 | 61.66 | 131 | 186 |
| worst classifier on set 2 | 1.34 | 1.44 | 621 | 610 |
| **meta-classifier** | **1.16** | **0.996** | **220** | **222.95** |

Summary of classifier performance on both sets of benchmarks in terms of total misclassification penalty in seconds. We first evaluated the performance using the full set of features described in Section , then using only the cheap features. The oracle classifier always makes the right decision, the anti-oracle always the wrong decision. The "default decision" classifier always makes the same decision and the "random decision" one chooses one of the possibilities at random. Three-fold cross-validation was used. All numbers are rounded.

It also shows that the classifiers we have learned on a data set that contains problem instances from many problem classes can be applied to a different data set with instances from different problem classes and still achieve a performance improvement. Based on this observation, we suggest that our meta-classifier is generally applicable.

Another observation we made is that the performance of the meta-classifier does not suffer even if a large number of the classifiers that it combines perform badly individually. This suggests that the classifiers complement each other – the set of instances that each one misclassifies are different for each classifier. Note also that the classifier which performs best on one set of instances is not necessarily the best performer on the other set of instances. The same observation can be made for the classifier with the worst performance on one of the instance sets. This means that we cannot simply choose "the best" classifier or discard "the worst" for a given set of training instances.

The time required to compute the features was 27 seconds per instance on average, and it took 0.2 seconds per instance on average to run the classifiers and combine their decisions. If we take this time into account, our system is slower than just using the default implementation. This is mostly because of the cost of computing all the features required to make the decision. We do however learn good classifiers in the sense that the decision they make is better than just using the standard implementation.

For the alldifferent constraint, there is little room for improvement to start with. We therefore need to focus on making a decision as quickly as possible. Most of the time required to make the decision is spent computing the features that the classifiers need. We removed the most expensive features – all the properties of the primal graph described in Section  apart from edge density.

The results for the reduced set of features are shown in Table  as well. The performance is not significantly worse and even better on the first set of instances, but the time required to compute all the features is only about 3 second per instance. On the first set of benchmarks, we solve each instance on average 8 seconds faster using our system (misclassification penalty of default decision minus that of our system divided by the number of instances in the set). We are therefore left with a performance improvement of an average of 5 seconds per instance. On the second set, we cannot reasonably expect a performance improvement – the perfect oracle classifier only achieves about 0.2 seconds per instance on average.

# Conclusions and future work

We have applied machine learning to a complex decision problem in constraint programming. To facilitate this, we evaluated the performance of constraint solvers representing all the decisions on two large sets of problem instances. We have demonstrated that training a set of classifiers without intrinsic knowledge about each individual one and combining their decisions can improve performance significantly over always making a default decision. In particular, our combined classifier is almost as good as the best classifier in the set and much better than the worst classifier.

We have conclusively shown that we can improve significantly on default decisions suggested in the state-of-the-art literature using a relatively simple and generic procedure. We provide strong evidence for the general applicability of a set of classifiers learned on a training set to sets of new, unknown instances. We identified several problems with using machine learning to make constraint programming decisions and successfully solved them.

Our system achieves performance improvements even taking the time it takes to compute the features and run the learned classifiers into account. For atypical sets of benchmarks, where always making the default decision is the right choice in almost all of the cases, we are not able to compensate for this overhead, but we are confident that we can achieve a real speedup on average.

We have identified two major directions for future research. First, it would be beneficial to analyse the individual machine learning algorithms and evaluate their suitability for our decision problem. This would enable us to make a more informed decision about which ones to use for our purposes and may suggest opportunities for improving them.

Second, selecting which features of problem instances to compute is a non-trivial choice because of the different cost and benefit associated with each one. The classifiers we learned on the reduced set of features did not seem to suffer significantly in terms of performance. Being able to assess the benefit of each individual feature towards a classifier and contrast that to the cost of computing it would enable us to make decisions of equal quality cheaper.

# Acknowledgements

The authors thank Peter Nightingale for helpful discussions about the alldifferent constraint and its implementations and descriptions of the problem features used in the analysis. Chris Jefferson provided further feature descriptions. We thank Jesse Hoey for useful discussions about machine learning and the anonymous reviewers for their feedback. In particular reviewer 2 provided helpful pointers to relevant machine learning research. Lars Kotthoff is supported by a SICSA studentship.