author: [^1]
Physik Institut, Universität Zürich, 8057 Zürich, Switzerland
E-mail:
title: Experience with CMS pixel software commissioning

# Introduction

The CMS experiment, at the Large Hadron Collider (LHC) includes a silicon pixel detector to allow tracking in the region closest to the interaction point . The detector, installed in July 2008, will be a key component for reconstructing interaction vertices and heavy quark decays in a particularly harsh environment, characterized by a high track multiplicity and heavy irradiation.

The detector consists of three barrel layers and two end-cap disks at each barrel end. The innermost barrel layer has a radius of 4.3 cm, while for the second and third layers the radius is 7.2 cm and 11 cm, respectively. The layers are composed of modular detector units. These modules consist of thin, segmented silicon sensors with highly integrated readout chips connected by the bump bonding technique. They are attached to cooling frames, the cooling tubes being an integral part of the mechanical structure. The barrel region is composed of 672 full modules and 96 half modules. The number or readout channels per module is 66'560 (full modules) or 33'280 (half modules).

The endcap disks extending from about 6 to 15 cm in radius are placed at $z=\pm34.5$ cm and $z=\pm46.5$ cm. Disks are divided into half-disks including 12 "U"-shaped cooling channels disposed in a turbine-like geometry to enhance charge sharing. The endcap part includes 672 detector modules of five different sizes.

The minimal pixel cell size is dictated by the readout circuit area required for each pixel. In finding and localizing secondary decay vertices both transverse ($r\phi$) and longitudinal ($z$) coordinates are important in the barrel region. Therefore a nearly square pixel shape is preferred. Because charge is often shared among several pixels, the use of analogue signal readout enables position interpolation improving the spatial resolution. In the barrel the charge sharing in the $r\phi$-direction is large due to the 4 T magnetic field of the CMS solenoid. With a sensitive detector thickness of 285 $\mu$m the pixel size is 100 $\mu$m and 150 $\mu$m along the $r\phi$ and $z$ coordinates, respectively .

One of the greatest challenges in the design of the pixel detector is the high radiation level on all components at very close distances to the colliding beams. At full LHC luminosity the innermost barrel layer will be exposed to a particle fluence of $3\times10^{14}$ n$_{\rm{eq}}/$cm$^2$/yr, the second and third layer to about $1.2\times10^{14}$ n$_{\rm{eq}}/$cm$^2$/yr and $0.6\times10^{14}$ n$_{\rm{eq}}/$cm$^2$/yr, respectively[^2]. All components of the pixel system are specified to stay operational up to a particle fluence of at least $6\times10^{14}$ n$_{\rm{eq}}/$cm$^2$.

The detector performances are expected to evolve with the exposure to irradiation and variation of the electric field across the sensor bulk. It is well known that sensor bias voltage will have to be increased with increasing irradiation to compensate the charge losses due to trapping. Thus, the event reconstruction software is designed to cope with a varying charge collection efficiency and precisely measure the hit position throughout the detector lifetime. The reconstruction techniques rely on periodical calibration procedures.

This paper describes the calibration and reconstruction software for the CMS Pixel detector and well as early results from data collected with cosmic ray trigger. The paper is structured as follows: The hit reconstruction techniques are discussed in Section . A description of the offline calibration procedures is given in Section . First results from data collected with cosmic ray trigger are presented in Section . The conclusions are given in Section .

# Hit reconstruction

The position of the pixel clusters is calculated in a two-step procedure using different reconstruction algorithms. During track seeding and pattern recongnition the hit position is calculated using the algorithm described in Section . This algorithm is fast but does not provide the ultimate precision, therefore, in the final track fit the hit position is recalculated using a more precise algorithm based on cluster shapes. The so-called *template* technique is described in Section . This section shows also a comparison of the position resolution of both techniques based on the CMSSW simulation software.

## First-pass hit reconstruction

Pixel clusters are formed by adjacent pixels with charge above the readout threshold. Both side and corner adjacent pixels are included in the cluster and a cut of 5'000 electron is applied on the total charge. The cluster is projected along the transverse ($x$) and longitudinal ($y$) direction by summing the charge collected in the pixels with the same coordinate. The Lorentz effect enhances the cluster lenght along the $x$ coordinate by $L=D\tan{(\Theta_L)}$ . For a cluster size of one the position is given by the center of the hit pixel corrected for the Lorentz shift. For larger clusters the hit position is determined by the expressions: $$\begin{aligned}
x  &=& x_C + \frac{Q_{\rm last}-Q_{\rm first}}{2(Q_{\rm last}+Q_{\rm first})}|W_x-W^x_{\rm inner}|-\frac{1}{2}L,\label{eq:pos_reconstruction1}\\
y  &=& y_C + \frac{Q_{\rm last}-Q_{\rm first}}{2(Q_{\rm last}+Q_{first})}|W_y-W^y_{\rm inner}|,
\label{eq:pos_reconstruction2}
\end{aligned}$$ where $Q_{\rm first}$ ($Q_{\rm last}$) is the charge in electrons collected in the first (last) pixel above threshold, $(x_C,y_C)$ is the geometrical center of the cluster and $W_{x,y}$ is the the total charge width given by: $$\begin{aligned}
W_x &=& D \tan(\alpha-\frac{\pi}{2}) + L, \\
W_y &=& D \tan(\beta-\frac{\pi}{2}).
\label{eq:charge_width}
\end{aligned}$$ $W_{\rm inner}^{x,y}$ represents to the inner lenght of the cluster and is equal to (size-2)$\cdot$pitch. Fig.  shows the definition of the track impact angle $\alpha$ with respect to the sensor plane along the $x$ direction. Similarly, $\beta$ corresponds to the impact angle along the $y$ coordinate.

## Clusters template hit reconstruction

The template-based reconstruction algorithm is a procedure that translates pre-stored cluster projection shapes, also called "templates", across measured cluster projections to find the best fit and hence an estimate of the hit position in both transverse and longitudinal coordinates . The Pixelav simulation  is used to generate the templates which are stored as functions of the impact angles along with large quantities of auxiliary information in a template object. The simulation was originally written to interpret beam test data from several unirradiated and irradiated sensors. It was extremely successful in this task, demonstrating that simple type inversion is unable to describe the measured charge collection profiles in irradiated sensors and yielding unambiguous observations of doubly-peaked electric fields in those same sensors .

During the first phase of template generation, the transverse ($x$) and longitudinal ($y$) projections for each simulated cluster are summed into respective 7-pixel and 21-pixel arrays. By construction, the hit pixel is the center pixel in each projection. The $x$ and $y$ coordinates of the hit are each binned in bins of width 0.125 pixel pitch. The bins are chosen so that the middle bin is centered on the pixel center and the end bins are centered on the pixel boundaries. This yields 9 bins spanning the central pixel where the end pixels differ by a full pixel pitch. The total charge, square of the charge, and number of entries are summed for each of the 9 (8 independent) bins. The template consists of the average signal $S^{y/x}_{i,j}$ in each projected pixel $i$ and bin $j$.

The second phase of the template generation uses the pre-stored results of the first phase to apply the actual template reconstruction algorithm to the same data samples used to generate the 9-bin templates. The second pass generates information on biases, errors, corrections, and goodness-of-fit that are combined with the results of the first pass to build a 448 kB template summary file that represents a given set of operating conditions as simulated by Pixelav.

The basic goal of the procedure is to translate the expected cluster shape until it best matches the observed cluster shape. This is accomplished by evaluating the following chisquare function for some or all of the template bins, $$\begin{aligned}
\chi^2(j) &=& \sum_i \frac{(P^{y/x}_i - N_jS^{y/x}_{i,j})^2}{(\Delta P^{y/x}_i)^2}  \label{eq:chione} \\
            N_j &=& \sum_i \frac{P^{y/x}_i}{(\Delta P^{y/x}_i)^2}\bigg{/} \sum_i \frac{S^{y/x}_{i,j}}{(\Delta P^{y/x}_i)^2} \nonumber
\end{aligned}$$ where $P^{y/x}_i$ are 1-d projections of the *measured* cluster and $\Delta P^{y/x}_i$ the respective rms. A more complete description of the template reconstruction technique can be found in .

The template algorithm was tested using samples of events with six 20 GeV muons generated by the CMSSW simulation software. A comparison of the template and first-pass algorithms is shown in Fig. . The rms resolutions in the two charge bands are shown in the standard algorithm (red lines), the template algorithm (blue lines), and the template algorithm after the low charge clusters have been removed with a simple charge cut (green lines). Note that the template algorithm still outperforms the standard one with all effects present and improves further when the low charge clusters are removed.

# Offline calibrations

A precise hit position determination relies on the correct calibration of the detector response. The CMS pixel hit reconstruction requires two sets of calibration constants: the front-end gain constants and the amplitude of the Lorentz angle. The former is needed to convert the pixel analog charge from ADC counts into electrons. The latter is used to calculate Eqs. -. The two calibration procedures are described in the following sections.

## Calibration of the front-end gain response

The measurement of the gain response is performed for each pixel readout chip (ROC) cell in two steps. Firstly, the gain response is measured by injecting a pulse of variable amplitude in each cell of the readout chip (ROC). The injection is performed several times for each pulse amplitude and the average charge in ADC counts is computed for each point. The gain response function of a single cell is shown in Fig.  as function of the injected pulse amplitude in DAC units. The front-end response is linear up to about 600 DAC units and then saturates. The saturation is beneficial for suppressing large Landau fluctuations and charge deposits due to secondary electrons.

The second step consists in performing a straight line fit to the linear part of the gain curve and converting DAC units into electrons. Offset and gain (i.e. the inverse of the fit slope) from the fit are stored in a ROOT file for each pixel cell, together with the fit $\chi^2$, its probability and the number of d.o.f. The fit procedure is successful for 99.8% of the pixels. About 0.2% of the pixels show a saturated gain response over the full amplitude range and the fit fails. The $\chi^2$ probability is larger than 0.1% for about 97% of the pixels. A visual check of the pixels with $P(\chi^2)<0.1\%$ shows deviations from linear trend at low input amplitudes. The average offset and gain from the rest of the pixels is assigned to the cells where the fit does not converge properly or a $P(\chi^2)<0.1\%$ is returned. The slope and offset distribution for all pixel cells are shown in Fig.  and , respectively. The relative spread (rms divided by mean value) is 37% for the offsets and 22% for the slopes.

The gain calibration constants are stored in the CMS offline database and used during event reconstruction. To contain the memory needs of the event reconstruction process the gains are averaged over ROC columns (80 channels) while pixel-granularity is used for the offsets. The final payload size is 67 MB. At the High Level Trigger (HLT) both offsets and gains are averaged over ROC columns, for a total payload size of about 2 MB. While the reduced payload is acceptable for triggering purposes the best position resolution is achieved during offline event reconstruction due to the high granularity.

The measured pixel charge in ADC counts is converted into electrons by applying the calibration constants discussed above and converting DAC units into electrons. This last step is performed by applying the DAC-to-electrons conversion constants measured during module production with a x-ray source .

## Lorentz angle calibration

Due to the dependence of the charge carrier mobility on temperature and detector exposure to radiation a periodic measurement in-situ of the Lorentz angle is highly desirable. During operations with colliding beams the angle can be measured with the *grazing angle* method as described in . The method relies on the measurement of the average cluster deflection with respect to the direction of the incoming track. It was first applied on data collected with a pion beam and prototype sensors at CERN  and later optimized for tracks from proton collisions using CMSSW detector simulations. As reported in  the Lorentz angle can be measured with a 2% accuracy with 1'000 tracks. The high track rate during collisions allows to measure the value in different regions of the detector, e.g. as function of the detector rapidity. A different value of the Lorentz angle can be expected in various parts of the detector due to their different exposure to irradiation.

Data collected with cosmic ray trigger lead to tracks crossing the detector with a wide range of impact angles and a different measurement technique can be applied. The technique relies on the measurement of the cluster size as function of the impact angle in the transverse plane. The cluster width in the transverse plane is minimal when the impact angle is parallel to the direction of the Lorentz deflection and is larger for all other angles. Thus, the Lorentz angle amplitude can be determined by measuring the minimum of the cluster size distribution.

# First results from cosmic ray data

The pixel detector was included in the CMS data taking period with cosmic ray trigger in autumn 2008. About 94% of the end-cap wheels and 99% of the barrel part were included in the run. The missing section of the end-cap wheel was mainly due to a short in a power distribution card developed after installation which will be repaired during the next winter shutdown.

Data were recorded both with and without 3.8 T magnetic field. Among 8.5 million tracks detected in the CMS silicon strip tracker about 77'000 tracks traversing the pixel detector volume were recorded. Noisy pixel cells where detected with the online data quality monitoring software  by applying a cut on the pixel frequency. In total, only 263 and 17 pixel cells were masked during data taking in the barrel and end-cap regions, respectively. Tracks in the pixel and strip tracker were promptly reconstructed with three different cosmic track finding algorithms currently available in the CMSSW reconstruction software . The pixel hit reconstruction algorithms are the ones described in Section .

The distribution of barrel hits in the transverse plane is shown in Fig. . The hit position was found to be correct in all detector parts except the barrel half-modules of the negative $z$ region for which the local coordinates were inverted. A bug fix was deployed for the reprocessing of the data sample. The calibrated charge distribution corrected for the track impact angle is shown in Fig. . The distribution peak is in line with the charge deposit expected for a minimum ionizing particle traversing 285 $\mu$m of silicon. The measured mean charge and rms are $19.4\times10^3$ and $10.8\times10^3$ electrons, respectively. No pixel inter-calibration was applied during prompt event reconstruction and a unique set of offset and gain constants were used for all pixels. The result of the gain calibration scans are currently under study and will be applied in future data reprocessing.

The time alignment between the pixel detector and the trigger signal from the muon system was optimized during the run. The track detection efficiency in the pixel detector was measured with different time delays with respect to the first level trigger signal. The efficiency was defined as the ratio $N_{\rm detected}/N_{\rm expected}$ where $N_{\rm expected}$ is the number of tracks reconstructed in the strip tracker with transverse impact parameter and longitudinal parameter below 10 cm and 25 cm, respectively, and at least one hit in the bottom part of the micro-strip Tracker Inner Barrel. The numerator $N_{\rm detected}$ was defined by the number of tracks including at least one hit in pixel barrel region. At the optimal delay the measured efficiency is about 70%. The fraction of undetected cosmic tracks can be explained by the longer shaping time set in the micro-strip front-end chips and by the jitter of the trigger signal.

# Conclusions

This paper has described the main aspects of the calibration and reconstruction software of the CMS pixel detector. Two reconstruction techniques were implemented for the measurement of the cluster position. A fast algorithm is used during track seeding and pattern recognition while a more precise and CPU intensive technique is applied during the final track fit to determine the track parameters. The latter, based on pre-computed average cluster shapes improves the hit position and can cope with charge trapping effects due to irradiation. The main offline calibration procedures consist in the measurement of the gain response and of the Lorentz angle amplitude.

The reconstruction and calibration workflow was successfully tested during the CMS commissioning phase with cosmic muon trigger. About 77'000 tracks and 200'000 hits were detected in the pixel detector. The very first results presented in this paper show excellent performances of the detector in terms of signal and noise. The analysis of the full data-set is expected to be completed during the winter shutdown.

[^1]: On behalf of the CMS Pixel Group.

[^2]: All fluences are normalized to the non-ionizing energy loss (NIEL) of 1 MeV neutrons (n$_{\rm{eq}}/$cm$^2$).


abstract: Instance retrieval requires one to search for images that contain a particular object within a large corpus. Recent studies show that using image features generated by pooling convolutional layer feature maps (CFMs) of a pretrained convolutional neural network (CNN) leads to promising performance for this task. However, due to the global pooling strategy adopted in those works, the generated image feature is less robust to image clutter and tends to be contaminated by the irrelevant image patterns. In this article, we alleviate this drawback by proposing a novel reranking algorithm using CFMs to refine the retrieval result obtained by existing methods. Our key idea, called query adaptive matching (QAM), is to first represent the CFMs of each image by a set of base regions which can be freely combined into larger regions-of-interest. Then the similarity between the query and a candidate image is measured by the best similarity score that can be attained by comparing the query feature and the feature pooled from a combined region. We show that the above procedure can be cast as an optimization problem and it can be solved efficiently with an off-the-shelf solver. Besides this general framework, we also propose two practical ways to create the base regions. One is based on the property of the CFM and the other one is based on a multi-scale spatial pyramid scheme. Through extensive experiments, we show that our reranking approaches bring substantial performance improvement and by applying them we can outperform the state of the art on several instance retrieval benchmarks.
author: Jiewei Cao$^1$[^1], Lingqiao Liu$^2$, Peng Wang$^1$, Zi Huang$^1$, Chunhua Shen$^2$, and Heng Tao Shen$^1$
$^1$The University of Queensland, Australia $^2$The University of Adelaide, Australia
{j.cao3, email@example.com, {lingqiao.liu, email@example.com, {huang, email@example.com
bibliography: qam.bib
title: Where to Focus: Query Adaptive Matching for Instance Retrieval Using Convolutional Feature Maps

# Introduction

Instance retrieval is the task of finding a particular instance from a large image corpus. In practice, instance retrieval has two major challenges: 1) the large visual appearance deformations due to the object's different positions, poses and illumination when being captured; and 2) various distractors from the natural and less contextual backgrounds. In the past decade, a lot of work uses local invariant descriptors to handle instance retrieval, including methods that use large visual codebooks and methods that use compact representations . Although promising performance has been reported on some constrained datasets for these methods, in a more realistic and challenging scenario where the target objects are with cluttered backgrounds, instance-level retrieval still remains a challenging problem .

Recent studies show the visual descriptors produced by aggregating the convolutional feature maps (CFMs) from convolutional neural networks (CNNs) achieve state-of-the-art performance for image retrieval . Comparing to the conventional local descriptors, these deep learned features capture more semantic information since a CNN is usually trained on large labeled image datasets . Most existing methods aggregate the CFMs into a compact global representation for an image, where the query-to-image similarity is evaluated on the image level. How to use these deep convolutional features to distinguish the target object from distractors in cluttered backgrounds is still challenging.

In this paper, we follow the direction of CNN-based methods and propose a novel reranking algorithm, namely query adaptive matching (QAM), using CFMs to address the image clutter problem during instance retrieval. The key idea of our method is that instead of relying on a global image representation, we represent each dataset image by a set of base regions. In further, we allow these base regions to freely merge into larger regions. Image features are then extracted from the combined region rather than the whole image. Moreover, we parameterize the regions that can be created via combining the base region and represent the combined feature as a function of a set of parameters. Thus, adjusting these parameters will be equivalent to choosing the region-of-interest to focus for a candidate image. Our method will cast the parameter selection issue as an optimization problem under the objective of maximizing the similarity between the query and the feature obtained from the combined region. The optimal parameters are expected to generate a focus region that mostly covers the object-of-interest.

To implement this idea, we also propose two ways of generating base regions. One is based on the property of the CFM and the other one is based on the multi-scale spatial pyramid. By conducting experiments on the various instance retrieval benchmarks, we show that the proposed QAM reranking, together with the two base region generation approaches can achieve promising results which outperform the state of the art. Besides that, we also discuss several practical issues of using CFMs for instance retrieval, including the choice of deep CNN models and the convolutional layers.

# Related Work

## Non-CNN Based Retrieval

In the past decade, instance retrieval is mainly handled by the methods using local invariant descriptors, such as SIFT . Previous work can be roughly divided into two categories: 1) methods that encode local descriptors into large visual codebooks and sparse representations, namely the Bag-of-Words (BoW) ; and 2) methods that aggregate local descriptors into dense and compact features . Due to the loss of spatial information and the degradation of discriminative power of the descriptor after visual word quantization, BoW models are usually followed by some post-processing steps, e.g., spatial verification or query expansion , in order to eliminate false positive results. In practice, BoW models adopt an inverted index for efficient search, but the number of images that can be indexed is limited by the search time and index size . A different strategy is to aggregate the local descriptors into compact representations, e.g., compressed Fisher Vector , VLAD and T-embedding with democratic aggregation . However, recent studies show that the neuron activations extracted from CNN serve as good image representations, which surpass conventional features in low dimensionality settings.

## CNN Based Retrieval

CNNs are widely used in computer vision since the success of "AlexNet" in large-scale image classification . Recent studies show that the neuron activations of CNNs can be used as generic features for image retrieval, where the features are from the *fully-connected layers*. However, these layers are trained on labeled objects to facilitate image classification and hence might not generalize to some instance types. These methods usually require fine-tuning the CNN on the target (or visually similar) datasets to obtain satisfactory retrieval performance.

Besides of the fully-connected layers, there is an emerging trend toward using the activations of the convolutional layers, named as *convolutional feature maps* (CFMs), as image features which shows superior performance. Specifically, Razavian et al. propose to segment the image into multiple square patches and extract patch descriptors using CNN. During searching, they cross-match all the patches to obtain the best match results. Obviously, this method cannot handle large-scale datasets due to the high computational cost. Babenko et al. propose a simple but effective CFMs aggregation method based on sum-pooling, which generates compact global representations (256 dimensions) for retrieval. But their performance still lags behind the traditional methods. Tolias et al. propose an aggregation method which first decomposes the CFMs into multiple regions at different scales and then aggregates them via sum-pooling. This method also generates compact representations and outperforms in most cases. Besides that, the authors also propose a new reranking method based on integral max-pooling of CFMs which can roughly locate the target objects. Their retrieval framework achieves comparable or even better performance compared to traditional methods. We use a retrieval pipeline similar to that in , but we propose a different reranking algorithm which can detect the discriminative regions of the target object more flexibly and precisely, and therefore achieves better retrieval performance.

# Instance Retrieval using CFMs

As mentioned in previous sections, recent studies have shown that using CFMs as image features achieves promising performance in instance retrieval and our work will follow this direction. Before elaborating our methods, this section will give a brief discussion on CFMs, their properties and their use for instance retrieval. These discussions provide the motivation and inspiration of our methods.

## Preliminary

The CFMs generated by a convolutional layer can be arranged in a tensor of the size $H \times W \times D$ (see Fig. ), where $H$ and $W$ denote the height and width of each feature map, and $D$ denotes the number of feature maps (or channels) in that layer. Due to the local connectivity of each filter, the activations at the same spatial location across all feature maps can be composed into a $D$-dimensional local descriptor for a certain image region, where the region size equals to the size of the filter's receptive field. This local feature is formally denoted as $\mathbf{x}_i \in \mathbb{R}^D$ in this paper and the subscript $i$ indicates the $i$-th location in the CFMs. In total, there are $H \times W$ such locations in the CFMs. The above arrangement is demonstrated in Fig. . The major advantage of CFMs over the fully-connected layers' activations is that it retains the spatial information of local image patterns and it is essentially more similar to the traditional hand-engineered local features. In fact, the CFMs from one convolutional layer can be viewed as an array of local features sampled from a dense sampling grid.

Throughout this paper, we use the following notations. Let $\mathbf{X} \in \mathbb{R}^{H \times W \times D}$ be the CFMs extracted from the $l$-th layer (after applying the ReLU). It can be equivalently represented as a set of local descriptors $\mathbfcal{X} = \{ \mathbf{x}_i \in \mathbb{R}^D | i \in \{ 1,\dots,H \times W \} \}$. In the following sections, we use three example images selected from different object types in INSTRE dataset for illustration purpose (see Fig. ).

## Understanding CFMs

Due to the parameter sharing design of the convolutional layer , a feature map of a convolutional layer can be viewed as the detection scores obtained by applying a detector, the filter of a convolutional layer namely, on $H \times W$ spatial locations and the activation value at the $i$-th location of a CFM characterizes the detector response at the same location. With this analogy, we can obtain an intuitive understanding of a CFM by visualizing it, that is, the highly activated locations suggest that around them it contains the visual patterns depicted in the convolutional layer filter. Fig. gives such an illustration. In Fig. , the feature maps with the top five average activation scores are selected for visualization. Also, comparison of the activation patterns from different convolutional layers is provided in Fig. . From it we can make the following two observations:

-   A CFM only has high activations on few locations with the presence of certain visual patterns. This suggests that the convolutional filter can be highly selective to certain visual patterns and as a result the CFM can become sparse.

-   The early convolutional layer of a CNN tends to capture some primary visual patterns, e.g. edges along certain directions or dots while the late convolutional layer is usually selective to visual patterns that corresponding to a shape or an object part, e.g. the circular shape or a wheel region.

## CFMs Aggregation for Instance Retrieval

To perform instance retrieval, an image needs to be represented by a feature vector or a set of feature vectors. To this end, the CFMs can be aggregated through some pooling strategies which have been explored in several recent works . Most of these works adopt a global pooling strategy, that is, local features from all locations in the CFMs will be aggregated together. In the simplest case, we can adopt the following pooling operations: $$\begin{aligned}
        \mathbf{f} = \frac{\sum_i \mathbf{x}_i}{\|\sum_i \mathbf{x}_i\|_2}.

\end{aligned}$$ In other words, the aggregated image representation $\mathbf{f}$ is obtained by first sum-pooling all the local features and then perform the $\ell_2$-normalization. In practise, the $\ell_2$-normalization has been found to be important and without it the retrieval performance will decrease significantly.

Despite being simple and effective, the global pooling strategy has the drawback of being less robust to the cluttered image background. This is because for instance retrieval the query can be only a cropped object. Thus in a candidate image the object-of-interest may only take up a part of the image whereas global pooling aggregates visual patterns from the whole image area. To alleviate this, one solution is to perform pooling on multiple regions of the image (or equivalently its corresponding CFMs) and create multiple pooled features to represent the image. Then a query will match against all those pooled features and the best matching score will be utilized as the similarity between the query and the image. However, this solution raises the issue of how to choose those regions. On the one hand the number of regions cannot be too large due to the storage bottleneck of a retrieval system. On the other hand the space of all possible sub-regions can be extremely large, especially considering that the object can take any shape and as a result the object shape can be much more complex than a bounding box. Thus, it is very challenging to design a region generation strategy for the multi-region pooling idea.

# Query Adaptive CFMs Matching Reranking

This paper adopts the multi-region pooling strategy discussed in Sec. and proposes an elegant solution to explore a large number of possible regions while keeping the storage for the pooled regions low. Our method first generates a small number of base regions which can be (softly) merged into larger regions. Thus the total number of regions that can be represented by the combination of base regions is very large. For a query image, we compare it against all possible region combinations and pick the best matching score as the query-to-image similarity. At the first glance, the above procedure can be prohibitively expensive. However, if we use a set of parameters to control the merging operation, the pooled feature of the combining region can be formulated as a function of those parameters. The above matching process can be cast as an optimization problem, that is, optimal parameters are sought for maximizing the similarity measure between the pooled feature and the query image feature. The optimal objective value represents the best possible matching score between a combined region and the query image. Certainly, this idea can be slower than direct image feature comparison and thus in this paper we use it as a reranking method and apply it only to a shortlist of images retrieved by an existing approach. In the following subsections, we will elaborate the formulation and implementation details of this idea.

## Query Adaptive Matching

Formally, our method assumes a set of base image regions have been extracted for each image by a base region generation method, e.g. the ones will be introduced in Sec. and Sec. . Let's denote the sum-pooling feature of the CFMs in each of the base region as $\mathbf{f}_k$ and the normalized sum-pooling feature obtained by merging a set of base regions can be represented as[^2]:

$$\begin{aligned}
\label{eq:feature_combine}
    \bar{\mathbf{f}} = \frac{\sum_{k} z_k \mathbf{f}_k}{\|\sum_{k} z_k \mathbf{f}_k\|_2},
\end{aligned}$$ where $z_k$ is a binary indicator and if its value equals to 1 its corresponding region will be added into the merged region. In practice, we can also relax its value to a positive real value which implies a soft merge operation. The similarity between the query image $\mathbf{q}$ and merged feature can be then represented by their inner product $\langle \mathbf{q}, \bar{\mathbf{f}} \rangle$. As seen, this similarity is a function of $\{z_k\}$ and our aim is to find the best similarity score that can be achieved by softly merging the base regions. Then this best match score will be used to rerank the shortlist. The search for the optimal $\{z_k\}$ can be written as the following optimization problem: $$\begin{aligned}
        & \max
        && \frac{  \langle \mathbf{q}, \sum^K\limits_{k=1} z_k \mathbf{f}_k\rangle }{ || \sum^K\limits_{i=1} z_k \mathbf{f}_k ||_2 }.\\
        & \text{s.t.}
        && z_k \geq 0, \forall k=1,...,K.
        \end{aligned}
        \label{eq:cvx:max_simple}$$ For notation simplification, Eq. can be expressed in a more compact way: $$\begin{aligned}
        & \max\limits_{\mathbf{z}}
        && \frac{ \mathbf{q}^T \mathbf{F} \mathbf{z} }{ \|\mathbf{F} \mathbf{z}\|_2 },\\
        & \text{s.t.}
        && \mathbf{z} \geq 0.
        \end{aligned}
     \label{eq:cvx:max_matrix}$$ where $\mathbf{z} = [z_1,  \cdots z_k, \cdots, z_K]^T$ and $\mathbf{F} = [\mathbf{f}_1, \cdots,\mathbf{f}_k, \cdots \mathbf{f}_K]$. For $\mathbf{q}^T \mathbf{F} \mathbf{z} \neq 0$, solving Eq. is equivalent to solving: $$\begin{aligned}
        & \min\limits_{\mathbf{z}}
        && \frac{ \|\mathbf{F} \mathbf{z}\|_2 }{ \mathbf{q}^T \mathbf{F} \mathbf{z} },\\
        & \text{s.t.}
        && \mathbf{z} \geq 0.
        \end{aligned}
        \label{eq:cvx:min_max_transform}$$ We further constrain $\mathbf{q}^T \mathbf{F} \mathbf{z} =1$ to avoid the arbitrary scaling of $\mathbf{z}$ and thus ensure that the optimization problem has the unique solution. The final optimization problem is as follows: $$\begin{aligned}
        & \min\limits_{\mathbf{z}}
        && \|\mathbf{F} \mathbf{z}\|^2_2,\\
        & \text{s.t.}
        && \mathbf{q}^T \mathbf{F} \mathbf{z} = 1,\\
        &&& \mathbf{z} \geq 0.
        \end{aligned}
        \label{eq:cvx:min_final}$$ Eq. is a quadratic programming (QP) problem which can be solved efficiently by existing optimization packages. We use the inverse of this optimal value, which equals to the objective value in Eq. , as the image similarity score.

## Base Region Generation

The above method applies to base regions generated in any possible ways. In this work, we particularly focus on two approaches to create such base regions.

### Feature Map Pooling (FMP)

Our first way of generating base regions is inspired by the observation made in Sec. . As discussed in Sec. , the activated region of a CFM at a late convolutional layer usually represents an object part. The union of them can then cover the whole image or at least those regions with meaningful visual patterns. Thus, the activated region of each CFM in a convolutional layer can be directly utilized as a base region. Specifically, given a set of CFMs $\mathbfcal{X} \in \mathbb{R}^{H \times W \times D}$, we define the set of non-zero elements' locations in the $d$-th feature map as a base region, that is, $\mathcal{R}_{d}=\{ i | x^{d}_i > 0 \}$. The pooled feature for this base region is then calculated as: $$\mathbf{f}_d = \sum\limits_{ i \in \mathcal{R}_{d} } \mathbf{x}_i.$$ In our implementation, we also have $\mathbf{f}_d$ $\ell_2$-normalized to compensate the region size discrepancy. We call this base region generation method as Feature Map Pooling, or FMP in short. With FMP, the $\mathbfcal{X}$ will be transformed into $D$ region descriptors which may be corresponding to different object parts, as shown in Fig. .

Once a convolutional layer of a pretrained CNN is chosen, the number of base regions will equal the number of feature maps or channels in that convolutional layer. However, for a given image, the activated regions for many CFMs can be highly overlapped. Thus it would be a waste of computation if we use all the base regions generated from the CFMs. To reduce the computational cost, in this paper we merge base regions as follows. We first represent each base region by their corresponding pooled feature $\mathbf{f}_d$. If two CFMs have similar activated regions, their corresponding pooled feature will be similar. Thus we run a clustering method, $k$-means clustering or spectral clustering , to group $D$ pooled features into $K$ clusters. The base regions whose pooled features are grouped together will then be merged into one base region. In our experiment, this merging operation reduces the number of base regions from 512 to around 30 and thus significantly reduces the computational cost.

### Overlapped Spatial Pyramid Pooling (OSPP)

Another way to generate base regions is to directly sample some rectangle regions from the image. Tolias et al. propose an region sampling method to sample regions at different scales and different locations. In our work, we adopt similar strategy to generate the base regions. More specifically, as shown in Fig. , we sample square regions at $L$ different scales. Given an image's CFMs $\mathbf{X} \in \mathbb{R}^{H \times W \times D}$, we uniformly sample regions of width $2 min(W,H)/(l+1)$ at the $l$-th level. Also, the regions are sampled to allows around 40% overlap between consecutive regions.

Note that even using a similar sampling strategy, our approach is different from the work in. The latter one pools the features generated from each sampled region together to obtain a single global-level image representation while our method uses sampled regions as base regions which are fed into the proposed query adaptive matching algorithm.

## Retrieval Pipeline

Finally, to achieve the best performance with our method, we adopt the following pipeline to perform retrieval.

-   **Initial Retrieval**. In offline pre-processing, the CFMs are extracted for all database images via pretrained CNNs. The global image representations are computed by an existing aggregation method (e.g., SPoC , R-MAC or CroW ). During online searching, we extract the same feature for query image and evaluate the similarities between the query and all the database images by calculating the feature similarity, e.g., inner product, to obtain an initial ranking list.

-   **Reranking**. A short list consisting of the top N candidates from the initial ranking list is selected for reranking. Base region descriptors are extracted offline and now they are used to represent each candidate images. We solve Eq. and obtain the similarity score between the query and a candidate image for reranking.

-   **Query Expansion** (QE). Finally, we select the top five images after reranking as in and aggregate their global representations with the query feature by average pooling. We use the aggregated feature as a new query and perform the retrieval again to obtain the final ranking list.

# Experiments

## Experiment Settings

### Datasets

We evaluate our methods on the following datasets:

-   **Oxford5k** contains 5,062 images corresponding to 11 Oxford landmarks. There are five possible queries for each landmark, and 55 queries in total. This dataset can be combined with the 100,071 distractors from *Flickr100k* for large-scale retrieval evaluation. We denote the combined dataset as *Oxford105k*.

-   **Paris6k** is similar to Oxford5k but consists of 6,412 images corresponding to 12 Paris landmarks. It can also be mixed with the Flickr100k, *Paris106k* namely, for large-scale retrieval evaluation.

-   **Sculptures6k** has 6,340 smooth sculpture images, which are different from the previous two landmark datasets. The whole corpus is divided equally into training and testing datasets. Since our focus is generic instance retrieval, we evaluate our methods without using any training images for CNN learning or fine-tuning. There are 70 queries in total.

-   **INSTRE** includes 23,070 images from 200 objects and one million distractor images. These objects can be categorized into three classes: architectures (buildings and sculptures), planar objects (designs, paintings and planar surface) and daily stereoscopic objects (toys and irregularly-shaped products). This dataset is challenging since there are diverse variations for each object and most of the objects are with cluttered and less contextual backgrounds. All the annotated images (23,070 in total) from the 200 objects are used as queries for performance evaluation purpose.

The query bounding boxes are provided on all the datasets and are used for cropping out the target objects during retrieval. The evaluation metric is mean average precision (mAP) for all the datasets.

### CFMs Extraction

We use VGG19 provided by Caffe and extract the CFMs from the last convolutional layer (*conv5_4*) because previous studies show that this layer achieves the best performance. Moreover, the original image sizes are kept for both database images and queries during feature extraction. However, for the large images (whose areas are greater than $1000 \times 1000$) and the small images (whose shorter sides are smaller than 256), we resize them accordingly but keep their original aspect ratios. All the input images are zero-centered by *RGB mean pixel subtraction* .

### Global Feature Generation

We evaluate two state-of-the-art CFMs aggregation methods: the sum-pooled convolutional features (SPoC) without center prior and the regional maximum activation of convolutions (R-MAC) . We use the optimal settings reported in and . Both SPoC and R-MAC require PCA-whitening and the parameters of PCA are learned on hold-out images for all datasets . Specifically, when performing retrieval on Oxford5k we learn the parameters on the images from Paris6, and vice versa. For Sculptures6k and INSTRE, we learn the parameters on Flickr100k. In the following experiments, we use the *inner product* as similarity measurement during retrieval for both SPoC and R-MAC. All the experiments are conducted on a Linux server with 40 3.0GHz CPU cores and 256GB memories.

## Performance of QAM Reranking

We first report the initial retrieval results of the aforementioned global representations in Tab. , which serve as the baselines of the proposed method. Note that the mAPs of SPoC reported here are higher than those reported in (e.g., the mAPs are increased from 0.531 to 0.666 on Oxford5k, and 0.501 to 0.606 on Oxford105k). This is due to the implementation differences: 1) in our implementation the image is not resized to square shape as in when being fed into a CNN for feature extraction; 2) the original dimensionality of the representation is kept during the PCA-whitening step. From the table we can see R-MAC outperforms SPoC on all the datasets except on Sculptures6k. Therefore, we use R-MAC for initial retrieval when evaluating the performance of our proposed QAM reranking.

```latex
\begin{savenotes}
\begin{table}[!t]
\centering
\caption{The mAPs of VGG19 with SPoC and R-MAC. ``DIM.'' denotes the feature dimensionality.}
\label{tab:exp:map_vgg19}
\scalebox{0.9}{
\begin{tabular}{l|c|cccccc}
    \hline
                                             & DIM.  & Ox5k  & Ox105k & Pa6k  & Pa106k & Sc6k  & INSTRE \\ \hline
    SPoC
                                            & 512  & 0.685  &0.622  &0.798  &0.704  &0.517  &0.306 \\
    R-MAC
                                            & 512  & 0.695 & 0.644 & 0.838  &0.763  &0.490  &0.415  \\
    \hline
\end{tabular}
}
\end{table}
\end{savenotes}
```

### QAM with FMP

QAM is based on the query feature $\mathbf{q}$ and the region descriptors $\mathbf{F}$ of a candidate image. To obtain $\mathbf{q}$, the query's CFMs are sum-pooled and $\ell_2$-normalized. For database images, we follow the procedure described in Sec. to obtain their region descriptors. The results are shown in Tab. . Here we choose R-MAC for initial retrieval, the results of which serve as the baseline of our reranking method. From the results we see improvement of the reranking over the initial retrieval on all the datasets.

To demonstrate the effectiveness of QAM with FMP more intuitively, we visualize some merged feature maps after QAM in Fig. . We use the three example instances in Fig.  as queries and visualize the top 10 ranked candidates after reranking. Recall that the objective of QAM is to find the optimal merging of base regions of candidate images which maximizes the similarity to the query feature. After solving the matching problem in Eq. , we obtain the optimal aggregation weighting $\mathbf{z}$. We multiply the $i$-th feature map by its corresponding weighting score $z_i$, and then sum up all the feature maps to form a merged map. The locations with large values (the warm areas) are selected by QAM as the discriminative parts w.r.t the query, and the locations with zero values (the blue areas) tend to correspond to the irrelevant background which are ignored by the QAM. As seen, QAM detects the discriminative components of the target object and simultaneously suppresses the distractors. Another advantage of QAM with FMP is that it can select object parts in irregular shapes which can largely benefit feature aggregation.

As mentioned in Sec. , FMP generates 512 base regions for each database image among which some are overlapped. To reduce computational cost, our treatment is to perform spectral clustering to pre-merge the regions offline and the number of clusters correspond to the number of regions after this merging. To obtain a trade-off between reranking performance and retrieval efficiency we test the performance of FMP under different number of clusters $K$ in Tab . As can be seen, initially increasing the number of clusters leads to better performance. One interesting discovery is that when $K=25$, the performance is comparable to or even better than that obtained without clustering. And after $K=25$, the results remain stable when continuously increasing $K$. Based on this observation, we fix $K=25$ in the following experiments. With this pre-merging step, we downsize the FMP features from $\mathbb{R}^{512 \times 512}$ to $\mathbb{R}^{25 \times 512}$.

```latex
\scalebox{0.9}{
\begin{tabular}{l|l|l|llll}
\hline
       & BEF.  & AFT.  & $K=10$ & $K=20$ & $K=25$ & $K=30$                    \\
\hline
Ox5k   & 0.695 & 0.709 & 0.689 & 0.706 & \textbf{0.710} & 0.709               \\
Pa6k   & 0.838 & 0.844 & 0.844 & 0.844 & \textbf{0.845} & \textbf{0.845}      \\
Sc6k   & 0.490 & \textbf{0.515} & 0.512 & 0.513 & 0.514 & \textbf{0.515}      \\

\hline
Ox105k & 0.644 & 0.658 & - & - & \textbf{0.662} & - \\
Pa106k & 0.763 & \textbf{0.774} & - & - & \textbf{0.774} & - \\
INSTRE & 0.415 & 0.436 & - & - & \textbf{0.446} & - \\

\hline
\end{tabular}
}
```

### QAM with OSPP

In this subsection, we evaluate the QAM reranking with the other base region generation strategy overlapped spatial pyramid pooling (OSPP). When reranking with OSPP, the query's CFMs are aggregated via R-MAC and for candidate database images, we follow the procedure described in Sec. to generate their region descriptors. The number of scales is set to 3 ($L=3$) for both query and candidates which is the same as in . The reranking results are show in Tab. . As can be seen, the reranking with OSPP consistently outperforms the initial retrieval on all the datasets, which demonstrates the effectiveness of the proposed reranking method.

Moreover, we draw a comparison between the two proposed reranking methods in Tab. . Based on different region generation mechanisms, these two reranking methods have different advantages. For FMP, it can generate base regions that correspond to object parts of irregular shapes which are more flexible than regular shapes such as rectangles. For OSPP, it considers the multi-scale information which makes the matching more robust. Thus, we see these two reranking strategies have different performances on the datasets. But generally, their performance are comparable and both outperform the R-MAC based initial retrieval.

### Computational Cost of QAM Reranking

Since efficiency is an important aspect of instance retrieval, in this subsection, we evaluate both the storage cost and computational cost of the proposed reranking strategies. Specifically, we report the average cost of performing one similarity matching. The results are shown in Tab. . For FMP based reranking, we use clustering to merge the region number of a database image from 512 to 25 offline which significantly reduces the memory footprints and the computational cost of QAM. For OSPP, the number of regions of a database image is 23 on average. The results demonstrate that both QAM strategies can be performed efficiently, about 15 ms, under tractable memory footprints. Since the reranking is conducted on the top 100 shortlist of initial retrieval, it costs about 1.6s for each query.

|      | Ox5k      | Pa6k      | Sc6k      | INSTRE    |
|:-----|:----------|:----------|:----------|:----------|
| BEF. | 0.695     | 0.838     | 0.490     | 0.415     |
| OSPP | **0.736** | 0.842     | 0.505     | 0.436     |
| FMP  | 0.710     | **0.845** | **0.514** | **0.446** |

The mAPs of QAM reranking with OSPP. "BEF." means the R-MAC results (in Tab. ) before QAM reranking. The cluster number is $K=25$ for FMP.

|      |      DIM.       | MEM. | TIME  |
|:-----|:---------------:|:----:|:-----:|
| FMP  | $25 \times 512$ | 51KB | 0.016 |
| OSPP | $23 \times 512$ | 47KB | 0.014 |

The computational cost per image on average for FMP and OSPP, including the feature dimensionality (DIM.), the memory footprints (MEM.) and the similarity computation time (TIME, in seconds).

```latex
\begin{savenotes}
\begin{table*}[!t]
\centering
\caption{
    Comparison with Existing Methods.
    We report results for different retrieval methods with/without (bottom/top) post-processing.
    The ``Remark'' column shows the feature type used by each method.
    ``DIM.'' is the feature dimensionality.
    Methods with ``*'' are reported using our own implementations.
    }
\label{tab:exp:vs_soa}
\begin{tabular}{l|c|c|cccccc}
    \hline
                                    & Remark        & DIM. & Ox5k  & Ox105k & Pa6k  & Pa106k & Sc6k  & INSTRE \\ \hline
    J{\'{e}}gou and Zisserman \cite{Jegou:Tembed/2014}
                                    & SIFT        & 512  & 0.528 & 0.461  & -     & -      & -     & -      \\
    Babenko et al. \cite{Babenko:NeuralCodes/2014}
                                    & CNN        & 512  & 0.557 & 0.522  & -     & -      & -     & -      \\
                                            % & 512  & 0.435 & 0.392  & -     & -      & -     & -      \\
    Razavian et al. \cite{Razavian:CnnINSBaseline/2015}
                                    & CNN        & 256  & 0.533 & 0.489  & 0.670 & -      & 0.377 & -      \\
    SPoC* \cite{Babenko:spoc/2015}  & CNN        & 512  & 0.685 & 0.622  & 0.798 & 0.704  & 0.517 & 0.306  \\
    R-MAC* \cite{Tolias:rmac/2015}  & CNN        & 512  & 0.695 & 0.644  & 0.838 & 0.763  & 0.490 & 0.415  \\
    %SPoC \cite{Babenko:spoc/2015}           & 512  & 0.589 & 0.578  & 0.798* & 0.704*  & 0.517* & 0.306*  \\
    %R-MAC \cite{Tolias:rmac/2015}           & 512  & 0.669 & 0.616  & 0.830 & 0.757  & 0.490* & 0.415*  \\

    \hline
    J{\'{e}}gou et al. \cite{Jegou:HE/2010} & BoW & -    & 0.747 & -      & -     & -      & -     & 0.268\tablefootnote{This result is reported in \cite{Wang:instre/2015}}  \\
    Chum et al. \cite{Chum:qerevisit/2011}  & BoW & -    & 0.827 & 0.767  & 0.805 & 0.710  & -     & -      \\
    Mikul{\'{\i}}k et al. \cite{Mikulik:FineQuantization/2013}
                                            & BoW & -    & \textbf{0.849} & \textbf{0.795} & 0.824  & 0.773 & - & -  \\
    Arandjelovic and Zisserman \cite{Arandjelovic:sculptures6k/2011}
                                            & BoB & -    & -     & -      & -     & -      & 0.502 & -      \\
    Tolias et al. \cite{Tolias:rmac/2015}   & CNN & -    & 0.773 & 0.732  & 0.865 & 0.798  & -     & -      \\
    \textbf{Ours}                           & CNN & -    & 0.781 & 0.747  & \textbf{0.874} & \textbf{0.828}  & \textbf{0.608} & \textbf{0.592}  \\
    \hline
\end{tabular}
\end{table*}
\end{savenotes}
```

## Comparison With Existing Work

Our complete retrieval pipeline consists of three steps: initial retrieval with R-MAC, QAM reranking with FMP, and QE in the end (see Sec. ). Here we compare our complete retrieval method to the state of the art in Tab. .

The results are divided into two parts, with the above part being the retrieval methods using different compact representations but involving no post-processing, e.g., reranking, spatial verification, or QE, and the other part corresponding to methods that use post-processing strategies.

From the first part we observe that when aggregated to comparable dimensionalities, CNN based representations can have better retrieval performance than SIFT based compact representations and among the CNN representations, R-MAC performs best since it considers both the multi-scale information and weak spatial information.

The second part of Tab. reports methods with post-processing. Jégou et al. improves the traditional BoW model with hamming embedding and weak geometric consistency constraints, but the result of this method on INSTRE reported in is far from satisfactory. Our method achieves the best performance on all datasets except for Oxford5k and Oxford105k, where the well developed BoW models are still better. Higher mAPs (more than 0.9) on Oxford5k and Paris6k are reported in and . But since their methods learn the visual codebooks on the original datasets and hence are highly tailored, the results are not directly comparable. Compared to in which specialized representations, namely Bag-of-Boundaries (BoB), for smooth object are proposed, our method outperforms it by 10% and has better generalization ability as well. Our main focus is on comparing to the latest work in since their representation is based on CFMs and we have similar retrieval pipelines. The most notable difference between their method and ours lies in the reranking strategy. For fair comparison, we use similar initial retrieval as . From the results we can see our method consistently has superior performance. Another point worth mentioning is that, as far as we know, our method achieves the best performance on the INSTRE dataset among the reported results which shows the robustness of our reranking method on dataset with cluttered backgrounds.

Finally we show the search time of different components of the retrieval pipelines for each query and the memory consumption of the whole retrieval framework on different dataset sizes in Tab. .

```latex
\scalebox{0.85}{
\begin{tabular}{c|llcl|r|r|r}
\hline
\multicolumn{1}{l|}{\multirow{2}{*}{}} & \multicolumn{4}{c|}{TIME (seconds)}                                                                                & \multicolumn{3}{c}{MEM.}                                                          \\ \cline{2-8}
\multicolumn{1}{l|}{}                  & \multicolumn{1}{c|}{INIT.} & \multicolumn{1}{c|}{QAM}   & \multicolumn{1}{c|}{QE}                                                                        & \multicolumn{1}{c|}{tot.} & \multicolumn{1}{c|}{R-MAC} & \multicolumn{1}{c|}{FMP} & \multicolumn{1}{c}{tot.} \\ \hline
5K                                     & \multicolumn{1}{l|}{0.001}  & \multicolumn{1}{l|}{1.161} & \multicolumn{1}{c|}{\multirow{3}{*}{-}} & 1.163                     & 10MB                        & 256MB                     & 266MB                    \\
100K                                   & \multicolumn{1}{l|}{0.017}  & \multicolumn{1}{l|}{1.160} & \multicolumn{1}{c|}{}                                                                          & 1.194                     & 200MB                       & 5.1GB                     & 5.3GB                    \\
1M                                     & \multicolumn{1}{l|}{0.127}  & \multicolumn{1}{l|}{1.374} & \multicolumn{1}{c|}{}                                                                          & 1.628                     & 2GB                         & 51.2GB                      & 53.2GB                     \\ \hline
\end{tabular}
}
```

## Performance with Deeper Networks

Recent works discover that deeper neural networks can lead to significant improvement on tasks like image classification and object detection. Here we explore two of these deeper networks on instance retrieval. One is GoogLeNet which is a 22-layer network incorporating the "Inception" modules and the other one is a recently proposed 152-layer convolutional neural network , named deep residual net (ResNet152).

### Feature Patterns

To intuitively understand the CFMs of these two networks, we first visualize the feature patterns of the CFMs of these two networks. Specifically, for each layer's CFMs, we calculate the $\ell_1$-norm for all the local descriptors ($||\mathbf{x}_i||_1$) and normalize them by the maximum norm value in this layer. The heat maps of these norm values indicate the activation degree of different image parts. From the better explored AlexNet or VGG net we know that generally the CFMs of lower layers correspond to some primary patterns like edges, and CFMs of higher layers capture some object-related semantics. And the heat maps tend to be spatially sparse. Here we observe some difference for the CFMs of GoogLeNet and ResNet152, the visualizations of which on the first example image from Fig. are shown in Fig. .

For GoogLeNet, the structures of the objects persist in CFMs from Inception 4a to 4e layer where the activated patterns are sparse and correspond to some meaningful object parts but the structure information tends to be lost in higher layers (5a and 5b). For ResNet152, the sparse property does not hold in lower layers (conv1 to res4) but persist in higher layers (res5a-res5c). Since the activated pattern plays an important role in CFMs aggregation, we attempt to unveil some relationship between the retrieval performance of CFMs and their corresponding feature patterns.

### Choosing the Right Layer

Here we evaluate the retrieval performance of CFMs from different layers using different global aggregation methods (Sec. ). For GoogLeNet, we select the last six Inception modules (Inception 4b to Inception 5b) for testing. Fig. shows the retrieval performance of different layers. As we can see, initially the mAPs on all datasets are improved from lower to higher layers (from 4b to 4e), but the last two Inception layers (5a and 5b) have inferior performance. The reason of the performance degradation may lie in our aformentioned analysis that the spatial structure and sparsity are weakened in the two higher layers. On Paris6k and Sculptures6k, GoogLeNet is outperformed by VGG19 and on Oxford5k GoogleNet performs better.

For ResNet152, we select the last five residual modules (res4b34 to res5c) for testing as shown in Fig. . From this figure we observe that generally higher layers perform better and the best performed setting of ResNet152, i.e., res5b with R-MAC, can signifiantly outperform VGG19. However, the improvements are bought at the price of more memory and computational cost: the dimensionality of ResNet152's best performed layer, res5b, is 2,048, which is four times higher than that in VGG19.

### Final Integration

Finally, we test our methods with CFMs from the res5b layer of ResNet152. The whole retrieval pipeline is still the same: initial retrieval with R-MAC, followed by QAM re-ranking with OSPP, and finally QE. Note that OSPP is applied here instead of FMP because it takes too much time to downsize the FMP features from $\mathbb{R}^{2048 \times 2048}$ to $\mathbb{R}^{25 \times 2048}$ via clustering for each image. The results are shown in Tab. . We can see the ResNet152 can significantly outperform VGG19 on all the datasets. The results demonstrate the effectiveness of deeper network in instance retrieval. However, these improvements are brought at the price of more memory and computational cost.

```latex
\begin{savenotes}
\begin{table}[!t]
\centering
%\small
\caption{
    The mAPs of our retrieval framework with different CFMs of different networks.
}
\label{tab:exp:map_res152}
\scalebox{0.8}{
\begin{tabular}{l|cccccc}
    \hline
                                    & Ox5k  & Ox105k & Pa6k  & Pa106k & Sc6k  & INSTRE \\ \hline
    VGG19                           & 0.781 & 0.747  & 0.874 & 0.828  & 0.608 & 0.592  \\
    ResNet152                       & \textbf{0.828} & \textbf{0.814}  & \textbf{0.893} & \textbf{0.858}  & \textbf{0.712} & \textbf{0.670}  \\
    \hline
\end{tabular}
}
\end{table}
\end{savenotes}
```

# Conclusion

In this work, we propose a reranking algorithm, namely query adaptive matching, for instance retrieval using convolutional feature maps. The key idea is to represent a candidate database image by a set of base regions and generate a target object focused representation resorting to the similarity to the query image. We formulate the similarity matching as well as the region merging process as an optimization problem which can be solved efficiently. Apart from this general framework, we propose two practical ways to generate the base regions. The experiments on several instance retrieval datasets demonstrate the effectiveness of the proposed reranking method.

[^1]: This work was done when Jiewei was visiting The University of Adelaide.

[^2]: In practice, $\mathbf{f}_k$ can be generated via other pooling methods, e.g. max-pooling, and we still use Eq. to combine features from multiple to-be-merged regions. This treatment makes our method more general applicable.


abstract: Lithium Niobate (LN or just "niobate") thin-film micro-photonic resonators have promising prospects in many applications including high efficiency electro-optic modulators, optomechanics and nonlinear optics. This paper presents free-standing thin-film lithium niobate photonic resonators on a silicon platform using MEMS fabrication technology. We fabricated a 35um radius niobate disk resonator that exhibits high intrinsic optical quality factor (Q) of 484,000. Exploiting the optomechanical interaction from the released free-standing structure and high optical Q, we were able to demonstrate acousto-optic modulation from these devices by exciting a 56MHz radial breathing mechanical mode (mechanical Q of 2700) using a probe.
address: School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, 14853
author: Renyuan Wang$^*$ and Sunil A. Bhave
title: Free-standing high quality factor thin-film lithium niobate micro-photonic disk resonators

# Introduction

The lack of inversion symmetry in the Lithium Niobate (LN or niobate) crystal exhibits itself via its characteristic strong piezoelectric effect, linear electro-optic effect, pyroelectric effect and strong second order optical nonlinearity . Leveraging the strong electro-optic effect and optical nonlinearity, the optics industry uses bulk LiNbO$_3$ for manufacturing photonic modulators and optical frequency doublers . In recent years, LiNbO$_3$ thin-film photonics have attracted a lot of attention for advanced photonic applications, as the micro-fabricated photonic structures can enable enhanced light/matter interaction from the highly confined optical mode and long interaction time. These can lead to reduced threshold excitation power, enhanced nonlinear optical effect, and agile electrically tunable photonic devices.

Microring resonators and microdisk resonators are fundamental building blocks for many micro-photonic devices including filters , lasers , and modulators . A high $Q$ thin-film LiNbO$_3$ resonator has the potential to enable many novel and high performance devices. While an-isotropic etching defined thin-film LiNbO$_3$ photonic structures are desired for applications such as photonic crystals and resonators, the $Q$ demonstrated to date from such devices is generally limited to few thousand, while the quality factor from bulk LiNbO$_3$ disk resonators can exceed 10$^8$ . The key limiting factor of the quality factor is the optical scattering loss from the rough side wall profile from the etching process. Surface smoothing by heating the device close to the melting point of LiNbO$_3$ has been attempted with limited success (Q$\sim$`<!-- -->`{=html}3 $\times$ 10$^4$). In , the optical $Q$ is improved to 7.2 $\times$ 10$^4$ by etching a deposited cladding layer of slab waveguides with the drawback of reduced optical confinement. Recently, our group has demonstrated a LiNbO$_3$ thin-film fabrication technology for manufacturing free-standing LiNbO$_3$ RF MEMS devices , where we were able to achieve smooth and nearly vertical etching profile of LiNbO$_3$ using a ion mill etching process on a full 4 inch 1um thick LiNbO$_3$ thin-film on LiNbO$_3$ substrate wafer. In contrast to focused ion beam etching, our process is batch fabrication compatible. Leveraging these MEMS fabrication techniques, we demonstrate here free-standing thin-film LiNbO$_3$ microdisk resonator on silicon substrate with more than 50x better optical $Q$ . Moreover, the free-standing structure allows the coupling between the mechanical domain and optical domain, with which we also demonstrate acousto-optic modulation from the resonator.

# Fabrication

There are three distinct fabrication challenges to fabricate the LN micro-photonic resonator on silicon substrate: 1. Achieving LiNbO$_3$ thin-film with bulk single crystal LiNbO$_3$ quality, 2. Achieving vertical side-wall and smooth surface profiles without mask residue, 3. Achieving clearance between the LN device and silicon substrate to enable efficient mode confinement.

The common technique for fabricating LiNbO$_3$ thin-film is by ion-slicing . The drawback is that this process requires an annealing step to recover the crystal quality from the damage caused by the ion implantation. The annealing process unavoidably cause thermal stress in the film, which limits the yield for achieving LiNbO$_3$ thin-film on full wafer scale. In contrast, we use a wafer bonding and grinding process as shown in Fig. . We start with a Z-cut white LiNbO$_3$ wafer. The bonding surface is activated by plasma (similar as in ). Then, the device wafer is flip-bonded to the Si handle wafer at room temperature. Therefore, the thermal stress is minimized, which allows us to achieve LiNbO$_3$ thin-film on full 4in wafer. The device wafer is then ground down to 1um thickness.

To date, etching of LiNbO$_3$ have used either metal or silicon dioxide as a hard-mask, which increases the complexity and cause compatibility issues of the fabrication. In addition, any residue of metal left behind can interact with circulating photons causing optical absorption and scattering, which will significantly reduce the optical Q of the device. Here, we using 4um thick photo-resist as the mask with ion mill etching. The resist can then be easily rinsed off by acetone with sonication. If necessary, an additional O$_2$ plasma clean ensures that no residue is left behind. The Argon ion mill etching can be controlled to be self-balancing between etching and re-sputtering by carefully adjusting the ion beam incident angle, where the re-sputtering can act as an extra sidewall "protection" during the etching. With this process, we achieved sidewall angle of 87 degrees with $<$`<!-- -->`{=html}10nm surface roughness (inset of Fig. ), thus enabling low scattering loww and meeting the requirement of a smooth vertical sidewall that is necessary for future integrated waveguide coupling.

Silicon has higher refractive index ($\sim$`<!-- -->`{=html}3.4) than LN ($\sim$`<!-- -->`{=html}2.3). So any light that couples into the niobate disk will leak into the silicon substrate. Using XeF$_2$ dry-etch to undercut the niobate disk resonator (Fig. ) enables us to achieve $>$`<!-- -->`{=html}40um clearance between the niobate disk and silicon substrate, thereby enabling outstanding optical mode confinement in niobate and thus high optical Q. In addition, the free-standing disk structure will enable optomechanical interactions between the mechanical and photonic modes.

## Optical characterization

We use a tapered optical fiber to couple light from tunable near-IR laser (Santec TSL-510) to the niobate disk resonator (Fig. ). A polarization controller is introduced to carefully adjust the light polarization in fiber such that it couples strongly with the photonic resonator. The transmitted optical signal is sent to a high-speed photodiode (Newport 1544A) and monitored on an Agilent (DSO9404A) oscilloscope. By sweeping the wavelength of the input light, we measure the optical transmission spectrum and extract the optical parameters (quality factor, group index, optical propagation loss) by fitting the transmission dip to a Lorentzian . Fig. shows the broad-band transmission spectrum of a 35um radius LN disk resonator from 1500nm to 1510nm. During the scan, the optical input power is kept below 20uW to ensure the resonator operates within the linear regime verified by the symmetric peaks. One resonance at 1502.61nm is fitted to the Lorentzen, and the inset shows the zoom-in view (red curve). The extracted intrinsic optical $Q$ is 484k, with a 95% confidence bounds from 458K to 509K. The loaded $Q$ of the resonance is 39k, and the extrinsic $Q$ is 42k. The intrinsic $Q$ is $\sim$`<!-- -->`{=html}40 times better comparing to previous state of art.

# Acousto-optic modulation through optomechanical interaction

As the free-standing resonator body supports both mechanical and optical resonance, the optomechanical interaction can cause acousto-optic modulation of the transmitted light. The mechanical motion of the disk is excited by the gradient force of electric field applied by a probe (Fig. ). This causes effective optical path length change causing the shift of the optical resonant frequency. When the wavelength of the input optical wave is biased at close to the 3dB transmission of the resonance, the frequency shift causes modulation of the transmitted optical power. Fig. shows the schematic of the experimental setup. The probe is excited by supplying an AC voltage from port 1 of an Agilent (N5230A) network analyzer. The RF output of the photodetector is connected to port 2 of the network analyzer, while the DC output is monitored on the oscilloscope to track the transmitted DC optical power. The output power from port 1 of the VNA is 8dBm, and Fig. shows the scattering parameter measurement result from 300kHz to 100MHz. The red curve shows the s-parameters noise floor without coupling the taper to the resonator, while the blue curve shows the s-parameters when the taper is coupled to the resonator and the input wavelength is biased at a high $Q$ resonance around 1540nm. The wavelength is slowly tuned into the resonance from the short wavelength, and biased at the side of the resonance when the peaks in the $S_{21}$ curve is maximized. The transmitted DC optical power is 70uW. The $S_{21}$ shows two peaks around 50MHz with very good signal to noise ratio. The peak at 56MHz in the $S_{21}$ measurement corresponds to the radial breathing mechanical mode as shown in the figure, while the 67MHz peak corresponds a shear mechanical wave traveling around the rim of the disk. The mechanical $Q$ of the radial mode is 2700 by measuring the 3dB bandwidth of the peak.

# Conclusion

The MEMS-based fabrication technology presented in this work opens up new avenues to realize optical resonators, modulators, frequency doublers and frequency combs that leverage the multi-domain RF, photonic, optomechanical coupling in a monolithic LN-on-Silicon platform. We demonstrated free-standing thin-film LiNbO$_3$ microdisk resonator with a intrinsic optical $Q$ of 4.84 $\times$ 10$^5$. In addition, we demonstrated acousto-optic modulation leveraging optomechanical interaction of the free-standing structure.


author: C.A. Watson[^1]; D. Steeghs; V.S. Dhillon; T. Shahbaz
title: Imaging the cool stars in the interacting binaries AE Aqr, BV Cen and V426 Oph

# Introduction

Starspots, the most easily observed manifestation of magnetic activity, are a ubiquitous feature of rotating late-type stars with outer convective zones. Indeed, naked-eye observations of spots on our own Sun first took place 1000's of years ago, and subsequent telescopic observations have been undertaken almost continuously for over 3 centuries. Despite this, our understanding of how magnetic fields are produced and evolve on the Sun and other stars, and how fundamental parameters affect magnetic field generation, is still critically lacking. Since Vogt & Penrod (1983) demonstrated that starspots could be seen as bumps traversing the line profiles of HR 1099, several indirect imaging techniques have been used to map the magnetic topographies on mainly (but not exclusively) isolated rapid rotators. Since it is well known that magnetic activity increases with increasing stellar rotation rate, much of this work has focused on determining how rotation impacts starspot emergence.

Interacting binaries such as Cataclysmic Variables (CVs) contain some of the most rapidly rotating late-type stars known. CVs typically consist of a late-type main-sequence star in orbit around a more massive white dwarf. The close proximity of both components causes the late-type (or donor) star to fill its Roche-lobe and transfer mass to a white dwarf. With orbital periods as short as an hour, and tides forcing synchronous rotation of the donor, images of these systems provide excellent tests of stellar dynamo theories at ultra-fast rotation rates. In addition, such systems also allow the impact of tides on the stellar dynamo to be assessed.

Furthermore, many interacting binaries show variations in orbital periods, brightnesses, and outburst durations that have been attributed to starspots and activity cycles. Despite their obvious importance, both to understanding stellar dynamos and understanding the observed behaviour of interacting binaries, little progress in this field has been made until recently. In this review, we will describe the astro-tomographic techniques required to image starspots on the cool stars in interacting binaries, before summarising some of the recent highlights in this field. Finally, we will conclude with some future opportunities.

# The technique of Roche Tomography

Unfortunately, interacting binaries are too small and too distant to be resolved directly, even in the worlds largest telescopes. Rutten & Dhillon (1994), however, described a technique, called Roche tomography, which uses phase-resolved spectra to reconstruct the line-intensity distribution on the donor stars. While analogous to Doppler imaging methods applied to single stars, Roche tomography differs in two important ways. First, the donor stars are tidally distorted into a Roche-lobe and are in synchronous rotation around the centre-of-mass of the binary. Second, the systems often show rapid variability due to the accretion. This means that one usually requires simultaneous photometry with which to slit-loss correct the spectra before continuum subtraction, whereas with Doppler imaging the spectra need not be slit-loss corrected and the continuum is normalised.

Otherwise, the concept of Roche tomography is relatively straight forward. Given the binary parameters, the donor's surface can be defined to lie on the critical surface of equal gravitational potential given by the Roche approximation at the point where Roche-lobe overflow occurs. The surface can then be modelled as a series of tiles lying on this critical surface. Each tile is then assigned a copy of the local specific intensity profile (which may either be Gaussian or taken from a spectral-type standard) and convolved with the instrumental resolution. One can then assign an intensity to each tile which scales its contribution to the total profile. Each tile's contribution is then integrated over the visible surface of the star taking into account limb-darkening, fore-shortening and the radial-velocity of the elements.

Roche tomography essentially carries out the reverse of the process just described. The contribution of each element is iteratively varied until a map is obtained which predicts data consistent with the observed data. The goodness of fit is measured using the reduced $\chi^2$ statistic. Unfortunately, the reduced $\chi^2$ constraint alone is not sufficient to select a unique map as there are many maps that can fit the data equally well, and so we also adopt an additional *regularisation* statistic. Following Horne (1985), we adopt the map of *maximum entropy*, which can loosely be thought of as the smoothest map, or map containing the least information, with respect to a *default map* which may contain some a priori information about the surface intensity distribution across the donor star. Generally, we assume a uniform default map whose pixels are set to the average value in the reconstructed map, and the default map is updated after each iteration. An efficient algorithm for maximising entropy subject to a $\chi^2$ constraint has been implemented by Skilling & Bryan (1984) in the fortran package memsys.

# Roche Tomography highlights

The earliest work using Roche tomography concentrated on reconstructing surface maps of CV donors using either single absorption or emission lines obtained using intermediate resolution spectrographs. The maps clearly showed the varying impact of irradiation from the compact object and accretion regions on these stars. While interesting in their own right, these results have been covered in several papers and reviews already and, instead of repeating the findings here, we refer the reader to the work by Watson et al. (2003). Instead, we shall focus on some of the more recent results showing the starspot distribution on a number of CV donors.

## AE Aqr – the first image of starspots on a CV donor star

Early, single line CV studies failed to reveal the presence of cool starspots on CV donors since they neither had the resolution nor the signal-to-noise to detect the small, faint signatures of starspots. (One should consider that most CVs have $m_v >$`<!-- -->`{=html}11, generally about 2–4 mags fainter than most stars that have been Doppler imaged). The first of these problems can be solved by employing a high resolution echelle spectrograph. Simply moving to a larger aperture telescope, however, does not solve the signal-to-noise problem.

Fortunately, the wide wavelength coverage of an echelle spectrograph allows 1000's of stellar lines to be observed in one spectrum. One can then apply a technique known as *Least Squares Deconvolution* (LSD – see Donati et al. 1997) which effectively combines all the lines to form one 'average' stellar absorption line of greatly increased signal-to-noise. Typically, this technique provides a multi-plex gain in signal-to-noise of $\sim$`<!-- -->`{=html}30. To put this in perspective, using LSD on an 8-m class telescope is the equivalent of attempting a single line study on a 40-m class telescope!

Armed with this technique, we obtained two nights echelle spectroscopy with the 4.2-m William Herschel Telescope (WHT) on La Palma on AE Aqr as part of a pilot study. This target was chosen as it is the brightest CV in the Northern sky, and the donor star contributes a large fraction ($\sim70\%$) to the total system light. AE Aqr also has a relatively long-orbital period of 9.87-hrs which meant that reasonably long exposures could be used before the donor's orbital motion started to smear out the line profiles. Application of LSD to this dataset resulted in a multi-plex gain in signal-to-noise of $\sim$`<!-- -->`{=html}26 over single-line studies.

The Roche tomogram of AE Aqr is published in Watson et al. (2006), and can be viewed as a movie on AN's webpages. Representative frames are displayed in Figure  and show several dark starspot features. The most prominent of these is a large, high latitude spot most clearly visible at orbital phase $\phi$=0.25. This spot is centred on a latitude of $\sim$`<!-- -->`{=html}65$^{\circ}$, in stark contrast to our Sun where few spots are seen above latitudes greater than $\sim$`<!-- -->`{=html}35$^\circ$. Such a high latitude feature on AE Aqr is, however, consistent with Doppler images of rapid rotators which often show the presence of polar spots.

Another feature of note is the dark region surrounding the $L_1$ point, which we have interpreted as irradiation of the inner face by the white dwarf and/or accretion regions. Such irradiation zones in similar locations in CV donors have been mapped previously (e.g. Davey & Smith 1996; Watson et al. 2003). There does, however, appear to be a chain of spots extending from the polar regions down to the irradiated $L_1$ point, making longitudes of the star facing the white dwarf companion appreciably more spotted than elsewhere. Indeed, in the Doppler image of the pre-CV V471 Tau, Hussain et al. (2006) also found that the side of the star facing the white dwarf was heavily spotted. This may well be due to tidal forces, which have been predicted to cause spots to emerge at preferred longitudes (e.g. Holzwarth & Schüssler 2003).

In total, Watson et al. (2006) estimated that 18 per cent of AE Aqr's Northern hemisphere was covered with spots, providing the first conclusive evidence that these systems were magnetically active. This put the canonical theory of CV evolution (which required strong magnetic fields on the donors in order to drain angular momentum from the binary and drive the systems to shorter orbital periods) on a more secure observational footing.

## BV Cen – starspots and slingshot prominences

Based on the success of the AE Aqr pilot study we began a campaign on the 6.5-m Magellan Clay telescope situated in Chile to map other CV donors. The larger aperture, wider spectral coverage and higher throughput of the Magellan setup compared to the WHT allowed systems 2 or 3 magnitudes fainter then AE Aqr to be imaged. The second CV we observed was the $\sim$`<!-- -->`{=html}14.7 hour period system BV Cen. A movie of BV Cen, reconstructed after applying LSD, is available on the AN website and some snapshots are shown in Figure .

The maps of BV Cen and AE Aqr (Figure ) make for interesting comparison. Again, BV Cen shows a high latitude spot at $\sim$`<!-- -->`{=html}65$^{\circ}$. Interestingly, the high latitude spot in both systems are displaced in the same direction towards the trailing hemisphere of the star (the side facing away from the orbital motion of the star). While we believe that the prominent dark region near the $L_1$ point is due to irradiation, we also see a chain of spots extending from the polar regions down to the $L_1$ point similar to that seen on AE Aqr. This provides some evidence that a mechanism is at work, perhaps tidal and/or Coriolis forces, that is forcing flux tubes to arise at these locations.

The fact that regions near the $L_1$ point seem heavily spotted, combined with the fact that low latitude spots are seen near the $L_1$ point, has interesting implications for the accretion dynamics of these systems. Many accreting binaries show sudden dips in their lightcurves, and it has been suggested that this is caused by starspots moving across the $L_1$ point and quenching the mass transfer, leading to a dimming in the system's brightness. In their modelling of the mass transfer history of AM Her, Hessman, Gänsicke & Mattei (2000) concluded that if starspots were to cause such dips then regions near the $L_1$ point would most likely be unusually heavily spotted, with a spot-filling factor around 50%. Certainly, both AE Aqr and BV Cen appear to be more densely spotted on the hemisphere facing the white-dwarf. Indeed, we have calculated that, for BV Cen, the spot-filling factor reaches 40% at longitudes near the $L_1$ point, which seems to support the conclusions of Hessman et al. (2000).

In addition to the presence of starspots in BV Cen, which we calculated covered some 25% of the Northern hemisphere, we also detected a transient narrow emission feature at zero velocity. Such emission has been seen previously in other CVs (see e.g. Steeghs et al. 1996), also situated at zero velocity, and have been interpreted as 'slingshot prominences' from the donor star sitting near the centre-of-mass of the binary. The preference for prominences to be seen at this location may be a combination of two things. First, prominences erupting from near the $L_1$ point will be more subject to illumination from the accretion regions, making them more visible to us. Second, it appears from the Roche tomograms of BV Cen and AE Aqr that spots are more likely near the $L_1$ point, which may increase the probability of a prominence erupting in these active regions. Furthermore, with such a strong concentration of activity near the $L_1$ point, magnetic fragmentation of the accretions stream may be responsible for the 'blobby' accretion seen in some CVs (e.g. Meintjes 2004; Meintjes & Jurua 2006).

## V426 Oph – caught during a prolonged low-state

Finally, we present a preliminary Roche tomogram of V426 Oph, again as a movie with some representative frames shown in Figure . With an orbital period of $\sim$`<!-- -->`{=html}6.85 hrs this is, to our knowledge, the fastest rotating star on which starspots have been imaged to date, the previous record being 7.44 hrs (RXJ1508.6 – Donati et al. 2000).

Unlike the Roche tomograms of AE Aqr and BV Cen, V426 Oph does not show such a prominent high latitude spot, though there is still a hint of a weak polar feature, again displaced slightly towards the trailing hemisphere. V426 Oph also shows spots appearing at lower latitudes. With a rotation rate almost 90 times solar, this is at odds with the models of Schüssler et al. (1996) which predict that magnetic flux tubes should only break the surface at high latitudes due to Coriolis forces and magnetic buoyancy.

Although the reason for the lack of a high latitude spot is unknown, one possibility is that we caught it at an unusual stage in its activity cycle. Interestingly, data from the AAVSO indicates that V426 Oph was undergoing some unusual behaviour at this time – being both dimmer during 2005 (when this data was taken), and exhibiting far fewer outbursts compared to other years. Was V426 Oph in an unusual state of activity, and is this responsible for its unusual behaviour during 2005? Further observations are required if any such link is to be made, or if we are to determine whether V426 Oph simply lacks a polar spot at all times.

Other than that, the inner face of V426 Oph appears to be quite uniformly irradiated (giving the star a darker appearance in the Roche tomograms at $\Phi$=0.5). Within the irradiated zone, however, individual starspots are still visible near the $L_1$ point, and V426 Oph also appears to exhibit a less distinctive chain of spots extending from high latitudes down to the $L_1$ point. Given the preliminary nature of the analysis presented here, we are hesitant to draw any more conclusions at this time.

# Future opportunities

While we are now beginning to reveal in some detail the magnetic nature of CVs, a sustained campaign is required to investigate whether these systems show solar type activity cycles, and to what extent such a cycle impacts on the observed behaviour of the binary. We are currently undertaking a monitoring campaign to determine whether AE Aqr displays any solar-like activity cycle.

Another interesting question is whether the donor stars in interacting binaries display an appreciable level of differential rotation. Scharlemann (1982) suggests that tidal forces should weaken, but not suppress, differential rotation. The fact that we see very similar spot distributions on AE Aqr and BV Cen suggests that the spot distribution may essentially be fixed. In turn, this suggests that differential rotation on these systems may be weak, perhaps even suppressed. Given that differential rotation is a key ingredient in stellar dynamo models, amplifying and transforming initial poloidal field into toroidal field, any suppression of differential rotation may provide a challenge to dynamo theory. Thus measurements of the differential rotation rate, via cross-correlation of latitude strips from images taken 1 or 2 weeks apart, is essential. Such studies on binaries have a unique advantage over those conducted on single stars in that the *sense* of the differential rotation can be deduced since the actual mean rotation period is known from the orbital period of the binary, allowing the corotation latitude of the donor star to be determined.

Finally, it would be interesting to prove that starspots can affect the accretion dynamics of interacting binaries by suppressing mass transfer. An observation of a starspot crossing the $L_1$ point, combined with a dip in the luminosity of the system, would be compelling evidence for such a scenario. It is quite comforting to see that, in the case of V426 Oph, we can still identify starspots even if they lie within irradiated regions. In summary, such work promises to reveal much about both the behaviour of binaries, and of the stellar dynamo under the most extreme conditions.

[^1]: Corresponding author: