abstract:
  .
  3D models provide a common ground for different representations of human bodies. In turn, robust 2D estimation has proven to be a powerful tool to obtain 3D fits "in-the-wild". However, depending on the level of detail, it can be hard to impossible to acquire labeled data for training 2D estimators on large scale. We propose a hybrid approach to this problem: with an extended version of the recently introduced SMPLify method, we obtain high quality 3D body model fits for multiple human pose datasets. Human annotators solely sort good and bad fits. This procedure leads to an initial dataset, *UP-3D*, with rich annotations. With a comprehensive set of experiments, we show how this data can be used to train discriminative models that produce results with an unprecedented level of detail: our models predict 31 segments and 91 landmark locations on the body. Using the 91 landmark pose estimator, we present state-of-the art results for 3D human pose and shape estimation using an order of magnitude less training data and without assumptions about gender or pose in the fitting procedure. We show that *UP-3D* can be enhanced with these improved fits to grow in quantity and quality, which makes the system deployable on large scale. The data, code and models are available for research purposes.
author: Christoph Lassner^1,2^
`email@example.com`; Javier Romero^3,\*^
`email@example.com`; Martin Kiefel^2^
`firstname.lastname@example.com`; Federica Bogo^4,\*^
`email@example.com`; Michael J. Black^2^
`email@example.com`; Peter V. Gehler^5,\*^
`firstname.lastname@example.com`
bibliography: bibliography.bib
title:  Unite the People: Closing the Loop Between 3D and 2D Human Representations

# Introduction

Teaching computers to recognize and understand humans in images and videos is a fundamental task of computer vision. Different applications require different trade-offs between fidelity of the representation and inference complexity. This led to a wide range of parameterizations for human bodies and corresponding prediction methods ranging from bounding boxes to detailed 3D models.

Learning-based algorithms, especially convolutional neural networks (CNNs), are the leading methods to cope with the complexity of human appearance. Their representational power has led to increasingly robust algorithms for bounding box detection , keypoint detection  and body part segmentation . However, they are usually applied in isolation on separate datasets and independent from the goal of precise 3D body estimation. In this paper we aim to overcome this separation and "unite the people" of different datasets and for multiple tasks. With this strategy, we attack the main problem of learning-based approaches for complex body representations: the lack of data. While it is feasible to annotate a small number of keypoints in images (, 14 in the case of the MPII-HumanPose dataset ), scaling to larger numbers quickly becomes impractical and prone to annotation inconsistency. The same is true for semantic segmentation annotations: most datasets provide labels for only a few body parts.

In this paper, we aim to develop a self-improving, scalable method that obtains high-quality 3D body model fits for 2D images (see Fig.  for an illustration). To form an initial dataset of 3D body fits, we use an improved version of the recently developed SMPLify method  that elevates 2D keypoints to a full body model of pose and shape. A more robust initialization and an additional fitting objective allow us to apply it on the ground truth keypoints of the standard human pose datasets; human annotators solely sort good and bad fits.

This semi-automatic scheme has several advantages. The required annotation time is greatly reduced (Sec. ). By projecting surfaces (Sec. ) or keypoints (Sec. ) from the fits to the original images, we obtain consistent labels while retaining generalization performance. The rich representation and the flexible fitting process make it easy to integrate datasets with different label sets, , a different set of keypoint locations.

Predictions from our 91 keypoint model improve the 3D model fitting method that generated the annotations for training the keypoint model in the first place. We report state-of-the art results on the HumanEva and Human3.6M datasets (Sec. ). Further, using the 3D body fits, we develop a random forest method for 3D pose estimation that runs orders of magnitudes faster than SMPLify (Sec. ).

The improved predictions from the 91 landmark model increase the ratio of high quality 3D fits on the LSP dataset by 9.3% when compared to the fits using 14 keypoint *ground truth* locations (Sec. ). This ability for self-improvement together with the possibility to easily integrate new data into the pool make the presented system deployable on large scale. Data, code and models are available for research purposes on the project homepage at <http://up.is.tuebingen.mpg.de/>.

# Related Work

Acquiring human pose annotations in 3D is a long-standing problem with several attempts from the computer vision as well as the 3D human pose community.

The classical 2D representation of humans are 2D keypoints . While 2D keypoint prediction has seen considerable progress in the last years and could be considered close to being solved , 3D pose estimation from single images remains a challenge .

Bourdev and Malik  enhanced the H3D dataset from 20 keypoint annotations for 1,240 people in 2D with relative 3D information as well as 11 annotated body part segments.

In contrast, the HumanEva  and Human3.6M  datasets provide very accurate 3D labels: they are both recorded in motion capture environments. Both datasets have high fidelity but contain only a very limited level of diversity in background and person appearance. We evaluate the 3D human pose estimation performance on both. Recent approaches target 3D pose ground truth from natural scenes, but either rely on vision systems prone to failure  or inertial suits that modify the appearance of the body and are prone to motion drift .

Body representations beyond 3D skeletons have a long history in the computer vision community . More recently, these representations have taken new popularity in approaches that fit detailed surfaces of a body model to images . These representations are more tightly connected to the physical reality of the human body and the image formation process.

One of the classic problems related to representations of the extent of the body is body part segmentation. Fine-grained part segmentation has been added to the public parts of the VOC dataset  by Chen et al. . Annotations for 24 human body parts and also part segments for all VOC object classes, where applicable, are available. Even though hard to compare, we provide results on the dataset. The Freiburg Sitting People dataset  consists of 200 images with 14 part segmentation and is tailored towards sitting poses. The ideas by Shotton et al.  for 2.5D data inspired our body part representation. Relatively simple methods have proven to achieve good performance in segmentation tasks with "easy" backgrounds like Human80k, a subset of Human3.6M .

Following previous work on cardboard people  and contour people , an attempt to work towards an intermediate-level person representation is the JHMDB dataset and the related labeling tool . It relies on 'puppets' to ease the annotation task, while providing a higher level of detail than solely joint locations.

The attempt to unify representations for human bodies has been made mainly in the context of human kinematics . In their work, a rich representation for 3D motion capture marker sets is used to transfer captures to different targets. The setup of markers to capture not only human motion but also shape has been explored by Loper et al.  for motion capture scenarios. While they optimized the placement of markers for a 12 camera setup, we must ensure that the markers disambiguate pose and shape from a single view. Hence, we use a denser set of markers.

# Building the Initial Dataset

Our motivation to use a common 3D representation is to (1) map many possible representations from a variety of datasets to it, and (2) generate detailed and consistent labels for supervised model training from it.

We argue that the use of a full human body model with a prior on shape and pose is necessary: without the visualization possibilities and regularization, it may be impossible to create sufficiently accurate annotations for small body parts. However, so far, no dataset is available that provides human body model fits on a large variety of images.

To fill this gap, we build on a set of human pose datasets with annotated keypoints. SMPLify  presented promising results for automatically translating these into 3D body model fits. This helps us to keep the human involvement to a minimum. With strongly increasing working times and levels of label noise for increasingly complex tasks, this may be a critical decision to create a large dataset of 3D body models.

## Improving Body Shape Estimation

In , the authors fit the pose and shape parameters of the SMPL  body model to 2D keypoints by minimizing an objective function composed of a data term and several penalty terms that represent priors over pose and shape. However, the connection length between two keypoints is the only indicator that can be used to estimate body shape. Our aim is to match the shape of the body model as accurately as possible to the images, hence we must incorporate a shape objective in the fitting.

The best evidence for the extent of a 3D body projected on a 2D image is encoded by its silhouette. We define the silhouette to be the set of all pixels belonging to a body's projection. Hence, we add a term to the original SMPLify objective to prefer solutions for which the image silhouette, $S$, and the model silhouette, $\hat{S}$, match.

Let $M(\vec{\theta}, \vec{\beta}, \vec{\gamma})$ be a 3D mesh generated by a SMPL body model with pose, $\vec{\theta}$, shape, $\vec{\beta}$, and global translation, $\vec{\gamma}$. Let $\Pi(\cdot, K)$ be a function that takes a 3D mesh and projects it into the image plane given camera parameters $K$, such that $\hat{S}(\vec{\theta}, \vec{\beta}, \vec{\gamma})=\Pi(M(\vec{\theta},
\vec{\beta}, \vec{\gamma}))$ represents the silhouette pixels of the model in the image. We compute the bi-directional distance between $S$ and $\hat{S}(\cdot)$ $$\begin{aligned}
E_{S}(\vec{\theta},\vec{\beta},\vec{\gamma};S,K) & = & \sum_{\vec{x}\in\hat{S}(\vec{\theta},\vec{\beta},\vec{\gamma})}\mathrm{dist}{(\vec{x},S)}^{2}\nonumber \\
 &  & +\sum_{\vec{x}\in S}\mathrm{dist}(\vec{x},\hat{S}(\vec{\theta},\vec{\beta},\vec{\gamma})),\label{eq:smpl}
\end{aligned}$$ where $\mathrm{dist}(\vec{x},S)$ denotes the absolute distance from a point $\vec{x}$ to the closest point belonging to the silhouette $S$.

The first term in Eq.  computes the distance from points of the projected model to a given silhouette, while the second term computes the distance from points in the silhouette to the model. We find that the second term is noisier and use the plain L1 distance to measure its contribution to the energy function while we use the squared L2 distance to measure the contribution of the first. We optimize the overall objective including this additional term using OpenDR , just as in .

```latex
\begin{centering}
    \begin{tabular}{|c|c|c|c|}
      \hline
      {\scriptsize{}Dataset} & {\scriptsize{}Foreground} & {\scriptsize{}6 Body Parts} & {\scriptsize{}AMT hours logged} \\
      \hline
      {\scriptsize{}LSP~\cite{LSP:2010}} & {\scriptsize{}\pbox{20cm}{1000 train,\\1000 test{\scriptsize{}}}} & {\scriptsize{}\pbox{20cm}{1000 train,\\1000 test{\scriptsize{}}}} & {\scriptsize{}361h foreground},\\
      \cline{1-3}
      {\scriptsize{}LSP-extended~\cite{lspext}} & {\scriptsize{}10000 train} & {\scriptsize{}0} & {\scriptsize{}131h parts} \\
      \hline
      {\scriptsize{}MPII-HPDB~\cite{mpii}} & {\scriptsize{}\pbox{20cm}{13030 train,\\2622 test{\scriptsize{}}}} & {\scriptsize{}0} & {\scriptsize{}729h}\\
      \hline
    \end{tabular}
    \par\end{centering}
```

Whereas it would be possible to use an automatic segmentation method to provide foreground silhouettes, we decided to involve human annotators for reliability. We also asked for six body part segmentation that we will use in Sec.  for evaluation. We built an interactive annotation tool on top of the *Opensurfaces* package  to work with Amazon Mechanical Turk (AMT). To obtain image-consistent silhouette borders, we use the interactive Grabcut algorithm . Workers spent more than 1,200 hours on creating the labels for the LSP  datasets as well as the single-person part of the MPII-HumanPose  dataset (see Tab. ). There is an increase in average annotation time of more than a factor of two comparing annotation for foreground labels and six body part labels. This provides a hint on how long annotation for a 31 body part representation could take. Examples for six part segmentation labels are provided in Fig. .

## Handling Noisy Ground Truth Keypoints

The SMPLify method is especially vulnerable to missing annotations of the four torso joints: it uses their locations for an initial depth guess, and convergence deteriorates if this guess is of poor quality.

Finding a good depth initialization is particularly hard due to the foreshortening effect of the perspective projection. However, since we know that only a shortening but no lengthening effect can occur, we can find a more reliable person size estimate $\hat{\theta}$ for a skeleton model with $k$ connections: $$%\hat\theta=\underset{i=1,\ldots,k}{\max}\,\mathbf{x}_i\cdot
%\underset{y}{\arg\max} f_i(y),
  \hat\theta=\mathbf{x}_i\cdot \underset{y}{\arg\max} f_i(y),\;\;
  i=\underset{j=1,\ldots,k}{\arg\max}\,\mathbf{x}_j,$$ where $f_i$ is the distribution over ratios of person size to the length of connection $\textbf{x}_i$. Since this is a skewed distribution, we use a corrected mean to find the solution of the $\arg\max$ function and obtain a person size estimate. This turns out to be a simple, yet robust estimator.

## Exploring the Data

With the foreground segmentation data and the adjustments described in the preceding sections, we fit the SMPL model to a total of 27,652 images of the LSP, LSP-extended, and MPII-HumanPose datasets. We use only people marked with the 'single person' flag in MPII-HumanPose to avoid instance segmentation problems. We honor the train/test splits of the datasets and keep images from their test sets in our new, joined test set.

In the next step, human annotators[^1] selected the fits where rotation and location of body parts largely match the image evidence. For this task, we provide the original image, as well as four perspectives of renderings of the body. Optionally, annotators can overlay rendering and image. These visualizations help to identify fitting errors quickly and reduce the labeling time to $\sim$`<!-- -->`{=html}12s per image. The process uncovered many erroneously labeled keypoints, where mistakes in the 3D fit were clear to spot, but not obvious in the 2D representation. We excluded head and foot rotation as criteria for the sorting process. There is usually not sufficient information in the original 14 keypoints to estimate them correctly. The resulting ratios of accepted fits can be found in Tab. .

```latex
\begin{centering}
    \resizebox{0.48\textwidth}{!}{
    \begin{tabular}{|c|c|c|c|}
      \hline
      \scriptsize{}LSP~\cite{LSP:2010}& \scriptsize{}LSP extended~\cite{lspext} &\scriptsize{} MPII-HP~\cite{mpii} & \scriptsize{}FashionPose~\cite{Fashionpose:2014}\\
      \hline
      \hline
      \scriptsize{}45\% &\scriptsize{} 12\% & \scriptsize{}25\% &\scriptsize{} 23\%\\
      \hline
    \end{tabular}
    \par
  }\end{centering}
```

Even with the proposed, more robust initialization term, the ratio of accepted fits on the LSP-extended dataset remains the lowest. It has the highest number of missing keypoints of the four datasets, and at the same time the most extreme viewpoints and poses. On the other hand, the rather high ratio of usable fits on the LSP dataset can be explained with the clean and complete annotations.

The validated fits form our initial dataset with 5,569 training images (of which we use a held-out validation set of 1,112 images in our experiments) and 1,208 test images. We denote this dataset as UPI-3D (**U**nited**P**eople in 3D with an added 'I' for "Initial"). To be able to clearly reference the different label types in the following sections, we add an 'h' to the dataset name when referring to labels from human annotators.

#### Consistency of Human Labels

The set of curated 3D fits allows us to assess the distribution of the human-provided labels by projecting them to the UPI-3D bodies. We did this for both, keypoints and body part segments. Visualizations can be found in Fig. .

While keypoint locations in Fig.  in completely non-matching areas of the body can be explained by self-occlusion, there is a high variance in keypoint locations around joints. It must be taken into account that the keypoints are projected to the body surface, and depending on person shape and body part orientation some variation can be expected. Nevertheless, even for this reduced set of images with very good 3D fits, high variance areas, , around the hip joints, indicate labeling noise.

The visualization in Fig.  shows the density of part types for six part segmentation with the segments head, torso, left and right arms and left and right legs. While the head and lower parts of the extremities resemble distinct colors, the areas converging to brown represent a mixture of part annotations. The brown tone on the torso is a clear indicator for the frequent occlusion by the arms. The area around the hips is showing a smooth transition from torso to leg color, hinting again at varying annotation styles.

# Label Generation and Learning

In a comprehensive series of experiments, we analyze the quality of labels generated from UPI-3D. We focus on labels for well-established tasks, but highlight that the generation possibilities are not limited to them: all types of data that can be extracted from the body model can be used as labels for supervised training. In our experiments, we move from surface (segmentation) prediction over 2D- to 3D-pose and shape estimation to a method for predicting 3D body pose and shape directly from 2D landmark positions.

## Semantic Body Part Segmentation

We segment the SMPL mesh into 31 regions, following the segmentation into semantic parts introduced in  (for a visualization, see Fig. ). We note that the Kinect tracker works on 2.5D data while our detectors only receive 2D data as input. We deliberately did not make any of our methods for data collection or prediction dependent on 2.5D data to retain generality. This way, we can use it on outdoor images and regular 2D photo datasets. The **S**egmentation dataset UPI-S31 is obtained by projecting the segmented 3D mesh posed on the 6,777 images of UPI-3D.

Following , we optimize a multiscale ResNet101 on a pixel-wise cross entropy loss. We train the network on size-normalized, cutout images, which could in a production system be provided by a person detector. Following best practices for CNN training, we use a validation set to determine the optimal number of training iterations and the person size, which is around 500 pixels. This high resolution allows the CNN to reliably predict small body parts. In this challenging setup, we achieve an intersection over union (IoU) score of 0.4432 and an accuracy of 0.9331. Qualitative results on five datasets are shown in Fig. .

The overall performance is compelling: even the small segments around the joints are recovered reliably. Left and right sides of the subjects are identified correctly, and the four parts of the head provide an estimate of head orientation. The average IoU score is dominated by the small segments, such as the wrists.

The VOC part dataset is a hard match for our predictor: instead of providing instances of people, it consists of entire scenes, and many people are visible at small scale. To provide a comparison, we use the instance annotations from the VOC-Part dataset, cut out samples and reduce the granularity of our segmentation to match the widely used six part representation. Because of the low resolution of many displayed people and extreme perspectives with, , only a face visible, the predictor often only predicts the background class on images not matching our training scheme. Still, we achieve an IoU score of 0.3185 and 0.7208 accuracy over the entire dataset without finetuning.

Additional examples from the LSP, MPII-HumanPose, FashionPose, Fashionista, VOC, HumanEva and Human3.6M datasets are shown in the supplementary material available on the project homepage[^2]. material. The model has not been trained on any of the latter four, but the results indicate good generalization behavior. We include a video to visualize stability across consecutive frames.

## Human Pose Estimation

With the 3D body fits, we can not only generate consistent keypoints on the human skeleton but also on the body surface. For the experiments in the rest of this paper, we designed a 91-landmark[^3] set to analyze a dense keypoint set.

We distributed the landmarks according to two criteria: disambiguation of body part configuration and estimation of body shape. The former requires placement of markers around joints to get a good estimation of their configuration. To satisfy the latter, we place landmarks in regular intervals around the body to get an estimate of spatial extent independent of the viewpoint. We visualize our selection in Fig.  and example predictions in Fig. .

In the visualization of predictions, we show a subset of the 91 landmarks and only partially connect the displayed ones for better interpretability. The core 14 keypoints describing the human skeleton are part of our selection to describe the fundamental pose and maintain comparability with existing methods.

We use a state-of-the-art DeeperCut CNN  for our pose-related experiments, but believe that using other models such as Convolutional Pose Machines  or Stacked Hourglass Networks  would lead to similar findings.

To assess the influence of the quality of our data and the difference of the loss function for 91 and 14 keypoints, we train multiple CNNs: (1) using all human labels but on our (smaller) dataset for 14 keypoints (UPI-P14h) and (2) on the dense 91 landmarks from projections of the SMPL mesh (UPI-P91). Again, models are trained on size-normalized crops with cross-validated parameters. We include the performance of the original DeeperCut CNN, which has been trained on the full LSP, LSP-extended and MPII-HumanPose datasets (in total more than 52,000 people) in the comparison with the models being trained on our data (in total 5,569 people). The results are summarized in Tab. .

```latex
\begin{centering}
    \begin{tabular}{|c||c|c||c|}
      \hline
      \firstname.lastname@example.com & \scriptsize{}{UP}I-P14h & \scriptsize{}{UP}I-P14 & \scriptsize{}{UP}I-P91\\
      \hline
      \hline
\scriptsize{}DeeperCut CNN~\cite{deepercut} & \scriptsize{}\textbf{93.45} & \scriptsize{}92.16 & \scriptsize{}NA \\
      \hline
      \hline
\scriptsize{}Ours (trained on {UP}I-P14h) & \scriptsize{}89.11 & \scriptsize{}87.36 & \scriptsize{}NA \\
      \hline
      \scriptsize{}Ours (trained on {UP}I-P91) & \scriptsize{}\textbf{91.15} & \scriptsize{}\textbf{93.24} & \scriptsize{}\textbf{93.54} \\
      \hline
    \end{tabular}
  \par\end{centering}
```

Even though the size of the dataset is reduced by nearly an order of magnitude, we maintain high performance compared to the original DeeperCut CNN. Comparing the two models trained on the same amount of data, we find that the model trained on the 91 landmarks from the SMPL data has a notable advantage of nearly six score points on the SMPL labeled data (row 2 vs. 3, column 2). Even when evaluating on the human labeled data, it maintains an advantage of two score points (row 2 vs. 3, column 1). This shows that the synthetic keypoints generalize to the human labels, which we take as an encouraging result.

We provide the third column for giving an impression of the performance of the additional 77 landmarks. When including the additional landmarks in the evaluation, the score rises compared to evaluating on the 14 core keypoints, indicating their overall performance is above average. A direct comparison to the 14 keypoint values is not valid, because the score is averaged over results of differing 'difficulty'.

#### Integrating a Dataset with a Different Label Set

The current state-of-the art pose estimators benefit from training on all human pose estimation datasets with a similar label set. The FashionPose dataset  would complement these well, but is annotated with a different set of keypoints: the neck joint is missing and the top head keypoint is replaced by the nose. Due to this difference, it is usually not included in pose estimator training.

Using our framework, we can overcome this difficulty: we adjust the fitting objective by adding the nose to and removing the top-head keypoint from the objective function. We fit the SMPL model to the FashionPose dataset and curate the fits. The additional data enlarges our training set by 1,557 images and test set by 181 images. This forms the full UP-3D dataset, which we use for all remaining experiments.

We train an estimator on the landmarks projected from the full UP-3D dataset. This estimator outperforms the plain DeeperCut CNN with a small margin from 0.897 email@example.com (DeeperCut) to 0.9028 firstname.lastname@example.com (ours) on the full, human labeled FashionPose test set.

## 3D Human Pose Estimation

In this section, we analyze the impact of using the 91 predicted keypoints instead of 14 for the SMPLify 3D fitting method. For the fitting process, we rely solely on the 91 predicted 2D landmarks and no additional segmentation or gender information (in contrast to the SMPLify method as described in , where gender information is used for fitting on the 3D datasets). Segmentation information is not required anymore to estimate body extent due to the landmarks on the body surface.

#### LSP dataset

On the LSP dataset, there is no ground truth for 3D body model fitting available. To be independent of biases towards a specific keypoint set, we rely on the acquired six body part segmentation to obtain meaningful performance scores (see Tab. ).

The six-part manual segmentation annotations consist of head, torso, left and right leg, and left and right arm (see Fig. ). While this representation is coarse, it provides a good estimate of the overall quality of a fit. It takes into account the body shape and not only keypoints, hence it is a fair judge for pose estimators aiming for slightly different keypoint locations.

```latex
\resizebox{0.48\textwidth}{!}{
    {\scriptsize{}}%
    \begin{tabular}{|c|c|c|}
      \hline
      & {\scriptsize{}FB Seg. acc., f1} & {\scriptsize{}P Seg acc., f1}\\
      \hline
      \hline
      {\scriptsize{}SMPLify on GT lms.} & {\scriptsize{}0.9176, 0.8811} & {\scriptsize{}0.8798, 0.6584}\\
      \hline
      {\scriptsize{}SMPLify on GT lms. \& GT seg.} & \textbf{\scriptsize{}0.9217, 0.8823} & \textbf{\scriptsize{}0.8882, 0.6703}\\
      \hline
      \hline
      {\scriptsize{}SMPLify on DeepCut CNN lms.~\cite{paper2}} & \textbf{\scriptsize{}0.9189, 0.8807} & \textbf{\scriptsize{}0.8771, 0.6398}\\
      \hline
      \hline
      {\scriptsize{}SMPLify on our CNN lms., tr. UPI-P14h} & {\scriptsize{}0.8944, 0.8401} & {\scriptsize{}0.8537, 0.5762}\\
      \hline
      {\scriptsize{}SMPLify on our CNN lms., tr. UP-P14} & {\scriptsize{}0.8952, 0.8475} & {\scriptsize{}0.8588, 0.5798}\\
      \hline
      {\scriptsize{}SMPLify on our CNN lms., tr. UP-P91} & \textbf{\scriptsize{}0.9099, 0.8619} & \textbf{\scriptsize{}0.8732, 0.6164}\\
      \hline
      \hline
      {\scriptsize{}DP from 14 landmarks} & {\scriptsize{}0.8649, 0.7915} & {\scriptsize{}0.8223, 0.4957}\\
      \hline
      {\scriptsize{}DP from 91 landmarks} & {\scriptsize{}0.8666, 0.7993} & {\scriptsize{}0.8232, 0.5102}\\
      \hline
      {\scriptsize{}DP from 14 lms., rotation opt.} & {\scriptsize{}0.8742, 0.8102} & {\scriptsize{}0.8329, 0.5222}\\
      \hline
      {\scriptsize{}DP from 91 lms., rotation opt.} & \textbf{\scriptsize{}0.8772, 0.8156} & \textbf{\scriptsize{}0.8351, 0.5304}\\
      \hline
    \end{tabular}{\scriptsize \par}
    }
```

Unsurprisingly, the segmentation scores of the SMPLify method improve when the segmentation term ( Sec. ) is added. Due to the longer-trained pose estimator, SMPLify as presented in  still has an overall advantage on the LSP dataset (compare rows three and four).

Training on our generated data for 14 joints and then using SMPLify improves the scores (compare lines four and five) thanks to cleaner data and better correspondence of keypoints and SMPL skeleton. Using our 91 landmark model gives a large performance boost of 3.6 f1 score points. We do not reach the performance of the fits performed on the DeepCut CNN  predictions, largely because of few extreme poses that our pose estimator misses with a large influence on the final average score.

#### HumanEva and Human3.6M Datasets

We evaluate 3D fitting on the HumanEva and Human3.6M datasets where 3D ground truth keypoints are available from a motion capture system. We follow the evaluation protocol of SMPLify  to maintain comparability, except for subsampling to every 5th frame. This still leaves us with a framerate of 10Hz which does not influence the scores. We do this solely due to practical considerations, since the SMPLify fitting to 91 landmarks can take up to twice as long as fitting to 14 keypoints. We provide a summary of results in Tab. .

|                                       | HumanEva | Human3.6M |
|:-------------------------------------:|:--------:|:---------:|
|             Zhou et al.               |  110.0   |   106.7   |
|         DP from 91 landmarks          |   93.5   |   93.9    |
|     SMPLify on DeepCut CNN lms.       |   79.9   |   82.3    |
| SMPLify on our CNN lms., tr. UPI-P14h |   81.1   |   96.4    |
|  SMPLify on our CNN lms., tr. UP-P14  |   79.4   |   90.9    |
|  SMPLify on our CNN lms., tr. UP-P91  | **74.5** | **80.7**  |

Average error over all joints in 3D distance (mm).

We do not use actor or gender specific body models, but one hybrid human model, and rely on the additional landmarks for shape inference. This makes the approach fully automatic and deployable to any sequence without prior knowledge. Even with these simplifications and a magnitude fewer training examples for our pose estimator, we achieve an improvement of 5.4mm on average on the HumanEva dataset and an improvement of 1.6mm on average on the Human3.6M dataset (4th versus 5th row).

The use of a pose estimator trained on the full 91 keypoint dataset UP-P91 improves SMPLify even more. Compared to the baseline model trained on UPI-P14h, performance improves by 6.6mm on the simpler HumanEva dataset, and by 15.7mm on Human3.6M. Even when training a 14 keypoint pose estimator, the higher consistency of our generated labels helps to solve this task, which becomes apparent comparing lines four and five.

## Direct 3D Pose and Shape Prediction

The 91 landmark predictions in 2D enable a human observer to easily infer the 3D shape and pose of a person: the keypoints on the body surface provide a good hint to estimate person shape and in combination with the skeleton orientation and pose can usually be identified (Fig. ). This observation inspired us to explore the limits of a predictor for the 3D body model parameters directly from the 2D keypoint input. For this purpose, we use the 3D poses and shapes from the UP-3D dataset to sample projected landmarks with the full 3D parameterization of SMPL as labels. We move a virtual 'camera' for every pose on 5 elevations to 36 positions around the 3D model to enhance the training set. On this data, we experimented with multi-layer perceptrons as well as Decision Forests. We preferred the latter regressor, since Decision Forests are less susceptible to noise. We train a separate forest to predict the axis-angle rotation vector for each of the 24 SMPL joints, as well as one to predict the depth. The input landmark positions are normalized w.r.t. position and scale to improve generalization. We experimented with distance-based features and dot-product features from the main skeleton connections, but these were not as robust as plain 2D image coordinates. It turned out to be critical to use full rotation matrices as regression targets: the axis-angle representation has discontinuities, adding noise to the loss function.

One Decision Forest predicts pose or shape in  0.13s[^4]. The predictions of all forests are independent, which means that the full pose and shape prediction can be obtained in between one and two orders of magnitudes faster than with SMPLify. This could allow the use of 3D pose and shape estimation for video applications, , action recognition.

Whereas the 3D model configuration does not always match the image evidence (see Fig. ), it recovers the rough pose. We provide scores in Tab.  and Tab.  with the name 'DP' (**D**irect **P**rediction). We additionally add the scores for a hybrid version, for which we predict pose and shape using Decision Forests and take few optimization steps to make the global rotation of the body model match the image evidence (with varying runtime depending on the initialization, but less than one second on our data).

The difference between the full optimization on the LSP dataset in f1 score is 0.1062 for the 91 landmark based method and reduces with rotation optimization to 0.086. On the 3D datasets, the direct prediction method outperforms all optimization based methods except for SMPLify that runs in the order of tens of seconds.

Together with our ResNet101-based CNN model, it is possible to predict a full 3D body model configuration from an image in 0.378s. The pose-predicting CNN is the computational bottleneck. Because our findings are not specific to a CNN model, we believe that by using a speed-optimized CNN, such as SqueezeNet , and further optimizations of the direct predictor, the proposed method could reach real-time speed.

# Closing the Loop

With the improved results for 3D fitting, which helped to create the dataset of 3D body model fits in the first place, a natural question is, whether the improved fitting method helps to enlarge the dataset.

We ran SMPLify on the 91 landmark predictions from our pose estimator and again asked human annotators to rate the 3D fits on all LSP images that were not accepted for our initial dataset. Of the formerly unused 54.75% of the data (1095 images), we found an improvement in six body part segmentation f1 score for 308 images ( Fig. ). We show three example images with high improvement in f1 score in Fig. , (b) to (d): improvement due to left-right label noise, depth ambiguity and perspective resolution compared to fits on the 14 ground truth keypoints. Human annotators accepted additional 185 images, which is an improvement of 20% over the number of accepted initial fits and an absolute improvement of 9.3% in accepted fits of the LSP dataset.

The most common reasons for improvement are (1) noisy annotations, (2) better perspective resolution and (3) the better match of keypoints to the SMPL skeleton. An even higher ratio of improvement can be expected for the datasets with more annotation noise, such as the LSP-extended and MPII-HumanPose datasets. This enlarged set of data could be used to again train estimators and continue iteratively.

# Discussion

With the presented method and dataset, we argue for a holistic view on human related prediction tasks. By improving the representation of humans we could integrate datasets with different annotations and approach established tasks at a new level of detail. The presented results include high fidelity semantic body part segmentation into 31 parts and 91 landmark human pose estimation. This sets a new mark in terms of levels of detail that previous work did not reach. At the same time, it helps to improve the state-of-the art for 3D human pose estimation on the two standard benchmark datasets HumanEva and Human3.6M.

We present a regression tree model that predicts the 3D body configuration from 2D keypoints directly. This method runs orders of magnitude faster than optimization based methods. This direct prediction captures the overall pose from simple 2D input reasonably well and we are optimistic that it can be scaled to reach near real-time performance. We show that the improved 3D fitting method allows more good fits that enlarge the training set. Here, we only took one iteration but are confident that a system that iterates over the two generative and discriminative stages can be deployed on large scale to continuously learn and improve with very limited human feedback.

[^1]: For this task, we did not rely on AMT workers, but only on few experts in close collaboration to maintain consistency.

[^2]: <http://up.is.tuebingen.mpg.de/>

[^3]: We use the term 'landmark' to refer to keypoints on the mesh surface to emphasize the difference to the so-far used term 'joints' for keypoints located inside of the body.

[^4]: For all timings, a test system with a 3.2Ghz six core processor and an NVIDIA GeForce GTX970 has been used.


abstract: Using ab-initio calculations, we study the electronic structure of gallium-doped germanium, which was found recently to be a superconductor, with a critical temperature of $0.5$ Kelvins, and a particularly low density of Cooper pairs. The calculations of the electronic properties reveal that no sign of an impurity band is observed, and that the Fermi level lies in the valence band of Germanium. Moreover, the calculation of the phonon frequencies shows that a new mode associated to the Ga atom is appearing, around $175$ cm$^{-1}$.
author: S. Lebègue
date: 2024-10-02
title:  Electronic structure of superconducting gallium-doped germanium from ab-initio calculations.

The surprising discovery of the superconductivity of MgB$_2$, with a critical temperature of about $40$ Kelvins, has revived the interest of the scientific community for the systems presenting strong covalent bonds. Later on, the superconductivity phenomenon was observed in related systems like (among others) boron-doped diamond, silicon, and silicon-carbide. These findings are explained by the fact that strong covalent bonds and light elements give rise to high phonon frequencies and large electron-phonon coupling, leading sometimes to superconductivity. In the framework of the standard electron-phonon coupling mechanism, ab-initio calculations are particularly useful, since it is possible to draw an almost complete picture of what is occurring, through the calculations of the electronic structure, the phonons spectra, and even an estimation of the critical temperature with the help of the McMillan formula. Therefore, superconductivity has been well explained for example in the case of MgB$_2$ or boron doped diamond. This is in contrast with the situation on copper oxides or iron-based superconductors, for which ab-initio calculations can reveal the electronic structure, but are limited concerning the description of the mechanism leading to superconductivity.

Very recently, a new member of the superconducting semiconductors has been found, by doping germanium with gallium atoms. The critical temperature was found to be $0.5$ K at ambient pressure, and a very low Cooper-pair density was observed. However, the information concerning the electronic structure of this material are still very limited. It is the aim of this publication to report results obtained by ab-initio calculations on Ga-doped germanium, in particular the electronic density of states, the bandstructure, and the phonon spectra.

To perform the calculations, we have used the code VASP (Vienna Ab-initio simulation package), implementing the projector augmented waves method within density functional theory (DFT) . The Perdew Burke Ernzerhof variant of the generalized gradient approximation (GGA) was used for the exchange-correlation (XC) potential. We have built a germanium $2 \times 2 \times 2$ supercell (containing $16$ atoms), with one Ga atom substituting a germanium atom. This corresponds to a $6.25 \%$ doping, which is close to the percentage used in the experiments ($6 \%$). Then, the system was allowed to relax completely. In this case, a $6 \times 6 \times 6$ $k$-points mesh was used, whereas to obtain the density of states and bandstructures, a denser $18 \times 18 \times 18$ mesh was used. The plane wave cut-off was set to $500$ eV.

Due to the doping, the crystal structure is modified to accommodate the gallium atom in the cell. In pure germanium, the Ge-Ge bond length is about $2.50~$ Å (when relaxed with PBE), while upon doping, the Ga-Ge distance is $2.47$ Å, and the Ge-Ge distance is maintained to almost $2.50$ Å. Therefore, the crystal structure is only slightly affected by the replacement of a Ge atom by a Ga atom, and these structural changes remain localized around the Ga impurity.

In Fig. , we present our calculated total and partial density of states (DOS, PDOS) projected on the gallium atom and on one of the germanium atom neighbouring it. A metallic regime is clearly observed, with the Fermi level now being approximately $0.6$ eV below the valence band maximum of pure Ge (we are referring to a valence band maximum although here the density of states has always a finite value, due to the well known underestimation of the band gap with standard XC functional such as GGA). Also, a Fermi surface with three sheets (not shown here) emerges around the center of the Brillouin zone.

Moreover, there is no sign of an impurity band in the electronic gap, so at first sight the model of rigid bands doping seems likely, but this is not confirmed by the bandstructure calculations: in Fig. , the bandstructures of pure (upper plot) and Ga-doped (lower plot) germanium are presented, for the $2 \times 2 \times 2$ supercell. Due to the doping, some degeneracies are left, the bandstructure is significantly modified, and therefore the picture of a simple rigid band model is not valid.

The phonons DOS ($\Gamma$ point only) of pure and Ga-doped Germanium are presented in Fig. . As expected, some degeneracies are lifted upon the introduction of the Ga atom, and the spectra is changed in comparison with the one of pure Ge. However, this is only affecting the high energy part of the spectrum, with a softening of the frequencies. In particular, some modes are appearing around $175$ cm$^{-1}$ and $178$ cm$^{-1}$. By inspection of the eigenvectors, the new modes at $175$ cm$^{-1}$ are associated with a large displacement of the Ga atom. Therefore, it is possible that this specific mode is responsible of the superconductivity in Ga-doped Germanium, through a change of the electron-phonon coupling, as observed earlier for boron doped diamond. However, this can be ensured only by the calculation of the Eliashberg function, since on the contrary, for B-doped SiC, it was found that the superconducting behaviour is controlled by the whole vibrational density of states.

In summary, we have computed the electronic and vibrational properties of Ga-doped Germanium. We observed that the effect of doping is to shift the Fermi level in the valence band of Germanium. Concerning the vibrational properties, a mode around $175$ cm$^{-1}$ is associated with Ga atoms. We hope that our work will stimulate further theoretical work, such as the calculation of the strength of the electron-phonon coupling and an estimate of the superconducting transition temperature, or further experiments, such as photoemission or neutron diffraction experiments, to obtain an even better picture of the properties of Ga-doped Germanium.

I acknowledge financial support from ANR grant No. ANR-06-NANO-053-02 and ANR Grant ANR-BLAN07-1-186138 as well as CINES-CCRT for computer time.


abstract: We present luminosity functions derived from a spectroscopic survey of AGN selected from *Spitzer Space Telescope* imaging surveys. Selection in the mid-infrared is significantly less affected by dust obscuration. We can thus compare the luminosity functions of the obscured and unobscured AGN in a more reliable fashion than by using optical or X-ray data alone. We find that the AGN luminosity function can be well described by a broken power-law model in which the break luminosity decreases with redshift. At high redshifts ($z>1.6$), we find significantly more AGN at a given bolometric luminosity than found by either optical quasar surveys or hard X-ray surveys. The fraction of obscured AGN decreases rapidly with increasing AGN luminosity, but, at least at high redshifts, appears to remain at $\approx 50$% even at bolometric luminosities $\sim 10^{14}L_{\odot}$. The data support a picture in which the obscured and unobscured populations evolve differently, with some evidence that high luminosity obscured quasars peak in space density at a higher redshift than their unobscured counterparts. The amount of accretion energy in the Universe estimated from this work suggests that AGN contribute about 12% to the total radiation intensity of the Universe, and a high radiative accretion efficiency $\approx 0.18^{+0.12}_{-0.07}$ is required to match current estimates of the local mass density in black holes.
author: M. Lacy , S.E. Ridgway, A. Sajina, A.O. Petric, E.L. Gates , T. Urrutia, L.J. Storrie-Lombardi
title: The *Spitzer* mid-infrared AGN survey. II-the demographics and cosmic evolution of the AGN population.

# Introduction

The existence of a large population of obscured AGN has been recognized for as long as AGN have been known. The vast majority of well-studied obscured objects, however, are local, of relatively low bolometric luminosity. The fraction of obscured AGN at the high (quasar) end of the AGN luminosity function, where the population is at cosmological distances has been the subject of debate. Reliable statistics exist only for the $\approx 10$% of radio-loud AGN, where luminous radio galaxies are seen to exist in similar numbers to radio-loud quasars when selected on an isotropic property such as low frequency radio emission (e.g. Willott et al. 2000). The situation for radio-quiet objects is less certain, and it is only in the last decade that progress has been made in quantifying the population of dust obscured radio-quiet quasars through advances in X-ray, optical and mid-infrared selection techniques.

Selection of AGN in the mid-infrared allows the discovery of powerful AGN and quasars whose optical and soft X-ray emission is hidden by dust (e.g. Lacy et al. 2004, 2007; Stern et al. 2005; 2012; Martínez-Sansigre et al. 2005; Donley et al. 2012; Eisenhardt et al. 2012). Mid-infrared selection has the powerful characteristic that it permits the selection of samples of AGN containing both moderately obscured and unobscured objects of similar bolometric luminosities, allowing an estimate to be made of the importance of the obscured AGN population to the AGN population as a whole. Mid-infrared selection is also a useful complement to other techniques for finding obscured quasars such as hard X-rays (Norman et al. 2002; Brusa et al. 2010), which tend to be restricted to smaller fields and thus lower-luminosity objects, and optical techniques based on narrow-line emission (Zakamska et al. 2003; Alexandroff et al. 2013), which are restricted to redshift ranges in which strong emission lines fall.

Mid-infared selection has its limitations, for example, Juneau et al. (2013) show that, at $z=0.3-1$ and low luminosities ($L_{\rm bol}\sim 10^{10}L_{\odot}$), both X-ray and mid-infrared selection miss a large fraction of AGN that are recovered by the Mass-Excitation technique, an optical diagnostic which compares the \[Oiii\]/H$\beta$ ratio to the stellar mass (Juneau et al. 2011). These objects are probably of such low luminosity that hot dust emission does not stand out against stellar light in mid-infrared color-color plots. Similar incompletenesses to lower luminosity AGN have been noted by Hickox et al. (2009), Park et al. (2010), Mendez et al. (2013) and Messias et al. (2014). In particular, Messias et al. show that the completeness of the AGN selection technique of Lacy et al. (2007) rises from $\approx 50$% compared to X-ray selection at X-ray luminosities $L_X=10^{43} {\rm ergs^{-1}}$ to 100% complete at $L_X=10^{44} {\rm ergs^{-1}}$ (i.e. 5$\mu$m luminosities $\nu L_{\rm 5mu}\sim 10^{43.6}$ to $10^{44.6} {\rm ergs^{-1}}$). Nevertheless, mid-infrared selection finds many AGN that are missing from X-ray surveys even at low luminosities (e.g. Park et al. 2008), so mid-infrared remains a powerful technique for finding AGN.

Quantifying the abundance of the obscured population is important for two reasons. First, it affects the argument of Soltan (1982), where the total amount of matter accreting onto black holes is compared to the remnant black hole mass density today. The comparison allows us to estimate the mean value of the radiative efficiency of accretion, $\eta$, which varies from $\approx 0.06$ for a non-rotating (Schwartzshild) black hole to $\approx 0.3$ for a maximally-rotating Kerr black hole. Previous estimates suggest $\eta \approx 0.1$ (e.g. Hopkins, Richards & Hernquist 2007, hereafter H07; Martínez-Sansigre & Taylor 2009; Delvecchio et al. 2014), though an analysis of the X-ray background (Elvis, Risaliti & Zamorani 2002) suggests a higher bound, $\eta > 0.15$, consistent with most black holes spinning and with current AGN censuses being significantly incomplete. The second reason is to aid in our understanding of unified schemes of AGN. In the standard orientation-based unified scheme (e.g. Antonucci 1993), the ratio of obscured to unobscured quasars defines the opening angle of the obscuring torus. (In a refinement of this model, Elitzur (2012) suggests that the clumpy nature of the torus means that on an object-by-object basis the angle to the line of sight is only a statistical predictor of obscuration, nevertheless the obscured to unobscured ratio is still related to the torus opening angle in a statistical sense.) The luminosity dependence of the obscured to unobscured ratio (Lawrence 1991; Simpson 2005; Gilli, Comastri & Hasinger 2007; Lusso et al. 2013) has been put forward as evidence of the "receding torus" model in which more luminous objects are able to move the inner radius of the obscuring torus out, reducing the covering factor of the dust. Competing with this theory, in the case of high luminosity and (especially) high redshift quasars is the evolutionary theory, whereby quasars begin life highly obscured as a result of mergers of dusty galaxies and eventually break out from their cocoons of dust and gas due to outflows (Sanders et al. 1988; Hopkins et al. 2008). In this case the ratio of obscured to unobscured objects can be used to estimate the fraction of its lifetime that a quasar exists in the obscured state.

To date, estimates of the fraction of dust-obscured quasars have been affected by either small sample size (Martínez-Sansigre et al. 2005; Lacy et al. 2007), a restricted redshift range based on a requirement that \[Oiii\] be detected in an optical spectrum (e.g. Reyes et al. 2008), or a very specific infrared color selection (Assef et al. 2014). All these estimates suggest that the obscured population of quasars equals or exceeds the unobscured population in space density, even at high luminosities. To increase the sample size available to us, we have undertaken a larger survey (detailed in Lacy et al. 2013; hereafter Paper I). This survey is drawn from the 54 deg$^2$ covered by the *Spitzer* Wide-area Infrared Extragalactic Survey (SWIRE; Lonsdale et al. 2003) and the *Spitzer* Extragalatic First Look Survey (XFLS; Lacy et al. 2005; Fadda et al. 2006). AGN and quasars were selected on the basis of their mid-infrared properties from several samples flux-limited at 24$\mu$m, with flux density limits ranging from $S_{24}=$`<!-- -->`{=html}0.61-9.6mJy. Further selection based on mid-infrared colors at 3.6-8.0$\mu$m was then used to remove most non-AGN from the survey and provide a candidate list of 786 objects for which optical and/or near-infrared spectroscopy was obtained. Of these, 672 were confirmed spectroscopically as AGN or quasars. The remainder either showed no firm evidence for an AGN in their spectra, had featureless spectra, or had spectra with only a single weak emission line. In this, the second paper in the series, we compare the fractions of obscured and unobscured quasars in this survey, derive the luminosity functions of the different AGN types, and compare to AGN surveys in other wavebands. We assume a cosmology with $H_0=70 {\rm kms^{-1} Mpc^{-1}}$, $\Omega_M=0.3$ and $\Omega_{\Lambda}=0.7$.

# The evolution of the AGN population as seen in the mid-infrared

## Photometric data

We used the optical through near-infrared photometric data tabulated in Table 2 of Paper I, combined with 12$\mu$m flux densities from the *Wide field Infrared Survey Explorer (WISE)* (Wright et al. 2010). The 12$\mu$m flux densities were particularly useful for constraining the mid-infrared SEDs of the AGN between the *Spitzer* 8$\mu$m and 24$\mu$m bands. We matched the objects in Paper I to the *WISE* all-sky survey using a $3^{''}$ match radius, recovering matches to 896 out of 963 (93%) of objects. (Some of those not matched had faint detections in WISE 12$\mu$m from which we could estimate flux densities.) These data were used as the basis of the spectral energy distribution (SED) fits described in the Appendix.

## The luminosity-redshift plane

Figure 1 shows the redshift-luminosity plane for our survey. The rest-frame 5$\mu$m luminosity, $L_{5\mu {\rm m}}$ was calculated by fitting the spectral energy distribution of each object with either a pure type-1 AGN model, or a combination of an AGN and a host stellar population, as detailed in the Appendix. The presence of infrared emission from the host galaxy may lead to some subtle selection effects, which are also discussed further in the Appendix, though we do not believe they have a significant effect on our derived luminosity function, at least at $z>0.4$.

The rest-frame 5$\mu$m luminosity of the AGN (without a stellar host contribution) was then used to calculate $L_{5\mu {\rm m}}$. The 14 samples from which the survey was drawn, with their differing 24$\mu$m flux density limits and areas, account for the relatively wide luminosity range at a given redshift (a factor of $\approx 30$), in all but the highest redshift bins. For the analysis presented in this paper, we use the "statistical sample" of Paper I, which comprises the 662 objects in a 90% spectroscopically-complete subsample of the entire survey (479 of which are confirmed AGN). The "statistical sample" was defined by exploiting the close correlation of $S_{24}$ and emission line flux density. Objects in each of the 14 samples were sorted in order of decreasing $S_{24}$. The value of $S_{24}$ at which the survey completeness fell below 90% was then noted, and objects with $S_{24}$ brighter than this included in the "statistical sample". This procedure prevents the samples becoming too biassed towards objects which are easy to obtain spectra for (e.g. normal type-1 quasars). As described in Paper I, we classify our AGN as normal, blue type-1 AGN, red type-1 AGN (dust reddened, but still showing broad lines in the rest-frame optical, i.e. with $A_V\sim 1$ towards the AGN), type-2 AGN (showing narrow lines only, even in the rest-frame optical, and thus with $A_V \stackrel{>}{_{\sim}}5$ towards the AGN), non-AGN (selected as AGN candidates but showing no signs of AGN activity in their optical spectra), stars, and objects with featureless spectra. Our statistical sample consists of 122 normal type-1 AGN, 86 red type-1's and 271 type-2's, 118 non-AGN, 23 stars and 42 with featureless spectra.

## The luminosity function

We performed maximum likelihood fits both to the luminosity functions of the AGN population as a whole, and to the individual luminosity functions for the normal (blue) type-1s, type-2s and red type-1s. From an initial $1/V_a$ estimate of the luminosity function, we noted that, although a power-law provided a visually reasonable fit to the data, there was a small, but significant flattening at low luminosities, consistent with the luminosity function being a broken power-law undergoing luminosity evolution (although our data do not probe below the break at $z>1$, so the faint-end evolution is unconstrained at high redshifts).

We therefore fit a double power-law function of the form: $$\phi(L,z) = \frac{\phi^{*}}{(L/L^{*})^{\gamma_1^0}+(L/L^{*})^{\gamma_2^0}},$$ where $\phi$ is the comoving space density of AGN, $\phi^{*}$ is the characteristic space density, both in units of comoving Mpc$^{-3}$, $L$ is the rest-frame luminosity at 5$\mu$m and $L^{*}$ is the break luminosity at 5$\mu$m, both in units of ${\rm ergs^{-1}Hz^{-1}}$. The evolution in $L^{*}$ is the usual cubic expression (e.g. H07): $${\rm log_{10}}L^{*}(z)={\rm log_{10}}L^*_0 +  k_1\epsilon + k_2\epsilon^2 + k_3\epsilon^3,$$ where $\epsilon={\rm log}_{10}((1+z)/(1+z_{\rm ref}))$, $L^*_0$ is a free parameter in the fit, corresponding to the break luminosity at redshift zero, $\gamma_1$ is the faint-end slope and $z_{\rm ref}$ is fixed at $2.5$. We also fit a model with a fixed faint-end by setting $L^*=L^*_0$ in the left-hand term in the denominator of Equation 1 and allowing the bright-end slope, $\gamma_2$ to vary as $$\gamma_2 = \gamma_2^0+\tau \epsilon .$$

To perform the actual fits, we evaluated the log-likelihood function, ${\rm ln}\;\mathcal{L}$, in the usual way (Marshall et al. 1983), by minimizing $S=-2{\rm ln}\mathcal{L}$ (using the Powell minimization routine from the scipy.optimize library[^1]): $$S = -2\sum_{i=1,n_a} {\rm ln}[\phi(L_i,z_i)] +
2 \sum_{j=1,n_f} \int_{L_{\rm min}}^{L_{\rm max}} \int \phi(L,z) A_j \frac{{\rm d}V}{{\rm d}z}{\rm d}z {\rm d}L$$ where the first term sums over the $n_a$ AGN being considered, and the second sums over the $n_f=14$ samples, each of area $A_j$. $L_{\rm min}$ is the minimum luminosity detectable at redshift $z$ in a given sample, and $L_{\rm max}$ the maximum luminosity (for samples where there is an upper flux density limit). For the k-corrections in this calculation, mean \[5.8\]-\[8.0\] and \[8.0\]-\[24\] spectral indices for each AGN type were measured and applied as appropriate to the AGN type and the redshift range of the calculation.

The fitting was performed by optimizing the parameters $\gamma_1$, $\gamma_2$, $\tau$, $\phi^*_0$, $k_1,k_2$, $k_3$ and ${\rm log_{10}}(L_{0}^{*})$ for AGN with $z>0.05$ and $L>10^{29.5}{\rm ergs^{-1}Hz^{-1}}$. We fit all three AGN types separately (normal type-1s, type-2s and red type-1s). We also fit the type-2s and red type-1s together to give a luminosity function for the total obscured population. In order to assess the possible effects of incompleteness, we fit to a set including the objects classified optically as non-AGN (some fraction of which are, in fact, AGN, see Paper I) and also containing the featureless spectrum objects which have no redshifts, distributed evenly in comoving volume between $z=0.5-5$, and refer to this as the "Maximal" fit. The best-fit parameters are given in Table 1.

For the purposes of comparison with previous work, we also fit a pure power law of the form $$\phi_{\rm pl} = \phi^{*}(L/L^{*}[z])^{\gamma_2}$$ where $\gamma_2$ is allowed to vary according to Equation 3.

To determine whether the luminosity evolution model of Equation 1 was a better fit than the fixed faint-end slope model, or the pure power-law model of Equation 5, we examined the likelihood ratio of the fits. For two fits giving values of $S_1$ and $S_2$ using models with $\nu_1$ and $\nu_2$ degrees of freedom, respectively, $\Delta S = S_1 - S_2$ is distributed as a $\chi^2$ distribution with $\nu_1-\nu_2$ degrees of freedom (e.g. Freeman et al. 1998). On this basis, the fixed faint-end slope model is slightly, but not significantly worse than the luminosity evolution model ($\Delta S=1$, with one degree of freedom). The smallness of this difference is perhaps to be expected given the poor constraints on the faint end and its evolution. $\Delta S$ for the pure power-law versus the broken power law model is 30 (Table 1), and the difference in degrees of freedom is only one, so the broken power-law is formally a much better fit.

To make the binned estimates, we binned the data in five redshift bins ($0.05<z<0.4$, $0.4<z<0.9$, $0.9<z<1.6$, $1.6<z<2.5$ and $2.5<z<3.8$) assigning a minimum luminosity to each bin at which the bin was approximately complete (unshaded region in Figure 1). The lower redshift limit was chosen to avoid $z<0.05$, where the Lacy et al. (2004) AGN selection can be confused by the 6.2$\mu$m Polycyclic Aromatic Hydrocarbon (PAH) emission. The bin spacing is in intervals of ${\rm log_{10}}(1+z)\approx 0.13$.

The values of the binned estimates were obtained using the formalism of Hasinger, Miyaji & Schmidt (2005), where the fitted luminosity function of Equation (1) is used to predict the number of sources in each bin, $n_{\rm pred}$ by integrating the luminosity function over the luminosity and redshift ranges for each bin for each individual field in the survey, properly accounting for the upper and lower flux limits in each field. The value of the luminosity function at the bin centre, $\psi_{\rm mdl}$, is then corrected by the ratio of the number of predicted sources in the bin to the actual number observed, $n_{\rm obs}$ - i.e.: $$\phi_{\rm b}^{ij}(L_i,z_j) = \frac{\psi_{\rm mdl} n_{\rm obs}}{n_{\rm pred}}.$$ Although this has the disadvantage of making the binned estimated somewhat model dependent, the advantage of this technique is that it compensates both for the tendency of objects to be found close to the bottom of a luminosity bin, and at the higher redshift end of redshift bins where the luminosity function is increasing rapidly with redshift (i.e. at $z<2$).

|  |  |  |  |  |  |  |  |  |  |  |
|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| AGN used | $n_a$ | $S$ | ${\rm log_{10}}(\phi^*$ | $\gamma_1^0$ | $\gamma_2^0$ | $k_1$ | $k_2$ | $k_3$ | ${\rm log_{10}}(L^*_0$ |  |
|  |  |  | $[{\rm Mpc^{-3}}])$ |  |  |  |  |  | ${\rm [ergs^{-1}Hz^{-1}]})$ | $\tau$ |
| All | 446 | 10997 | -4.75$\pm0.02$ | 1.07$\pm 0.06$ | 2.48$\pm 0.05$ | 1.05$\pm 0.03$ | -4.71$\pm 0.13$ | -0.034$\pm 0.19$ | 31.92$\pm 0.02$ | \[0\] |
| Normal Type-1 | 119 | 3374 | -5.18$\pm 0.05$ | 0.25$\pm 0.18$ | 2.68$\pm 0.10$ | 0.537$\pm 0.08$ | -5.48$\pm 0.23$ | 0.768$\pm 0.76$ | 31.99$\pm 0.08$ | \[0\] |
| Type-2 | 243 | 6065 | -5.04$\pm 0.03$ | 1.068$\pm 0.12$ | 2.75$\pm 0.10$ | 1.01$\pm 0.04$ | -4.26$\pm 0.14$ | -0.241$\pm 0.25$ | 31.76$\pm 0.02$ | \[0\] |
| Red Type-1 | 84 | 2451 | -5.39$\pm 0.05$ | 0.44$\pm 0.20$ | 2.60$\pm 0.12$ | 1.36$\pm 0.09$ | -4.96$\pm 0.27$ | 0.232$\pm 0.56$ | 32.03$\pm 0.03$ | \[0\] |
| Type-2+Red Type-1 | 327 | 8159 | -4.98$\pm 0.03$ | 1.09 $\pm 0.06$ | 2.61$\pm 0.08$ | 1.165$\pm 0.04$ | -4.45$\pm 0.14$ | -0.23$\pm 0.20$ | 31.91$\pm 0.01$ | \[0\] |
| Maximal | 588 | 13830 | -4.66 $\pm 0.02$ | 1.38$\pm 0.04$ | 2.47$\pm 0.07$ | 1.24$\pm 0.05$ | -4.49$\pm 0.07$ | 0.336 $\pm 0.22$ | 31.94$\pm 0.01$ | \[0\] |
| Power-law, All | 446 | 11027 | -5.28$\pm 0.02$ | \- | 2.16$\pm 0.04$ | 1.30$\pm 0.03$ | -4.10$\pm 0.10$ | -0.475$\pm 0.035$ | \[32.0\] | 1.5$\pm 0.1$ |
| Fixed faint end, All | 446 | 10998 | -4.91$\pm0.02$ | 0.57$\pm 0.03$ | 2.72$\pm 0.05$ | 1.26$\pm 0.03$ | -4.90$\pm 0.13$ | 0.539$\pm 0.19$ | 32.02$\pm 0.02$ | 1.5$\pm 0.3$ |

Best fitting luminosity function parameters

Notes:- $n_a$ is the number of AGN used in the fit. $\phi^*, \gamma_1, \gamma_2$ and $L^*_0$ are defined in Equation (1), $k_1,k_2$ and $k_3$ in Equation (2). Parameters fixed in the fit are indicated with square brackets. To convert these to approximate bolometric luminosity functions add ${\rm log_{10}}(8.0 \nu)=14.68$ to ${\rm log_{10}}(L^*_0)$.

## Evolution of the AGN number density with redshift

Figure shows the evolution of the luminosity function of all spectroscopically-confirmed *Spitzer* AGN in our statistical sample with $z>0.05$ and $L_{\rm 5 \mu m}>10^{29.5} {\rm ergs^{-1}Hz^{-1}}$. The left-hand panel, Figure (a), shows that the luminosity evolution model seems to fit the data well across most of the range of redshift and luminosities. In the center panel, (b) we show the alternative fixed faint-end slope and simple power-law models, both of which seem to represent the data reasonably well (though the single power-law is, statistically, a worse fit than either of the broken power-law models). On the right-hand panel (c) we show the "Maximal" model compared to the luminosity evolution one. Deeper surveys, with AGN identifications at $z>1$, are needed to determine the faint end behaviour and evolution, though Juneau et al. (2013) show that the faint end of the luminosity function seems to have evolved little since $z\approx 1$. The evolution at $z>2$ is relatively poorly constrained, but, as described in Section 2.6, there is evidence for a turnover in number densities of high luminosity AGN at $z\approx 2.8$. AGN at the break luminosity contribute most of the luminosity density at a given redshift. Our finding that the break luminosity is a strong function of redshift, decreasing with cosmic time at $z<2.8$ is consistent with the "downsizing" scenario of galaxy evolution (Cowie et al. 1996), in the sense that, for a given mean Eddington rate, the characteristic mass of the black holes contributing the bulk of the AGN luminosity density is decreasing with cosmic time.

When the optically-classified "non-AGN" and blank spectra with randomly-assigned redshifts are added into the survey to produce the Maximal luminosity function, as shown in the right-hand panel of Figure 2 as the dot-dashed line, we see an increase in the magnitude of the faint-end slope. At high luminosities, above the break, however, we see little more than an increase in the normalization (as expected). The larger numbers of low-$z$ AGN is consistent with the population of optically-classified "non-AGN" being in fact dominated by low luminosity, heavily obscured AGN. These conclusions must remain speculative, however, as it is very difficult to rule out the possibility that the some of the "non-AGN" are interlopers.

## Comparison with other AGN luminosity functions

In Figure , we plot a comparison of our mid-infrared AGN luminosity function with the hard X-ray luminosity functions from LaFranca et al. (2005) and Aird et al. (2010), the mid-infrared luminosity function from Matute et al. (2006; based on surveys using the *Infrared Space Observatory*) at $z<0.8$, and the estimate of the bolometric luminosity function of H07. In Figure we plot the normal (blue) type-1 quasar luminosity function from our survey, and compare to both the combined Sloan Digital Sky Survey (SDSS) and 2-degree Field (2dF) quasar surveys luminosity function (Croom et al. 2009), and the mid-infrared luminosity for type-1 AGN from Brown et al. (2006) at $z>0.8$.

For both simplicity and ease of comparison, a single bolometric correction, independent of redshift and luminosity, was assumed for each population (Table 2). For our mid-infrared AGN, we assume a uniform bolometric correction of 10. This number was derived as follows. We began with the bolometric correction from 5$\mu$m to the total type-1 quasar emission, $C_{\nu}^{'}= 8.0$ from Richards et al. (2006). We then corrected $C_{\nu}^{'}$ for unobscured type-1s to allow for the fact that the mid-infrared emission is likely to be much more isotropic than the optical/UV, and have its origin in reprocessed UV light originally emitted away from our line of sight. We thus derive a "true" bolometric correction for type-1 objects, $C_{\nu 1} \approx 0.6 C_{\nu}^{'}=5$, where 0.6 is the typical ratio of the UV/optical to the total quasar emission. Considering the type-2s on the other hand, the lower mid-infrared luminosities of type-2 objects at a given radio luminosity (Paper I, section 6) suggests that type-2 objects have higher mid-infrared extinctions than the type-1s. (This is also manifest as a steeper 8-24$\mu$m spectral index for the type-2s, $\approx 1.4$ compared to $\approx 1.0$ for the type-1s.) Residual extinction at mid-infrared wavelengths results in an underestimate of the mid-infrared luminosity for the type-2s by a factor $\approx 3$. Thus the effective value of the bolometric correction for the type-2 population will be $C_{\nu 2}\approx 3 C_{\nu 1}=15$. For simplicity, we average these two corrections to obtain a value of $C_{\nu}=10$. However, we will return to this issue in more detail in future work, where we will undertake a detailed comparison of type-1 and type-2 AGN SEDs.

At low redshifts ($z<0.3$), we expect to be incomplete due to contaminating emission from PAHs of up to $\approx 50$% (Paper I, section 2.2), and indeed we find that at $\nu L_{5\mu m}\sim 10^{44-45}$ we are below most other AGN luminosity functions in our $0.05<z<0.4$ bin, including the mid-infrared one of Matute et al. (2006). At $0.4<z<0.9$, however, the luminosity function matches those in other wavebands much better, and we see a small excess over the X-ray luminosity function at low luminosities ($\stackrel{<}{_{\sim}} 10^{44} {\rm ergs^{-1}}$), and a very large one over the optical quasar luminosity function. Considering the blue type-1 objects in our sample alone, however, Figure shows that we agree very well with prior estimates of the normal type-1 luminosity function at all redshifts.

As is already well-known (Hasinger 2008; Aird et al. 2010), the hard X-ray luminosity function peaks at much lower redshift than either the mid-infrared or the optical quasar luminosity function. This difference between the X-ray and other luminosity functions has often been ascribed to a luminosity-dependent bolometric correction (e.g. Steffen et al. 2006; H07, Han et al. 2012). Any redshift-dependent contribution due to absorption effects has been thought to be low, though it has long been known that X-ray absorbers are commonly seen in individual high redshift quasars (Elvis et al. 1994b). The partial correlation analysis of Steffen et al. (2006) in particular seems to indicate a stronger luminosity than redshift dependence on the ratio between optical and X-ray fluxes, $\alpha_{OX}$ at a given optical or X-ray luminosity (though for X-ray luminosities the result is only significant at 3$\sigma$), which would translate into a difference in the bright-end slope of the luminosity function. However, we find no evidence that a strong luminosity-dependent bolometric correction is required based on our comparison of luminosity functions, and we suspect that the apparently high (13.6$\sigma$) significance of the anticorrelation $\alpha_{OX}$ on *optical* luminosity is a by-product of luminosity-dependent obscuration effects, i.e. AGN with lower luminosities tend to have higher gas columns in the X-ray, increasing the magnitude of $\alpha_{OX}$. In Figure 3, we see that, at $0.4<z<1.6$, the mid-infrared, optical quasar, and hard X-ray luminosity functions are very similar at the high luminosity ($\nu L_{5\mu m} >10^{45} {\rm ergs^{-1}}$) end of the luminosity function. Instead of a difference in the slope of the AGN luminosity functions between infrared and X-ray, which is what we would expect if there was a strong luminosity dependence to the bolometric correction, what we see is a fall-off in the $z>1.6$ hard X-ray AGN population at all luminosities (particularly when the Aird et al. luminosity function is considered, though this luminosity function is thought to be somewhat low at $z>3$ (Kalfountzou et al. 2014)). This suggests that the difference in the luminosity functions is predominantly a redshift effect. The lack of a luminosity-dependent bolometric correction in the X-ray is particularly illustrated when the bolometric luminosity function of H07 (which assumes a luminosity-dependent bolometric correction) is plotted in Figure 3, when we see a significant overprediction of the space density of $\nu L_{5\mu m} >10^{45.5}{\rm ergs^{-1}}$ AGN at $z\stackrel{<}{_{\sim}}2$.

| Wavelength                        | Correction | Reference              |
|:----------------------------------|:----------:|:-----------------------|
| Mid-infrared (5$\mu$m)            |    10.0    | This paper             |
| Hard X-ray (2-10keV)              |    40.0    | Elvis et al. (1994a)   |
| Optical/UV (1670${\rm \AA}$ rest) |    4.5     | Richards et al. (2006) |

Assumed bolometric corrections

## Evolution by type of AGN

Figure shows the evolution of the mid-infrared AGN population split up by type. The top two panels, (a) and (b), show the luminosity functions at mean redshifts of 0.65 and 1.25, respectively, for the different types of AGN, with the plotted points showing the binned estimates. The black dotted line indicates the fit for all confirmed AGN, with the cyan line that for the Maximal model. Compared to the type-1 luminosity functions (shown in blue and red, for normal and red type-1s, respectively), the type-2 luminosity function (green) has a much steeper faint-end slope. In the lower two panels, (c) and (d), the number density of AGN above luminosities of $\nu L_{5 \mu m}=10^{45}$ and $10^{46} {\rm ergs^{-1}}$, respectively, are plotted against redshift. Up to $z\approx 2$ all the populations shown (all AGN, blue and red type-1s, type-2s and the combination of red type-1s and type-2s, representing the obscured population) show a similar evolution. The blue type-1 population shows the same evolution as optically-selected quasars, with a peak in number density at $z\approx 2.8$, but the obscured quasar population (magenta line in panel (d)) shows a peak in number density that is broader and shifted to earlier epochs ($z>3$). This is consistent with some theoretical predictions (Fanidakis et al. 2012), results from studies of the radio galaxy population (Jarvis et al. 2001), and a study of the infrared luminosity function of quasars from the Sloan Digital Sky Survey (Vardanyan, Weedman & Sargsyan 2014). Our models are fairly poorly constrained beyond the peak, however, with only 13 objects of any type at $z>2.8$, so the reality of the difference at high redshifts may be affected by small number statistics. Nevertheless, we have performed some tests to investigate whether the difference in the evolution we see might be statistically significant. We take for a null hypothesis that the evolution of obscured AGN (type-2 and red type-1s) and the unobscured (type-1) population is the same. We then compare the $S$ values for fits which fix the evolution of the obscured population to that of the unobscured one, and vice-versa. Fixing the values of the evolution parameters $k_1,k_2$ and $k_3$ for the obscured population to those fit for the unobscured population, and allowing the remainder of the parameters to be fit results in a difference in the $S$ values of 17, which, with the three degrees of freedom from fixing the $k_1-k_3$ parameter values allows us to eliminate the null hypothesis that the evolution is the same at $>99.9$% confidence. Forcing the evolution terms for the unobscured population to be the same as those for the obscured one results in a less significant $\Delta S =9.5$, $\approx 97$% confidence. We also compared to the evolution of all AGN. As expected, fixing the evolution of the obscured AGN to match that of all AGN resulted in no significant difference, but the difference from forcing the type-1 evolution to match that of all AGN resulted in ruling out the null hypothesis at the 95% confidence level.

In summary, we find that the evolution of the obscured and unobscured AGN populations are significantly different, at $>99.9$% confidence using our strongest statistical test. The most obvious difference in the luminosity function evolution is at high redshifts, where the number density of high luminosity obscured quasars appears to peak at a higher redshift than that of the unobscured population. Small number statistics, however, mean this interpretation should be treated with caution.

## The obscured fraction as a function of luminosity and redshift

Figure shows histograms of number of objects as a function of redshift and luminosity, split up by type. Figure plots the fraction of obscured objects as a function of both luminosity and redshift with the measured points plotted with error bars and the luminosity function models plotted as dashed lines. These plots show that the fraction of obscured objects varies from $\approx 90$% for Seyfert galaxies to $\approx 50$% at the highest quasar luminosities. In addition, there is evidence that the nature of the obscuration in the survey seems to change as a function of redshift and/or luminosity. At low luminosities and redshifts, most of the obscured objects are highly obscured, type-2, objects, but at high luminosities and redshifts the obscured population is dominated by less obscured red type-1 objects. The right-hand panel suggests that, despite the change in the nature of the obscuration, the overall obscured fraction is not a strong function of redshift when comparing bins at $z<0.8$ and $z>0.8$, and indeed, as discussed in Section 2.6 below, it is not until $z>2$ that we see a possible difference. Of course selection effects may well be playing a role here as heavily obscured objects become progressively harder to identify at high redshifts even in samples selected in the mid-infrared. As we discuss in Paper I, however, we believe such selection effects are not having a severe effect on our sample, even at $z>3$.

## The AGN luminosity density and the black hole mass density

Figure shows the AGN bolometric luminosity density and corresponding accreted black hole mass density since $z=5$. Panel (a) shows the luminosity density as a function of redshift, with dusty type-1 objects dominating at high redshifts, normal type-1s at $z\sim 2$ and type-2 objects at $z<1$. As our best fit luminosity evolution model has a steep faint-end slope, we need to assume a low luminosity cutoff at high luminosities in order to prevent an excessive amount of AGN luminosity density from being estimated. We assume, as a fiducial model, that the fitted form of the luminosity function we have adopted can be extrapolated a factor of three below the current limit of the survey. This results in luminosity densities in line with those we obtain from our alternative fixed faint-end slope model. With this assumption, we find more AGN than previous surveys, and integrating to the present day (e.g. Martínez-Sansigre & Taylor 2009) indicates a total energy, $U$, emitted by all AGN over the history of the Universe of ${\rm log_{10}} (U/{\rm erg\; Mpc^{-3})} =
37.5 \pm 0.2$, where the lower bound comes from cutting off the faint-end of the luminosity function at the luminosity corresponding to the flux limit of the deepest sample in our survey (0.61mJy at 24$\mu$m), and the upper bound from extrapolating the luminosity function to ten times fainter than this limit. To obtain the observed energy density today due to AGN, $u$, we need to divide the integrand by $(1+z)$ (Soltan 1982), resulting in a value of $u=2.8\pm 1.2\times 10^{-15} {\rm erg s^{-1} cm^{-3}}$, or a radiation intensity $b=uc/4\pi=6.7\pm 2.9 {\rm nW sr^{-1}}$, within the range of $b=3.5-8 {\rm nW sr^{-1}}$ estimated from the X-ray background, and suggesting that $\approx 12$% of the total radiation intensity of the Universe is contributed by AGN (Elvis et al. 2002). We therefore need a radiative efficiency of accretion, $\eta$ at the high end of previous estimates to match the local mass density in black holes ($\eta \approx 0.15$, Elvis et al. (2002); $\eta \approx 0.06$, Martínez-Sansigre & Taylor 2009) (Figure 8 (b), (c)). We use the compendium of local black hole mass density estimates of Graham & Driver (2007), with a mean of $10^{5.7}M_{\odot} {\rm Mpc^{-3}}$ to estimate a best fit value of $\eta=0.18^{+0.12}_{-0.07}$.

As a function of cosmic time, the evolution of the black hole mass density shows a very rapid rise with at $z>3$, then continues to rise, but at a smaller rate. Fifty percent of the current mass density has accreted by $z\approx 1.8$. At $z>3$, we predict that most black hole mass accretion occurs in dusty type-1 AGN, this switches to normal type-1 quasars at $z\sim 2$, and to type-2 AGN at $z<1$ when powerful quasars become very rare.

# Discussion

We present evidence that, when largely free from the effects of dust obscuration, the AGN luminosity function for actively-accreting black holes seems to be well represented by a double power-law with a break that is a strong function of luminosity, though the form of the faint end remains unconstrained at $z\stackrel{>}{_{\sim}}1$. Downsizing, in the sense that the characteristic AGN luminosity at which most of the AGN luminosity density is contributed is a strongly increasing function of redshift, is clearly present at $z<2$. Luminosity functions are often difficult to interpret in terms of physical models; however, at low luminosities, we know that most local AGN are not triggered by major merger events (e.g. Kocevski et al. 2012). At high luminosities, especially in the obscured population, major mergers may indeed play the dominant role (e.g. Urrutia et al. 2008; Treister et al. 2012; Hopkins, Kocevski & Bundy 2013). We therefore speculate that the increase in the break luminosity with redshift is driven by the increased frequency of major mergers of gas-rich galaxies at high redshifts.

A comparison of the mid-infrared luminosity function with those from other wavelengths indicates that we are seeing many more dust-obscured AGN than the normal type-1 AGN population, and more than the hard X-ray population, particularly at high redshifts. We find that the evolution of the obscured and the unobscured AGN population are significantly different, and we see tentative evidence for an earlier peak in the space density of obscured quasars, consistent with that seen in the radio-loud population. Our study then is consistent with the evolutionary explanation for obscured AGN, whereby AGN begin heavily obscured and outflows clear out material from the line of sight to the AGN.

Our new luminosity function has allowed us to improve the constraints on the luminosity density produced by AGN, and on the integral of this over cosmic time. Our overall AGN luminosity density agrees well with the range predicted from the X-ray background (Elvis et al. 2002), suggesting that we have indeed accounted for the vast majority of AGN in the Universe, and that about 12% of the luminosity density of the Universe is contributed by AGN. When compared to the mass density in compact central objects in galaxies today, this luminosity density allows an estimate of the mean radiative efficiency of accretion onto them. This number constitutes one of the best constraints we have on the nature of the compact central objects in galaxies. Although it is usually assumed that these are black holes, there are some theoretical alternatives, such as gravastars, that can deliver comparable radiative efficiencies (Harko, Kovács & Lobo 2009). However, it seems likely that the radiative efficiencies of such objects are $\eta <0.15$ (Bambi 2012), comparable to the current limit from the X-ray background (Elvis et al. 2002), but below our best estimate of $\eta=0.18$. Significant systematic uncertainties remain, however. AGN missing from our survey, because they are not confirmable in other wavebands (such as the "non-AGN" objects in this paper) or are missing from the mid-infrared selection criteria (such as low luminosity AGN) will only serve to increase our estimate of the radiative efficiency. However, there remain uncertainties on the local black hole mass density, the appropriate bolometric corrections to apply to both obscured and unobscured objects, and the low luminosity end of the luminosity function at high redshifts that could result in either an increase or a lowering of our estimate for $\eta$. The bolometric correction uncertainty can be addressed by using a full multiwavelength dataset, including archival Herschel data, to obtain accurate template SEDs for the obscured AGN population. Improved depth can be obtained by using more sophisticated AGN selection techniques based on the complete infrared SED to exploit the full depth of SWIRE to select AGN up to four times fainter than the current limit, and photometric redshifts based on new deep near-infrared surveys in the SWIRE fields (SERVS, Mauduit et al. 2012; VIDEO, Jarvis et al. 2013, and UKIDSS DXS, Lawrence et al. 2007) to obtain redshifts for these AGN, whose emission lines can be too faint to detect even using 8m class telescopes.

We thank the referee whose comments significantly improved the paper. The National Radio Astronomy Observatory is a facility of the National Science Foundation operated under cooperative agreement by Associated Universities, Inc.

# Optical through mid-infrared SED fitting

In order to disentangle the contribution of host galaxy light from our AGN fluxes, we performed SED fitting on the photometry described in Section 2.1, using the modeling techniques of Sajina et al. (2006, 2012, see also Lacy et al. 2007b and Hiner et al. 2011). We fit the SEDs at wavelengths from optical through 24$\mu$m using the photometry described in Section 2.1. For unobscured and red type-1s we fit a pure quasar template from Richards et al. (2006) (with an SMC reddening law applied in the case of the dusty red quasars, and a host galaxy stellar population added in a few low luminosity objects). These appear to work fairly well, apart from the 1-5$\mu$m region where the AGN SED is known to vary as a function of luminosity (e.g. Gallagher et al. 2007). For type-2s, we used the mid-infrared AGN template of Sajina et al. (2012), and a host galaxy stellar population, modeled as a single stellar population from Bruzual & Charlot (2003) with an age of 0.1,0.64, 1.4 or 5 Gyr picked to best fit the optical SED. 0.64 Gyr was used as the default in cases where the optical/near-IR data were too sparse to distinguish between models (the vast majority of cases), however, 21 were fit with with a 1.4Gyr population, 23 with a 100Myr population, and ten with a 5Gyr population. All the stellar population models had similar behavior in the near-infrared region of the spectrum, however, so the exact choice of stellar age does not affect the AGN flux density at rest-frame 5$\mu$m by more than a few percent. Examples of our fits are shown in Figure 9. Full SED fits through the far-infrared will be discussed in detail in a future paper.

The SED fitting highlights some ways in which our sample may be subject to subtle selection effects due to the prescence of mid-infrared emission from the host galaxy, particularly at low luminosities and redshifts. Our use of flux limits at 24$\mu$m ensures that photospheric emission from stars contibutes negligibly ($\stackrel{<}{_{\sim}}1$% ) to the flux density at the selection frequency. However, there is the potential for some objects at low redshifts to have their 24$\mu$m flux densities boosted by emission from warm dust due to star formation. Although this will not contribute significantly to the AGN flux density estimated at 5$\mu$m, it may lead to objects being included in the sample that would otherwise have fallen below the flux density limit at 24$\mu$m. However, these same objects are, to a greater extent, selected against in the survey as their 7.7$\mu$m PAH features will lift them out of the AGN selection region at $z<0.4$ (Section 2.5). As the slope of the warm dust SED is very steep at wavelengths between $\sim 10\mu$m and 24$\mu$m, the contribution of warm dust emission to the observed 24$\mu$m flux density at higher redshifts is likely to be less than a few percent.

[^1]: http://scipy.org