abstract: Super-resolution methods form high-resolution images from low-resolution images. In this paper, we develop a new Bayesian nonparametric model for super-resolution. Our method uses a beta-Bernoulli process to learn a set of recurring visual patterns, called dictionary elements, from the data. Because it is nonparametric, the number of elements found is also determined from the data. We test the results on both benchmark and natural images, comparing with several other models from the research literature. We perform large-scale human evaluation experiments to assess the visual quality of the results. In a first implementation, we use Gibbs sampling to approximate the posterior. However, this algorithm is not feasible for large-scale data. To circumvent this, we then develop an online variational Bayes (VB) algorithm. This algorithm finds high quality dictionaries in a fraction of the time needed by the Gibbs sampler.
author: Gungor Polatkan; Mingyuan Zhou; Lawrence Carin; David Blei; Ingrid Daubechies
bibliography: refs2.bib
title: A Bayesian Nonparametric Approach to Image Super-resolution

# Introduction

The sparse representation of signals with a basis is important in many applications. It has been extensively used in image denoising, inpainting, super-resolution, classification and compressive sensing .

Many real data sets can be sparsely represented in some basis; typically this basis itself has to be learned from the data . For example, an image can be represented by weighted combinations of recurrent patterns of pixels. This construction may be beneficial, both while building a model for more accurate representation of the data (e.g. superior image denoising models) and while deriving and implementing an inference procedure for more efficient algorithms.

In this paper we consider image super-resolution (SR), the problem of recovering a high-resolution (HR) image from a low-resolution (LR) image. It has many applications, e.g., to smart phones, surveillance cameras, medical imaging, and satellite imaging.

There are a variety of approaches for image super-resolution. In general, rendering an HR image from an LR image has many possible solutions. We must use regularization of some form, i.e., prior information about the HR, to guarantee uniqueness and stability of the extension. For this purpose, researchers have proposed several methods . *Interpolation-based methods*, such as the Bicubic method and Bilinear method, often over-smooth images, losing detail. *Example-based approaches* use machine learning to avoid this ; they train on ground-truth HR and LR images, learning a statistical relationship between the two. These relationships are later used to reconstruct unknown HR images from corresponding LR images. Freeman et al. () proposes a method that stores a training set of preprocessed patches and uses a nearest-neighbor search to super-resolve. Kim et al. () proposes using kernel ridge regression with a regularized gradient descent. Another class of SR algorithms use texture similarity to match image regions with known textures . Finally, there are methods for single-image super-resolution. One classic example is which uses recurring patterns at same and different scales in a single image.

In this work, our focus is on SR via example-based sparse coding. ScSR (Super-resolution via Sparse Representation) is such an algorithm pioneered in . This algorithm is based on sparse coding via L1 regularized optimization. In , image data are represented using a collection of dictionary elements (recurring patterns of pixels) that are weighted across different positions. Although very powerful, this model requires one to specify the number of dictionary elements and the variance of the noise model in advance—parameters that may be difficult to assess for real-world images. It also only provides a batch learning algorithm, i.e., computing model parameters via a gradient descent algorithm on a fixed small subset of the data.

Bayesian nonparametric methods circumvent all these limitations. These methods adapt the structure of the latent space to the data and provide a powerful representation because they infer parameters that otherwise have to be assigned *a priori* . The full posterior distribution can be approximated via MCMC or variational inference, yielding sparse representations and learned dictionaries.

Bayesian nonparametric methods have been used in many image analysis applications: to learn deep architectures used for object recognition in , for image inpainting and denoising in , for image segmentation in , and to learn nonparametric multiscale representations of images in .

In this paper, we develop a Bayesian nonparametric method for super-resolution. We show that inference in our model is feasible, performing super-resolution with both a sampling based algorithm and an online variational inference algorithm. In the latter, we approximate the posterior distributions via a stochastic gradient descent over a variational objective that enables us to use the full data set and process the data segment by segment. We also provide human evaluation experiments which shows that signal-to-noise ratio (a typical quantitative measure of success in image analysis applications) is not necessarily consistent with human judgement. We devise a new model, new algorithms, and study a human-based evaluation. We make the following contributions:

-   We develop a sparse Bayesian nonparametric model for SR, learning the number of dictionary elements and the noise variance from the data.

-   We develop an online variational Bayes (VB) algorithm finding high quality "coupled dictionaries" in a fraction of the time needed by traditional inference.

-   We devise large scale human evaluation experiments to explicitly assess the visual quality of results.

Our approach to SR gives a rich nonparametric representation with scalable learning.

The remainder of the paper is organized as follows: Section describes the proposed super-resolution model and non-parametric prior, Section contains the derivation of the posterior inference algorithms, Section presents the experimental results and implementation details, Section includes the discussion and future work.

# Proposed Approach

Bayesian factor analysis can be used to learn factors / dictionaries from natural images. Zhou et al. () used beta process factor analysis in image denoising, inpainting and compressive sensing. These models learn both the dictionary elements and their number from the data.

We build here a nonparametric factor analysis model that couples an HR image to a corresponding LR image. In training, we learn the HR/LR relationship from observed HR/LR pairs. To perform super-resolution, we condition on an observed LR image and compute the conditional expectation of its corresponding HR image. A more detailed description of the training process is as follows. We create training data by taking observed HR images and forming corresponding LR images. Figure depicts the preprocessing and data extraction steps. We first down-sample the HR images. Then, we up-sample those by interpolating with a deterministic weighting function (e.g. bicubic interpolation). We extract same-sized patches from the same locations of both the HR and interpolated LR images, and consider those patches as coupled to each other. These are the data on which we train the model.

In the model, each small patch is generated from latent global dictionary elements—small images functioning as factor loadings—using local sparse weights and Gaussian noise. We will first explain how these latent variables are generated and then present how they are used to generate the observations.

We learn two dictionaries: one for high resolution images and one for low resolution images. In terms of notation, $\textbf{d}_{k}^{(l)}$ represents the LR dictionary element, and $\textbf{d}_{k}^{(h)}$ is the HR dictionary element. $P^{(l)}$ and $P^{(h)}$ represent the dimensionality of the low and high resolution dictionary elements, respectively. To model each dictionary element, we use a zero-mean Gaussian distribution, $$\begin{aligned}
\textbf{d}_{k}^{(l)} \mbox{ } &\thicksim  \mbox{ }  \mathcal{N}(0,P^{(l)^{-1}}\textbf{I}_{P^{(l)}}) & \textbf{d}_{k}^{(h)} \mbox{ } &\thicksim  \mbox{ }  \mathcal{N}(0,P^{(h)^{-1}}\textbf{I}_{P^{(h)}}). \nonumber
\end{aligned}$$ The matrix form of the dictionaries are $\textbf{D}^{(l)}$ and $\textbf{D}^{(h)}$ where $k$th columns of those matrices are $\textbf{d}_{k}^{(l)}$ and $\textbf{d}_{k}^{(h)}$, respectively.

Following , we assume that the sparse weights are shared by both resolution levels for combining dictionary elements to produce images. This is the key property of the model that allows us to frame super-resolution as inference. Sparse weights have two components: real valued weights $s_{ik}$ and binary valued assignments $z_{ik}$. To model the weights $s_{ik}$, we use a zero-mean Gaussian distribution with precision $\alpha$. $\textbf{z}_i$ is a binary vector that encodes which dictionary elements are activated for the corresponding observation. $p(\textbf{z})$ represents the prior of this variable and we will elaborate on this in next section. These are given as $$\begin{aligned}
s_{ik} \mbox{ } &\thicksim  \mbox{ }  \mathcal{N}(0,1/\alpha)& z_{ik} \mbox{ } &\thicksim  \mbox{ } p(z_{ik}).\nonumber
\end{aligned}$$

We place Gamma priors on the precisions of the sparse weights and observation noise ($\alpha$ and $\gamma$). The two resolution levels share these variables as well, $$\begin{aligned}
\gamma \mbox{ } &\thicksim  \mbox{ } \mathrm{Gamma}(c,d),        &  \alpha &\sim \textrm{Gamma}(e,f) \nonumber.
\end{aligned}$$

Let $\textbf{x}_i^{(h)}$ and $\textbf{x}_i^{(l)}$ represents patches extracted from HR and LR images, respectively, as shown in Figure . Given the (global) dictionary elements and (local) sparse weights, the observations are modeled as $$\begin{aligned}
\boldsymbol{\epsilon}_{i}^{(l)}  &\thicksim    \mathcal{N}(\textbf{0}, \gamma^{-1}\textbf{I}_{P^{(l)}}) &\boldsymbol{\epsilon}_{i}^{(h)}  &\thicksim    \mathcal{N}(\textbf{0}, \gamma^{-1}\textbf{I}_{P^{(h)}}) \\
\textit{\textbf{x}}_i^{(l)}  &=  \textbf{D}^{(l)}(\textbf{s}_i\odot \textbf{z}_i) +  \boldsymbol{\epsilon_i}^{(l)} &\textit{\textbf{x}}_i^{(h)}  &=  \textbf{D}^{(h)}(\textbf{s}_i\odot \textbf{z}_i) +  \boldsymbol{\epsilon_i}^{(h)}\nonumber
\end{aligned}$$ where $\{(l), (h)\}$ represents LR and HR, respectively. Here, $N$ is the total number of patches, and $\odot$ represents the element-wise multiplication of two vectors. Figure illustrates the graphical model.

To use this model in SR, we must be able to compute the posterior distributions of the hidden variables. In the training phase, we must compute the posterior distributions $p(\textbf{D}^{(h)}, \textbf{D}^{(l)} | \{\textbf{x}_i^{(h)}, \textbf{x}_i^{(l)}\})$ of the dictionaries, given a collection of HR/LR image pairs. In testing, we use their posterior expectation to reconstruct a held-out HR image from an LR image, $$\begin{aligned}
\mathbb{E} [\textbf{x}_j^{(h)} | \textbf{x}_j^{(l)}, \{\textbf{x}_i^{(h)}, \textbf{x}_i^{(l)}\} ]  \approx \hat{\textbf{D}}^{(h)}(\hat{\textbf{s}}_j\odot \hat{\textbf{z}}_j) \label{construct}
\end{aligned}$$ where $\hat{\textbf{D}}^{(h)}$ is the mean of the posterior distribution $p(\textbf{D}^{(h)} | \{\textbf{x}_i^{(h)}, \textbf{x}_i^{(l)}\})$ and $(\hat{\textbf{s}}_j\odot \hat{\textbf{z}}_j)$ are the posterior expectation of the sparse weights from the LR image patches ($\textit{\textbf{x}}_j^{(l)}$) via posterior inference. (We discuss algorithms for posterior inference in Section .)

## Beta-Bernoulli Process Prior (BP)

We now discuss the prior for the factor assignments $\textbf{z}_i$. We use a beta-Bernoulli process (BP) , a prior on infinite binary matrices which is connected to the Indian buffet process (IBP). Each row encodes which dictionary elements are activated for the corresponding observation; columns with at least one active cell correspond to factors. The distinguishing characteristic of this prior is that the number of these factors is not specified a priori. Conditioned on the data, we examine the posterior distribution of the binary matrix to obtain a data-dependent distribution of how many components are needed.

The IBP metaphor gives the intuition. Consider a buffet of dishes at a restaurant. Suppose there are infinite number of dishes and we are trying to specify the infinite binary matrix indicating which customers (observations) choose which dishes (factors/dictionary elements). In the Indian buffet process (IBP), $N$ customers enter the restaurant sequentially. Each customer chooses dishes in a line from a buffet. The first customer starts from the beginning of the buffet and takes from each dish, stopping after Poisson($\tau$) number of dishes. The $i$th customer starts from the beginning as well, but decides to take from dishes in proportion to their popularity within the previous $i-1$ customers. This proportionality can be quantified as $\frac{m_k}{i}$ where $m_k$ is the number of previous customers who took this $k$th dish. After considering the dishes previously taken by other customers, the $i$th customer tries a Poisson($\frac{\tau}{i}$) number of new dishes. Which customers chose which dishes is recorded by the infinite binary matrix with $N$ rows (indicating the customers/observations) and infinite columns (indicating the dishes/factors/dictionary elements). One important (and surprising) property of this process is that the joint probability of final assignment is independent of the order of customers getting into the restaurant which is called exchangeability property of the prior .

The probabilistic construction is as follows. Each observation $i$ is drawn from a Bernoulli process (a sequence of independent identically distributed Bernoulli trials), $\textbf{x}_i \sim \mbox{BeP}(B)$ where $B$ is drawn from a beta process $B \sim \mbox{BP}(c_0, B_0)$. $B_0$ represents the base measure with $B_0={\cal N}(0, 1/\beta \textbf{I})$. As $K \rightarrow \infty$, the $i$th observation is $\textbf{x}_i = \sum_{k=1}^{\infty}z_{ik}\delta_{d_{k}}$ where $z_{ik}$ denotes whether the dictionary element $\textbf{d}_{k}$ is used while representing the $i$th observation or not, and the sample from the beta process is given by $B=\sum_{k=1}^{\infty}\pi_k\delta_{d_{k}}$. Here, $\pi_k$ represents the usage probability of dictionary element $\textbf{d}_k$.

In inference, we use a finite beta-Bernoulli approximation . The finite model truncates the number of dictionary elements to $K$ and is given by $$\begin{aligned}
\pi_k \sim \mathrm{Beta}(c_0\eta_0, c_0(1-\eta_0)), \mbox{ }z_{ik} \sim \textrm{Bernoulli}(\pi_k)
\end{aligned}$$ where $c_0$ and $\eta_0$ are scalars and $\quad k \in 1, \ldots K$. As $K$ tends to infinity, the finite beta-Bernoulli approximation approaches the IBP/BP. If the truncation is large enough, data analyzed with this prior will exhibit fewer than $K$ components .

## Super-resolution via Posterior Distributions

Our algorithm has 2 stages: fitting the model on pairs of HR and LR images, and super-resolving new LR images to create HR versions.

**Training: Coupled Dictionary Learning Stage.** In training, we observe $\textbf{x}_i^{(h)}$ and $\textbf{x}_i^{(l)}$. All other random variables are latent. The key inference problem to be solved is the computation of the posterior distributions of the hidden variables. In the training phase, we must compute the posterior distributions $p(\textbf{D}^{(h)}, \textbf{D}^{(l)} | \{\textbf{x}_i^{(h)}, \textbf{x}_i^{(l)}\})$ of the dictionaries given a collection of HR/LR image pairs. We rewrite the coupled model in a form similar to the single scale model: $$\begin{aligned}
  \textit{\textbf{x}}_i^{(c)} =
 \!\left(\!
    \begin{array}{c}
       \textit{\textbf{x}}_i^{(l)} \\
      \textit{\textbf{x}}_i^{(h)}
    \end{array}
  \!\right)\!,
  \textbf{d}_{k}^{(c)} =
 \! \left(\!
    \begin{array}{c}
       \textbf{d}_{k}^{(l)} \\
      \textbf{d}_{k}^{(h)}
    \end{array}
  \!\right)\!,
\boldsymbol{\epsilon}_{i}^{(c)} =
  \!\left(\!
    \begin{array}{c}
\boldsymbol{\epsilon}_{i}^{(l)} \\
\boldsymbol{\epsilon}_{i}^{(h)}
    \end{array}
  \!\right)\!
  \label{eq:coupling}
\end{aligned}$$ where the superscript $(c)$ corresponds to combination of $(l)$ and $(h)$. Writing the fully-observed model in this way reveals that we can train the dictionaries with similar methods as for the single-scale base model (Training amounts to approximating the posteriors of these values). The differences are that we use combined patches $\textit{\textbf{x}}_i^{(c)}$ and combined dictionaries $\textbf{d}_{k}^{(c)}$. This leads to shared sparse weights for the two resolution levels. (The details of how we compute the distribution $p(\textbf{D}^{(h)}, \textbf{D}^{(l)} | \{\textbf{x}_i^{(h)}, \textbf{x}_i^{(l)}\})$ are discussed in Section .)

**Super-resolving a Low Resolution Image.** With fitted dictionaries in hand, we now show how to form HR images from LR images via posterior computation.

In this prediction setting, the HR image $\textbf{x}_i^{(h)}$ is unknown; the goal is to reconstruct it from the LR image patches $\textbf{x}_i^{(l)}$, the posterior estimates of the dictionaries $(\hat{\textbf{D}}^{(h)}, \hat{\textbf{D}}^{(l)})$, and the precisions $\hat{\gamma}, \hat{\alpha}$ of the noise and the sparse weights, $$\begin{aligned}
  \textit{\textbf{x}}_i^{(c)} =
  \left(
    \begin{array}{c}
       \textit{\textbf{x}}_i^{(l)} \\
     -
    \end{array}
  \right),\mbox{ }
  \textbf{d}_{k}^{(c)} =
  \left(
    \begin{array}{c}
       \textbf{d}_{k}^{(l)} \\
      \textbf{d}_{k}^{(h)}
    \end{array}
  \right),\mbox{ }
\boldsymbol{\epsilon}_{i}^{(c)} =
  \left(
    \begin{array}{c}
\boldsymbol{\epsilon}_{i}^{(l)} \\
-
    \end{array}
  \right).
\end{aligned}$$ First we find estimates of the sparse factor scores, $(\hat{\textbf{s}}_i\odot \hat{\textbf{z}}_i)$, by using the LR image patches $\textit{\textbf{x}}_i^{(l)}$ and posterior estimates of the dictionaries and precisions $\gamma$ and $\alpha$. The fitted value of $\alpha$ determines the strength of a "regularization term" that controls the sparsity of the factor scores.

More precisely, this prediction setting has 3 steps. The input is a set of held-out LR image patches $\textbf{x}_i^{(l)}$, the posterior estimates of the dictionaries $(\hat{\textbf{D}}^{(h)}, \hat{\textbf{D}}^{(l)})$, and the precisions $\hat{\gamma}, \hat{\alpha}$ of the noise and the sparse weights. The steps are as follows:

1.  We find estimates of the sparse factor scores, $(\hat{\textbf{s}}_i\odot \hat{\textbf{z}}_i)$, conditioned on the LR image patches $\textit{\textbf{x}}_i^{(l)}$ and estimates $(\hat{\textbf{D}}^{(h)}, \hat{\textbf{D}}^{(l)}), \hat{\gamma}, \hat{\alpha}$ from the training stage.

2.  Eq. determines the HR patches $\hat{\textbf{x}}_i^{(h)}\!$.

3.  We replace each $\textbf{x}_i^{(l)}$ by its corresponding collocated $\hat{\textbf{x}}_i^{(h)}\!$; the whole HR image, $\hat{\textbf{X}}^{(h)}\!$, is the pixel-wise average of those overlapping reconstructions.

**Post-processing:** Following , we apply a post-processing step that, when down-sampled, the reconstructed HR image, $\hat{\textbf{X}}^{(h)}$, should match the given LR image $\textbf{X}^{(l)}$. Specifically, we solve the following optimization: $$\begin{aligned}
  \hat{\textbf{X}}^{(h)*} = \underset{\textbf{X}}{\operatorname{argmin}} || f(\textbf{X}) -  \textbf{X}^{(l)} ||_{2}^{2} + c|| f(\textbf{X}) -  \hat{\textbf{X}}^{(h)} ||_{2}^{2}
     \nonumber
\end{aligned}$$ where $f()$ is a linear operator consisting of an anti-aliasing filter followed by down-sampling. This optimization problem is solved with gradient descent.

# Posterior Inference

In the proposed approach, all of the priors are in the conjugate exponential family. In a first implementation, we use Gibbs sampling. We iteratively sample from the conditional distribution of each hidden variable given the others and the observations. This defines a Markov chain whose stationary distribution is the posterior . The corresponding sampling equations are analytic and provided in the appendix A-B (appendix is in the supplementary material).

The Gibbs sampler has difficulty with scaling to large data, because it must go through many iterations, each time visiting the entire data set before the sampler mixes. For this reason, both our Gibbs sampler and ScSR use $10^5$ patches sampled from $3\times10^6$. We now develop here an alternative algorithm to Gibbs sampling for SR that scales to large and streaming data. Specifically, we develop an online variational inference algorithm.

Variational inference is a deterministic alternative to MCMC that replaces sampling with optimization. The idea is to posit a parameterized family of distribution over the hidden variables and then optimize the parameters to minimize the KL divergence to the posterior of interest . Our algorithm iteratively tracks an approximate posterior distribution, which improves as more data are seen.

In typical applications, the variational objective is optimized with coordinate ascent, iteratively optimizing each parameter while holding the others fixed. However, in Bayesian settings, this suffers from the same problem as Gibbs sampling—the entire data set must be swept through multiple times in order to find a good approximate posterior. In the algorithm we present here, we replace coordinate ascent optimization with *stochastic optimization*—at each iteration, we subsample our data and then adjust the parameters according to a noisy estimate of the gradient. Because we only to subsample the data at each iteration, rather than analyze the whole data set, the resulting algorithm scales well to large data. This technique was pioneered in  and was recently exploited for online learning of topic models  and hierarchical Dirichlet processes .

We first develop the coordinate ascent algorithm for the coupled model. Then we derive the online variational inference algorithm, which can more easily handle large data sets.

## Variational Inference for the Coupled model

We use the coupling perspective in Section to derive the batch variational Bayes (VB) algorithm. The single-scale base model is the BPFA model of , which gives a mean-field variational inference algorithm. The batch VB algorithm derived here is the coupled version of that.

We first define a parametrized family of distributions over the hidden variables. Let $\textbf{Q} =
\{\boldsymbol{\pi}, \textbf{Z, S, D}, \gamma, \alpha \}$ denote the hidden variables for all $i, k$. We write coupled data as in Equation ; in the new set-up the variables to be learned become $\textbf{Q} =\{\boldsymbol{\pi}, \textbf{Z, S}, \textbf{D}^{(c)}, \gamma, \alpha \}$. We use a fully factorized variational distribution, $$q(\textbf{Q}) =
  q_{\tau}(\boldsymbol{\pi})q_{\phi}(\textbf{D}^{(c)})q_{\nu}(\textbf{Z})q_{\theta}(\textbf{S})q_{\lambda}(\boldsymbol{\gamma})q_{\epsilon}(\boldsymbol{\alpha}).$$ Each component of this distribution is governed by a free variational parameter, $$\begin{aligned}
q_{\tau_k}(\pi_k) &= \mathrm{Beta}( \tau_{k1}, \tau_{k2}) & q_{\nu_{ik}}(z_{ik}) &= \mathrm{Bernoulli}( \nu_{ik})\nonumber\\
q_{\phi_{kj}}(d_{kj}) &= \mathcal{N}(\phi_{kj}, \Phi_{kj}) & q_{\lambda}(\gamma) &= \mathrm{Gamma}(\lambda_{1}, \lambda_{2})\nonumber\\
 q_{\theta_{ik}}(s_{ik}) &= \mathcal{N}(\theta_{ik}, \Theta_{ik})&q_{\epsilon}(\alpha) &= \mathrm{Gamma}(\epsilon_{1}, \epsilon_{2}) \nonumber
\end{aligned}$$ We optimize these parameters with respect to a bound on the marginal probability of the observations. This bound is equivalent, up to a constant, to the negative KL divergence between $q$ and the true posterior. Thus maximizing the bound is equivalent to minimizing KL divergence to the true posterior. Let $\boldsymbol{\Xi}
= \{c_0,\eta_0,c,d,e,f\}$ be the hyper-parameters. The variational lower bound is $$\begin{aligned}
 & \log(p(\textbf{X}^{(c)}|\boldsymbol{\Xi})) \geq  H(q)+ \sum\nolimits_{k
    =1}^{K}\Big\{\mathbb{E}_{\textbf{q}}[\log p(\pi_k|c_0, \eta_0, K)] \nonumber\\
    &+
  \sum\nolimits_{i =
    1}^{N}\mathbb{E}_{\textbf{q}}[\log
  p(z_{ik}|\boldsymbol{\pi})]
 +\sum\nolimits_{j=1}^{J}\mathbb{E}_{\textbf{q}}[\log\big(p(d_{kj}|\beta_{kj})
  \big)] \nonumber \\
  &   + \sum\nolimits_{i = 1}^{N}\mathbb{E}_{\textbf{q}}[\log \big( p(s_{ik}|\alpha)  p(\alpha|e,f) \big)] \Big\} \label{elbo}\\
  &+\sum\nolimits_{i = 1}^{N}\Big\{\mathbb{E}_{\textbf{q}}[\log
  p(\textbf{x}_{i}^{(c)}|\textbf{Z,S,}\textbf{D}^{(c)}\gamma)]+\mathbb{E}_{\textbf{q}}[\log p(\gamma|c,d)]\Big\}, \nonumber
\end{aligned}$$ where $\textrm{H}(q)$ is the entropy of the variational distribution and dimensionality of the dictionary elements $J$ is twice as big as the single-scale model. We denote this function ${\cal L}(q)$.

Holding the other parameters fixed, we can optimize each variational parameter exactly; this gives an algorithm that goes uphill in ${\cal L}(q)$ . (Further, this will provide the algorithmic components needed for the online algorithm of Section .)

Update equations for each free parameter optimizing this bound are given below. In all equations, $\textbf{I}_P$ represents $P\times P$ identity matrix, and $\tilde{\textbf{x}}_{i(-k)}^{(c)}$ represents the reconstruction error using all but the $k$th dictionary element, that is $$\tilde{\textbf{x}}_{i(-k)}^{(c)} = \textbf{x}_{i}^{(c)} - \textbf{D}^{(c)}(\textbf{s}_i\odot \textbf{z}_i) + \textbf{d}_k^{(c)}(s_{ik}\odot z_{ik}).$$ The expectation based on the variational distribution is then given by $$\mathbb{E}_q[\tilde{\textbf{x}}_{i(-k)}^{(c)}] = \textbf{x}_{i}^{(c)}  + \boldsymbol{\phi}_{k}^{(c)}(\theta_{ik} \nu_{ik}) - \sum_{k=1}^{K}\boldsymbol{\phi}_{k}^{(c)}(\theta_{ik} \nu_{ik}).$$

**Update for the binary factor assignment $z_{ik}$**: The variational parameter for factor assignment $z_{ik}$ is $\nu_{ik}$. We first consider two values of the variational distribution for two values (0,1) of $z_{ik}$, $$\begin{aligned}
q(z_{ik}  = 1) \mbox{  } &\propto \mbox{  }  \exp( \mathbb{E}_q[\ln(\pi_k)]) \mbox{exp}\Big( -  \frac{  \frac{\lambda_1}{\lambda_2} \big( ( \theta_{ik}^2+\Theta_{ik})  (\boldsymbol{\phi}_{k}^{(c)^T} \boldsymbol{\phi}_{k}^{(c)} + \sum_{j}\Phi_{kj}) - 2\theta_{ik} \boldsymbol{\phi}_{k}^{(c)^T}\mathbb{E}_q[\tilde{\textbf{x}}_{i(-k)}^{(c)}]  \big)}{2}\Big)  \\
q(z_{ik}  = 0) \mbox{  } &\propto \mbox{  }  \exp(\mathbb{E}_q[ln(1-\pi_{k})]), \mbox{   }\mbox{where}
\end{aligned}$$ $$\begin{aligned}
 \mathbb{E}_q[\ln(\pi_k)]  \mbox{  } &= \mbox{  }   \psi\big(c_0\eta_0 + \sum_{i}\nu_{ik}\big) -  \psi(c_0 + N)\\
 \mathbb{E}_q[\ln(1-\pi_k)]  \mbox{  } &= \mbox{  }   \psi\big(c_0(1-\eta_0) - \sum_{i}\nu_{ik} + N\big) -  \psi(c_0 + N)

\end{aligned}$$ Then the update equation for the variational parameter $\nu_{ik}$ is given as $$\begin{aligned}
 \nu_{ik}   \mbox{  } &= \mbox{  }   \frac{q(z_{ik}  = 1|-)}{q(z_{ik}  = 1|-)+q(z_{ik}  =01|-)}
\end{aligned}$$

**Update for the shared sparse weight $s_{ik}$**: The variational distribution for the sparse weight $s_{ik}$ is Gaussian parametrized with mean $\theta_{ik}$ and variance $\Theta_{ik}$. Coordinate ascent update equation for these free variational parameters are $$\begin{aligned}
\Theta_{ik}  \mbox{ } &= \mbox{ } \Big( \frac{\epsilon_1}{\epsilon_2} + \frac{\lambda_1}{\lambda_2}\nu_{ik} (\boldsymbol{\phi}_{k}^{(c)^T} \boldsymbol{\phi}_{k}^{(c)} + \sum_{j}\Phi_{kj}) \Big)^{-1}\\
\theta_{ik}  \mbox{ } &= \mbox{ } \frac{\lambda_1}{\lambda_2} \Theta_{ik}\nu_{ik}  \boldsymbol{\phi}_{k}^{(c)^T}\mathbb{E}_q[\tilde{\textbf{x}}_{i(-k)}^{(c)}] .
\end{aligned}$$

**Update for the $k$th coupled dictionary element** $\textbf{d}_{k}^{(c)}$: The variational distribution for the couple dictionary element $\textbf{d}_{k}^{(c)}$ is Gaussian parametrized with mean $\boldsymbol{\phi}_{k}^{(c)}$ and variance $\boldsymbol{\Phi}_{k}^{(c)}$. Coordinate ascent update equation for these free variational parameters are

$$\begin{aligned}
\boldsymbol{\Phi}_{k}^{(c)}  \mbox{ } &= \mbox{ } \Big( 2P \textbf{I}_{2P}  + \frac{\lambda_1}{\lambda_2}\sum_{i=1}^{N}( \theta_{ik}^2+\Theta_{ik})  \nu_{ik} ^2\Big)^{-1}\\
\boldsymbol{\phi}_{k}^{(c)}  \mbox{ } &= \mbox{  }  \frac{\lambda_1}{\lambda_2}\boldsymbol{\Phi}_{k} ^{(c)}  \sum_{i=1}^{N}\theta_{ik}  \nu_{ik} \mathbb{E}_q[\tilde{\textbf{x}}_{i(-k)}^{(c)}].
\end{aligned}$$ The updates for high resolution (h) and low resolution (l) components can be given separately as $$\begin{aligned}
\boldsymbol{\Phi}_{k}^{(h)}  \mbox{ } &= \mbox{ }  \Big( 2P \textbf{I}_{P}  + \frac{\lambda_1}{\lambda_2}\sum_{i=1}^{N}( \theta_{ik}^2+\Theta_{ik})  \nu_{ik} ^2\Big)^{-1}
&\boldsymbol{\Phi}_{k}^{(l)}  \mbox{ } &= \mbox{ } \Big( 2P \textbf{I}_{P}  + \frac{\lambda_1}{\lambda_2}\sum_{i=1}^{N}( \theta_{ik}^2+\Theta_{ik})  \nu_{ik} ^2\Big)^{-1}\\
\boldsymbol{\phi}_{k}^{(h)}  \mbox{ } &= \mbox{  }  \frac{\lambda_1}{\lambda_2}\boldsymbol{\Phi}_{k}^{(h)}   \sum_{i=1}^{N}\theta_{ik}  \nu_{ik}  \tilde{\textbf{x}}_{i(-k)}^{(h)}
&\boldsymbol{\phi}_{k}^{(l)}  \mbox{ } &= \mbox{  }  \frac{\lambda_1}{\lambda_2}\boldsymbol{\Phi}_{k}^{(l)}   \sum_{i=1}^{N}\theta_{ik}  \nu_{ik}  \tilde{\textbf{x}}_{i(-k)}^{(l)}.
\end{aligned}$$

**Update for the dictionary usage probabilities $\pi_k$**: The variational distribution for the dictionary usage probabilities $\pi_k$ is a beta distribution parametrized with the shape parameters ($\tau_{k1}, \tau_{k2}$). Coordinate ascent update equation for these free variational parameters are $$\begin{aligned}
\tau_{k1}  \mbox{  } &= \mbox{ } c_0 \eta_0+  \sum_{i=1}^{N}\nu_{ik}  \\
\tau_{k2}   \mbox{  } &= \mbox{ }  \mbox{ } N-  \sum_{i=1}^{N}\nu_{ik} + c_0(1-\eta_0).
\end{aligned}$$

**Update for the precision $\gamma$**: The variational distribution for the precision $\gamma$ of the observation noise $\boldsymbol{\epsilon}_i$ is a gamma distribution parametrized with ($\lambda_1, \lambda_2$). Coordinate ascent update equation for these free variational parameters are $$\begin{aligned}
\lambda_1  \mbox{ } &= \mbox{  }  c + NP\\
\lambda_2  \mbox{ } &= \mbox{  }  d + \frac{1}{2}\sum_{i=1}^{N} \Big \{ ||\textbf{x}_{i}^{(c)}  - \sum_{k=1}^{K}\boldsymbol{\phi}_{k}^{(c)}(\theta_{ik} \nu_{ik}) ||_2^2 +\sum_{k=1}^{K}\nu_{ik}( \theta_{ik}^2+\Theta_{ik})  (\boldsymbol{\phi}_{k}^{(c)^T} \boldsymbol{\phi}_{k}^{(c)} + \sum_{j}\Phi_{kj})\\
\mbox{ } &- \mbox{  } \sum_{k=1}^{K} \nu_{ik}  \boldsymbol{\phi}_{k}^{(c)^T} \boldsymbol{\phi}_{k}^{(c)}   \theta_{ik}^2 \Big \}.
\end{aligned}$$

**Update for the precision $\alpha$**: The variational distribution for the precision $\alpha$ of the sparse weights $s_{ik}$ is a gamma distribution parametrized with ($\epsilon_1, \epsilon_2$). Coordinate ascent update equation for these free variational parameters are $$\begin{aligned}
\epsilon_1 \mbox{ } &= \mbox{  }   e + \frac{1}{2}NK\\
\epsilon_2  \mbox{ } &= \mbox{  }    f + \frac{1}{2}\sum_{i=1}^{N} \sum_{k=1}^{K}(\theta_{ik}^2+\Theta_{ik}^2).
\end{aligned}$$

## Online Variational Inference

We now develop online variational inference. We divide the variational parameters into *global* variables and *local* variables. Global variables depend on all of the images. These are the dictionary probabilities $\boldsymbol{\pi}_k$, dictionary elements $\textbf{d}_k$, precisions $\boldsymbol{\alpha}$ and $\gamma$. Local variables are the ones drawn for each image. These are the weights $\textbf{s}_{i}$, binary variables $\textbf{z}_{i}$. The algorithm iterates between optimizing the local variables using local (per-image) coordinate ascent, and optimizing the global variables. This same structure is found in many Bayesian nonparametric models  .

The basic idea is to optimize Equation  via stochastic optimization . This means we repeatedly follow noisy estimates of the gradient with decreasing step sizes $\rho_t$. If the step sizes satisfy $\textstyle \sum_t \rho_t = \infty$ and $\textstyle \sum_t \rho^2_t
<\infty$ then we will converge to the optimum of the objective. (In variational inference, we will converge to a local optimum.)

The noisy estimates of the gradient are obtained from subsampled data. We write the objective $\mathcal{L}$ as a sum over data points. Defining the distribution $g(n)$ which uniformly samples from the data, we can then write $\mathcal{L}$ as an expectation under this distribution, $$\begin{aligned}
  \mathcal{L} &=\sum\nolimits_{n=1}^{N}\ell(\boldsymbol{\tau},
  \boldsymbol{\nu}_n, \boldsymbol{\phi},
  \boldsymbol{\theta}_n,
    \boldsymbol{\lambda}_n, \boldsymbol{\epsilon}_n, \textbf{X}_n).
  \label{subobjective}\\
  &=N\mathbb{E}_{g}[\ell(\boldsymbol{\tau},
\boldsymbol{\nu}_n, \boldsymbol{\phi},
\boldsymbol{\theta}_n, \boldsymbol{\Theta}_n, \boldsymbol{\lambda}_n,
\boldsymbol{\epsilon}_n,  \textbf{X}_n)
]\label{newobjective}
\end{aligned}$$ The gradient of the objective can be written as a similar expectation. Thus, sampling data at random and computing the gradient of $\ell_n$ gives a noisy estimate of the gradient.

There are two further simplifications. First, when we subsample the data we optimize the local variational parameters fully and compute the gradient of $\ell_n$ with respect to only the global variational parameters. Second, we use the natural gradient  rather than the gradient. In mean field variational inference, this simplifies the gradient step as follows. Suppose we have sampled an image $n$ and fitted its local variational parameters given the current settings of the global variational parameters. Let $\tilde{\boldsymbol{\tau}},
\tilde{\boldsymbol{\phi}}, \tilde{\boldsymbol{\Phi}}, \tilde{\boldsymbol{\lambda}}, \tilde{\boldsymbol{\epsilon}}$ be the global variational updates from Section as though we observed $N$ copies of that image. (Note that these depend on its local variational parameters.) Following a noisy estimate of the natural gradient of ${\cal L}$ is equivalent to taking a weighted average of the current and the newly fitted global parameters, e.g. $$\begin{aligned}
  \boldsymbol{\phi} = (1-\rho_t)\boldsymbol{\phi} + \rho_t\tilde{\boldsymbol{\phi}}.  \label{up4}
\end{aligned}$$ It follows that there is no additional computational cost to optimizing the global variational parameters with stochastic optimization versus coordinate ascent.

In our implementation, we decrease the step-size $\rho_t$ by $\rho_t = (\rho_0 +
t)^{-\kappa}$. The learning rate parameter $\rho_0$ down-weights early iterations; the parameter $\kappa$ controls the speed of forgetting previous values of the global variables.

The full online VB algorithm is listed in Algorithm . (Note that we sample the data in mini-batches, rather than one at a time. When the mini-batch size is equal to one data point, we recover the algorithm as described above.)

## Initialization with MCMC.

We initialize both the batch and online VB with a few iterations (e.g. 5 or 10) of MCMC.[^1] This is useful for two reasons: (1) It provides a good initialization and thus faster convergence, (2) Noisy random-walks of MCMC help VB avoid low-quality local optima: at the beginning of each e-step, MCMC initializes $\textbf{s}_i$ and $\textbf{z}_i$ by sampling from their approximate posterior distribution, given the most recent global variables. These samples are noisy estimates of the sparse weights near their posterior means. For instance, when the factor assignment $z_{ik}$ equals $0$, the MCMC draws the sparse weight $s_{ik}$ from the prior $\mathcal{N}(0, 1/\alpha)$ whereas in VB it would be exactly 0. Providing the freedom to "jiggle" gives the algorithm the opportunity, similar to simulated annealing, to jump away from one local optimum to reach a better optimum.

# Experiments

We use three data sets. To train, we use the set of 68 images collected from the web by . We test on the Berkeley natural image data set (20 100$\times$`<!-- -->`{=html}100 images) and a benchmark set of images (11 images of various size) used by the community to evaluate SR algorithms.[^2] These data sets provide us with a rich set of HR-LR pairs.

| **PSNR** | Bic.  |    NNI    | Bil.  |  SME  | ScSR256 |  ScSR512  |  BP   |    OBP    |
|:--------:|:-----:|:---------:|:-----:|:-----:|:-------:|:---------:|:-----:|:---------:|
|  Baboon  | 23.63 |   23.12   | 23.05 | 23.10 |  24.33  |   24.36   | 24.27 | **24.39** |
| Barbara  | 25.35 |   25.10   | 24.92 | 24.42 |  25.88  |   25.89   | 25.98 | **25.99** |
|   Boat   | 29.95 |   28.39   | 28.94 | 29.72 |  31.23  |   31.29   | 31.17 | **31.31** |
|  Camera  | 30.32 | **35.20** | 28.94 | 26.33 |  30.68  |   30.46   | 31.51 |   30.94   |
|  House   | 32.79 |   30.34   | 31.61 | 33.28 |  34.26  | **34.31** | 34.08 |   34.27   |
| Peppers  | 31.99 |   29.88   | 31.18 | 33.06 |  33.05  |   33.06   | 32.45 | **33.08** |
| Parthen. | 28.12 |   27.28   | 27.42 | 27.28 |  29.05  | **29.10** | 28.96 |   29.06   |
|   Girl   | 34.76 |   33.44   | 33.98 | 33.98 |  35.57  |   35.58   | 35.62 | **35.66** |
|  Flower  | 40.04 |   37.96   | 38.94 | 39.72 |  41.06  |   41.11   | 41.26 | **41.33** |
|   Lena   | 32.83 |   31.00   | 31.72 | 33.57 |  34.47  |   34.54   | 34.56 | **34.68** |
| Raccoon  | 30.95 |   29.82   | 29.95 | 31.73 |  32.39  |   32.43   | 32.43 | **32.62** |

Test Results with SR ratio = 2. PSNR for the illuminance channel is presented (the higher the better). **BP**: Proposed algorithm trained via Gibbs sampler, **O-BP** Proposed algorithm trained via Online VB, seeing more data, **ScSR**: Super-Resolution via Sparse Representation , **NNI**: Nearest neighbor interpolation, **SME**: Sparse Mixing Estimation .

Throughout this work, unless otherwise mentioned we use the same parameters (without any tuning): we set the SR ratio to 2 or 4 and the patch size to $8\times 8$.[^3] The hyper-parameters are $c=d=e=f=10^{-6}$ and $c_0=2, \eta_0=0.5$, these are standard uninformative priors used in e.g. . The truncation level $K$ in BP is set to 512. Most images use fewer factors, e.g. Baboon uses 487, House 438 and Barbara 471 factors. We apply all algorithms only to the illuminance channel and use Bicubic interpolation for the color layers (Cb, Cr) for all compared methods.

We study our methods with two kinds of posterior inference—Gibbs sampling (BP) and online variational inference (O-BP), which scales to larger data sets.[^4] To compare, we study both interpolation and example-based algorithms. Bicubic interpolation is the gold standard in the SR literature. We also study nearest neighbor interpolation, bilinear interpolation and sparse mixing estimation (SME) . To compare with an example-based method, we use super-resolution via sparse representation (ScSR, ). [^5] [^6] [^7] Both BP's and ScSR's dictionary learning stages use $10^5$ patches sampled from the training data, however O-BP uses the whole set in online fashion. The HR and LR dictionaries trained by our approach are shown in Figure . The HR dictionary consists of sharper edges.

| **PSNR** | Bic.  |  NNI  | Bilin. | ScSR256 |  ScSR512  |  BP   |   O-BP    |
|:--------:|:-----:|:-----:|:------:|:-------:|:---------:|:-----:|:---------:|
|    N1    | 29.74 | 27.44 | 28.39  |  31.52  |   31.55   | 31.52 | **31.56** |
|    N2    | 29.52 | 27.71 | 28.27  |  31.16  | **31.20** | 31.17 | **31.20** |
|    N3    | 22.97 | 21.95 | 22.12  |  23.94  | **24.00** | 23.80 |   23.94   |
|    N4    | 21.63 | 20.98 | 20.90  |  22.59  | **22.66** | 22.38 |   22.41   |
|    N5    | 24.85 | 23.85 | 24.01  |  26.01  | **26.06** | 25.77 |   25.90   |
|    N6    | 25.34 | 24.61 | 24.70  |  26.20  | **26.26** | 26.08 |   26.07   |
|    N7    | 26.66 | 25.43 | 25.73  |  27.92  |   27.92   | 27.77 | **27.97** |
|    N8    | 26.08 | 24.71 | 25.23  |  27.27  | **27.43** | 27.01 |   27.26   |
|    N9    | 26.02 | 25.29 | 25.42  |  26.82  | **26.89** | 26.58 |   26.73   |
|   N10    | 24.79 | 24.07 | 23.92  |  26.23  | **26.25** | 25.91 |   26.16   |
|   N11    | 26.86 | 25.22 | 25.97  |  28.06  |   28.04   | 27.99 | **28.16** |
|   N12    | 28.16 | 26.65 | 27.07  |  29.63  |   29.66   | 29.78 | **29.86** |
|   N13    | 25.15 | 24.18 | 24.22  |  26.40  | **26.36** | 26.31 |   26.33   |
|   N14    | 26.82 | 25.98 | 25.92  |  27.99  | **28.01** | 27.86 |   27.94   |
|   N15    | 25.78 | 24.64 | 24.81  |  27.00  |   27.04   | 26.90 | **27.06** |
|   N16    | 27.28 | 25.85 | 26.16  |  28.88  | **29.01** | 28.83 |   28.96   |
|   N17    | 27.79 | 26.33 | 26.81  |  29.21  | **29.24** | 29.02 |   29.16   |
|   N18    | 29.13 | 27.75 | 28.18  |  30.38  |   30.41   | 30.25 | **30.43** |
|   N19    | 24.57 | 23.19 | 23.50  |  26.07  | **26.10** | 25.92 |   26.02   |
|   N20    | 22.00 | 21.13 | 21.05  |  23.26  |   23.28   | 23.26 | **23.29** |

Test Results with SR ratio = 2. PSNR for the illuminance channel is presented (the higher the better). **BP**: Proposed algorithm trained via Gibbs sampler, **O-BP** Proposed algorithm trained via Online VB, seeing more data, **ScSR**: Super-Resolution via Sparse Representation , **NNI**: Nearest neighbor interpolation.

As a quantitative measure of performance we compute the signal to noise ratio (PSNR), a measure that is widely used in image recovery applications. We present the PSNR results for benchmark images in Table and natural images in Table . These PSNR based results can be summarized as: (1) The online learning algorithm and ScSR performs similarly, (2) They both slightly perform better than the Gibbs sampler. (3) All of the example based algorithms perform better than the interpolation based techniques.

## Evaluation and Crowdsourcing via Mechanical Turk

Though signal to noise ratio (PSNR), is a widely used metric in image recovery applications, this is not enough to measure human judgement. For this purpose, we also performed human evaluation experiments on Amazon Mechanical Turk (MTurk, http://www.mturk.com).

The Amazon Mechanical Turk (MTurk) is a web interface for deploying small tasks to people, called *Turkers*. Typically an MTurk experiment works as follows: the *requesters*, people organizing the experiments and paying Turkers, prepare tasks called HITs (Human Intelligence Tasks). Each HIT might be a comparison of images, labeling of text etc. Once the HITs are completed, requesters can approve or reject the HITs based on their reliability measures (for instance trivial solution HITs, as we explain next, and the time spend on each HIT are frequently used measures for reliability). Approved results are acquired to be used in the analysis.

While preparing HITs, we used the natural image data. We asked Turkers to visually assess and select the better of two HR reconstructions of each image. We considered all ordered combinations of the algorithms, each equally likely, e.g., BP vs ScSR, BP vs Bicubic etc. We initially collected $42,807$ decisions from 208 unique Turkers. For quality control we gave test pairs in which a ground truth HR image was used, i.e., a comparison of an algorithmic reconstruction vs a true HR image. All of the judgments of the Turkers who failed to pass this test (Turkers who selected the algorithmic reconstruction instead of true HR) were removed. This reduced the data to $20,469$ decisions from 161 unique reliable Turkers.

The results of the human evaluation are in Figure . In the bottom table, win rates for each one-to-one comparisons are provided. Each number represents the winning rate of the method in the column. For instance, $0.93$ for O-BP vs Nearest (O-BP is on the column and Nearest on the row) means that out of 100 binary comparisons of O-BP and Nearest, 93 of the times Turkers voted in favor of O-BP. In general, we observe that example-based methods perform significantly better than interpolation-based methods. Within the example-based approaches, the models are similar. However, our approach does not use the first and second-order derivative filters for the LR patches used by ScSR as features, yet we perform similarly; moreover we do not need to set the noise precision and the number of dictionary elements, both required parameters of ScSR (We used the parameters provided by in ScSR.).

In the PSNR results, ScSR and O-BP seem to perform similarly and both slightly better than BP. However, in the human evaluation we observed that BP reconstructions are found to be better. (Based on 95% confidence intervals, both the BP vs O-BP and BP vs ScSR results are statistically significant. The O-BP vs ScSR difference is statistically insignificant.) This shows that PSNR is not necessarily consistent with the human assessment of images . Sample visual results are shown in Figures , and . (The remaining results are in the appendix E and F.)

## Nonparametric property of the model.

In this section, we demonstrate the importance of a Bayesian nonparametric method for image super-resolution. As we mentioned in Section , we use a beta-Bernoulli process (BP) for the factor assignments $\textbf{z}_i$ that encodes which dictionary elements are activated for the corresponding observation. In the binary matrix (whose rows are the factor assignment $\textbf{z}_i$'s), the columns with at least one active cell correspond to factors that are used.

The distinguishing characteristic of this prior is that the number of the factors to be learned is not specified a priori. Conditioned on the data, we examine the posterior distribution of the binary matrix to obtain a data-dependent distribution of how many components are needed. For the parametric ScSR, the number of dictionary elements must be set $\textit{a priori}$. This is illustrated by the following experiment. For both model, we train on $10^4$ patches, for different values of $K$ (starting from scratch each time); for ScSR, $K$ is the target number (which needs to be set before starting the algorithm), while for our approach, $K$ functions as an upper bound on the number of dictionary elements (which should not be too low). Figure shows that, unlike ScSR, our approach is less sensitive to the value of $K$ if it is sufficiently large. The Barbara image uses 700, 801 and 816 factors in our approach for $K$ equals to $1024, 2048$ and $4096$ respectively.

## Online learning, Computational Time and Scaling

In this section, we compare the scale properties of the algorithms presented in this paper. In online learning, instead of subsampling the patches during the dictionary learning stage, we use the full data set and process the data segment by segment (so called "mini-batches"). We use the training data set of Section . The learning parameters are set to $\kappa = 0.501$ and $\rho_0 = 3$.

Figure shows the evolution of the mean PSNR on the held-out natural image data set by the online and the batch algorithms as a function of the number of image patches seen (visualizations of the learned dictionaries are provided in appendix D). The number of patches seen represents the computational time since both algorithms' time complexity is linear with number of observations. For online VB, the number of patches seen represents the total number of data seen after each iteration. For batch VB, this represents cumulative sum of the number of same data seen after each variational-EM iteration. Even before the second iteration of the batch VB (100K) is completed, online VB with 5K mini-batch converges – reaches to a local optima better than batch VB. This means that the online algorithm finds dictionaries at least as good as those found by the batch VB in only a fraction of the time. As also shown in Table , it finds high quality dictionaries. This may be because stochastic gradient is robust to local optima .

For dictionary training, the convergence time for online VB with 5K mini-batch size is 16 hours. In Gibbs sampling, we throw away the first 1500 samples for the burn-in period and later collect 1500 samples to approximate the posterior distributions. This takes approximately 50 hours on the same machine with an unoptimized Matlab implementation on $10^5$ number of patches. Running Gibbs sampling same amount of time with online VB, i.e. collecting less number of samples such as 500, reduces held-out PSNR between $0.2$ dB to $0.5$ dB, depending on the image. This is consistent with the findings in .

# Discussion

We developed a new model for super-resolution based on Bayesian nonparametric factor analysis, and new algorithms based on Gibbs sampling and online variational inference. With online training, our algorithm scales to very large data sets. We evaluated our method against a leading sparse coding technique and other state of the art methods. We evaluated both with traditional PSNR and by devising a large scale human evaluation. This is a new real-world application that can utilize online variational methods.

The choice of the inference algorithm depends on the usage case. Our results suggest that with more computation time Gibbs sampling performs slightly better (based on human evaluation). If speed is important, our online algorithms can be used without much loss.

Regarding the evaluation metric, the standard in image analysis has been signal-to-noise ratio (PSNR). However, its practical relevance has been questioned . The human eye is sensitive to details which are not always captured in this metric, and that is why we ran a human evaluation. Our experiments show that the signal-to-noise ratio is not necessarily consistent with human judgement.

As future work, our approach can be used as a building block in other, more complicated, probabilistic models. For example, our approach could be developed into a time series to perform super-resolution on video or a hierarchical model can be built that fully generates the whole image instead of patch based approach.

| **PSNR** | Bicubic |  NNI  | Bilinear | ScSR  |  BP   |     |     |     |     |     |
|:--------:|:-------:|:-----:|:--------:|:-----:|:-----:|:---:|:---:|:---:|:---:|:---:|
|    N1    |  24.58  | 22.80 |  23.77   | 25.36 | 25.15 |     |     |     |     |     |
|    N2    |  24.81  | 23.54 |  24.18   | 25.51 | 25.28 |     |     |     |     |     |
|    N3    |  18.97  | 18.43 |  18.60   | 19.39 | 19.38 |     |     |     |     |     |
|    N4    |  18.11  | 17.78 |  17.83   | 18.50 | 18.37 |     |     |     |     |     |
|    N5    |  21.17  | 20.60 |  20.78   | 21.72 | 21.54 |     |     |     |     |     |
|    N6    |  22.12  | 21.68 |  21.84   | 22.43 | 22.41 |     |     |     |     |     |
|    N7    |  22.62  | 21.62 |  22.20   | 23.13 | 23.12 |     |     |     |     |     |
|    N8    |  22.00  | 20.82 |  21.53   | 22.59 | 22.47 |     |     |     |     |     |
|    N9    |  22.90  | 22.27 |  22.59   | 23.10 | 23.16 |     |     |     |     |     |
|   N10    |  21.39  | 21.04 |  21.22   | 21.53 | 21.55 |     |     |     |     |     |
|   N11    |  22.82  | 21.28 |  22.22   | 23.51 | 23.41 |     |     |     |     |     |
|   N12    |  24.09  | 23.10 |  23.53   | 24.74 | 24.66 |     |     |     |     |     |
|   N13    |  21.13  | 20.42 |  20.77   | 21.42 | 21.45 |     |     |     |     |     |
|   N14    |  23.06  | 22.50 |  22.72   | 23.31 | 23.33 |     |     |     |     |     |
|   N15    |  21.79  | 21.15 |  21.45   | 22.11 | 22.16 |     |     |     |     |     |
|   N16    |  22.52  | 21.58 |  22.05   | 23.00 | 22.92 |     |     |     |     |     |
|   N17    |  23.70  | 22.66 |  23.19   | 24.25 | 24.10 |     |     |     |     |     |
|   N18    |  25.21  | 24.38 |  24.74   | 25.48 | 25.62 |     |     |     |     |     |
|   N19    |  20.33  | 19.55 |  19.89   | 20.79 | 20.76 |     |     |     |     |     |
|   N20    |  18.31  | 17.92 |  18.00   | 18.54 | 18.67 |     |     |     |     |     |

Test Set Results with SR ratio = 4. PSNR for the illuminance channel (higher the better). **BP**: Algorithm presented in this paper with Beta Process (BP) Prior trained with Gibbs Sampler, **ScSR**: Super-Resolution via Sparse Representation, **NNI**: Nearest neighbor interpolation.

[^1]: For batch VB, these MCMC samples are collected on the same subset of the data on which batch VB will process. For online VB, they are collected from the mini-batches. In both cases, scale problem of MCMC is not an issue since we only collect few samples (e.g. 5 or 10). As we mentioned before, scale is a problem for MCMC since it needs to go over the data many times for convergence (e.g. thousands of iterations). Time scaling is discussed in more detail in Section

[^2]: We are using SR ratio=2 or 4. For SR ratio 2, the images which do not have even number of rows/columns are cut to have even number of rows/column to prevent any possible mismatch and error in computing PSNR in all algorithms. For instance the last column of pixels from an image of size $330 \times 171$ is excluded so the corresponding image have the size $330 \times 170$.

[^3]: The visual results for SR ratio 4 are in the appendix G.

[^4]: The software for each algorithm presented and all of the visual results will be publicly available.

[^5]: We used the code and implementation provided by . We also used their training images, in order to have a fair comparison, and we did not change any of their parameters (including noise variance).

[^6]: We provide visual comparisons to in appendix H. provides very sharp edges by artificially enhancing them. However, this makes images unrealistic (looking like graphically rendered). Sparse coding allows any single-image SR algorithm as a pre-processing step. Instead of bicubic interpolation (see Figure ) might be used with sparse coding to boost the sharpness of the edges.

[^7]: The dependent hierarchical Beta process (dHBP), another bayesian nonparametric prior, is proposed in . It removes the exchangeability assumption of beta-Bernoulli construction. This prior assumes that each observation $i$ has a covariate $\ell_i \in \mathbb{R}^{\mathcal{L}}$. In this model, the closer the two sparse factor assignments $z_i$ and $z_j$ in the covariate space, the more likely they share similar dictionary elements. It applies dHBP using spatial information as covariates to image inpainting and spiky noise removal, and shows significant improvement over BP. We obtained preliminary results with dHBP for super-resolution. However, in this setting we did not observe improvement over BP.


abstract: Pop III stars are the key to the character of primeval galaxies, the first heavy elements, the onset of cosmological reionization, and the seeds of supermassive black holes. Unfortunately, in spite of their increasing sophistication, numerical models of Pop III star formation cannot yet predict the masses of the first stars. Because they also lie at the edge of the observable universe, individual Pop III stars will remain beyond the reach of observatories for decades to come, and so their properties are unknown. However, it will soon be possible to constrain their masses by direct detection of their supernovae, and by reconciling their nucleosynthetic yields to the chemical abundances measured in ancient metal-poor stars in the Galactic halo, some of which may bear the ashes of the first stars. Here, I review the state of the art in numerical simulations of primordial stars and attempts to directly and indirectly constrain their properties observationally.
author: Daniel J. Whalen
title: The First Stars

```latex
\begin{multicols}{2}
\section{The Simulation Frontier}

Unlike with star formation in the Galaxy today, there is little disagreement over the initial conditions
of star formation in the primordial universe.  The original numerical simulations suggested that Pop
III stars formed in small pregalactic structures known as cosmological minihalos at $z \sim$ 20 - 30,
or $\sim$ 200 Myr after the Big Bang (Bromm, Coppi \& Larson, 1999,2001; Abel, Bryan \& Norman,
2000, 2002; Nakamura \& Umemura, 2001).  These models predicted that the stars formed in
isolation, one per halo, and that they were 100 - 500 M$_{\odot}$.  Pop III stars profoundly
transformed the halos that gave birth to them, expelling their baryons in supersonic ionized flows
and later exploding as supernovae (e.g. Whalen, Abel \& Norman, 2004; Kitayama \& Yoshida 2005;
Whalen \& Norman 2008; Whalen et al. 2008a).  Radiation fronts from these stars also engulfed
nearby halos, either promoting or suppressing star formation in them, thereby regulating the rise of
the first stellar populations (Shapiro et al. 2004; Iliev et al. 2005; Susa \& Umemura 2006; Whalen et
al., 2008b,2010; Hasegawa et al. 2009; Susa et al. 2009).

The original estimates of Pop III stellar masses were not obtained by modeling the actual formation
and evolution of the stars.  They were derived by comparing infall rates at the center of the halo at
early stages of collapse to Kelvin-Helmholz contraction times to place upper limits on the final mass
of the star. Later simulation campaigns in the same vein revealed a much broader range of final
masses for Pop III stars, 30 - 300 M$_{\odot}$ (O'Shea \& Norman, 2007), and that they could form
as binaries in a fraction of the halos (Turk et al., 2009).  Heroic numerical efforts have only recently
achieved the formation of a hydrostatic protostar at the center of the halo (Yoshida et al., 2008) and
the collapse of the central flow into an accretion disk (Stacy et al., 2010; Clark et al., 2011; Smith et
al., 2011; Greif et al., 2011,2012).

\begin{myfigure}
\centerline{\resizebox{70mm}{!}{\includegraphics{f1.eps}}}
\caption{The formation and fragmentation of a Pop III protostellar disk in the Arepo code (Greif et al.,
2012).}
\label{fig:popIIIdisk}
\end{myfigure}

In particular, the disk calculations indicate that they are unstable to fragmentation, raising the
possibility that Pop III stars may have only been tens of solar masses, not hundreds, and that they
may have formed in small swarms of up to a dozen at the centers of primeval halos.  Computer
models of ionizing UV breakout in the final stages of Pop III protostellar disks have also found that
the I-front of the nascent star exits the disk in bipolar outflows that terminate accretion onto the star
and mostly evaporate the disk by the time the star reaches $\sim$ 40 M$_{\odot}$ (Hosokawa et al.,
2011).  This result reinforces the sentiments of some in the community that while the Pop III IMF
was top-heavy, primordial stars may only have been 10 - 40 M$_{\odot}$.

\subsection{High-Mass or Low-Mass Pop III Stars?}

In spite of their increasing sophistication, these simulations should be taken to be very preliminary
for several reasons.  First, Pop III accretion disks form in smoothed-particle hydrodynamics (SPH)
models but not in adaptive mesh refinement (AMR) simulations, although the AMR models have
not evolved the collapse of the halo to the times achieved by SPH calculations.  This raises the
question of whether one technique better captures the transport of angular momentum out of the
center of the cloud than the other, and whether accretion is ultimately spherical or through a disk.
Second, the stability of the disk itself remains an open question because although the simulations
can now fully resolve the disk they do not yet incorporate all of its relevant physics.  In particular,
they lack high-order radiation transport, which regulates the thermal state of the disk and its
tendency to fragment.  Furthermore, the role of primordial magnetic fields in the formation and
evolution of the disk is not well understood (Turk et al., 2012; Widrow et al., 2012).

The use of sink particles to represent disk fragments in the original SPH simulations of Pop III
protostellar disk formation called into question the longevity of the fragments.  Once they are
created in the simulations they are never destroyed, unlike real fragments that could be torn
apart by gravitational torques and viscous forces (Norman, 2010).  More recent moving mesh
simulations performed with the Arepo code that do not rely on sink particles find that the
fragments persist but only evolve the disk for 10 - 20 yr (Greif et al., 2012).  Perhaps most
importantly, no simulation has followed the disks for more than a few centuries, far short of the
time required to assemble a massive star.  Thus, it remains unclear if the fragments in the disk
remain distinct objects or merge with the largest one at the center, building it up into a very
massive star over time through protracted, clumpy accretion.

\subsection{Accretion Cutoff and the Final Masses of Pop III Stars}

This latter point directly impacts estimates of final masses for Pop III stars inferred from numerical
simulations that attempt to model how ionizing UV from the star reverses infall and evaporates the
accretion disk.  At the heart of such models is a simple recipe for the evolution of the protostar that
provides a prescription for its radius and luminosity as a function of time and acquired mass.  The
Hosokawa et al. (2011) 2D calculations take the growth of the protostar to be relatively steady, in
which case it contracts and settles onto the main sequence at $\sim 30$ M$_{\odot}$.  At this point
the star becomes extremely luminous in ionizing UV radiation that halts accretion onto the star in a
few hundred kyr at a final mass of $\sim$ 40 M$_{\odot}$.  If accretion instead turns out to be
clumpy, the protostar could remain puffy and cool and reach much larger masses before burning
off the disk.

A finer point is that all current accretion cutoff simulations evolve both radiation and hydrodynamics
on the Courant time, a practice which is known to lead to serious inaccuracies in I-front propagation
in density gradients (Whalen \& Norman, 2006).  Such coarse time steps may result in premature
I-front breakout and accretion cutoff, and hence underestimates of the final mass of the star.  Three
dimensional simulations with more accurate radiation--matter coupling schemes, both steady and
clumpy accretion scenarios, more realistic prescriptions for protostellar evolution based on
nucleosynthesis codes such as KEPLER (Weaver et al., 1978; Woosley et al., 2002) and a variety of
halo environments may better constrain the Pop III IMF.  However, in judging the power of such
simulations to model the masses of the first stars, it should be remembered that no simulations
realistically bridge the gap in time between the formation and fragmentation of a protostellar disk
and its photoevaporation up to a Myr later.  We note in passing that fragments can also stop
accreting if they are ejected from the disk by 3-body gravitational effects (Greif et al., 2011; Johnson
\& Khochfar, 2011).  These fragments could become very low-mass Pop III stars ($\sim$ 1 M$_{
\odot}$); if so, some of them may live today.

\section{Constraining the Pop III IMF with Stellar Archaeology}

\begin{figure*}
\begin{center}
\begin{tabular}{cc}
\epsfig{file=f2.eps,width=0.40\linewidth,clip=} &
\epsfig{file=f3.eps,width=0.40\linewidth,clip=} \\
\end{tabular}
\end{center}
\caption{Comparing Pop III SN yields to the chemical abundances of three of the most metal-poor
stars (left panel) and the extremely metal-poor (EMP) stars in the Cayrel et al. (2004) and Lai
et al. (2008) surveys (right panel).  In the left panel, the abundances in HE0557-4840 agree well
with the yields from SN model z15G in the Joggerst et al. (2010) study.  In the right panel we
show that higher explosion energy rotating $Z=0$ stars reproduce EMP abundances well.  The
existence of 15 M$_{\odot}$ Pop III stars is required to produce this good agreement with
observations.}
\label{fig:yields}
\end{figure*}

Unfortunately, because they lie at the edge of the observable universe, individual Pop III stars
will remain beyond the reach of direct detection for decades to come, even with their enormous
luminosities (Schaerer 2002) and the advent of the next generation of near infrared (NIR)
observatories such as the James Webb Space Telescope (JWST) and the Thirty-Meter Telescope
(TMT).  However, there have been attempts to indirectly constrain the masses of Pop III stars by
comparing the cumulative elemental yield of their supernovae to the fossil chemical abundances
found in ancient metal-poor stars in the Galactic halo, some of which may be second-generation
stars.  Stellar evolution models indicate that 15 - 40 M$_{\odot}$ primordial stars died in core
collapse (CC) supernovae (SNe) and that 40 - 140 M$_{\odot}$ stars collapsed to black holes,
perhaps with violent pulsational mass loss prior to death (Heger \& Woosley, 2002).  Pop III stars
between 140 and 260 M$_{\odot}$ can die in pair-instability (PI) SNe, with energies up to 100
times those of Type Ia SNe that completely unbind the star and leave no compact remnant
(Chatzopoulos \& Wheeler 2012 have recently discovered that rotating Pop III stars down to 65
M$_{\odot}$ can also die as PI SNe).  These explosions were the first great nucleosynthetic
engines of the universe, expelling up to half the mass of the progenitor in heavy elements into
the early IGM.  Primordial stars above 260 M$_{\odot}$ collapsed directly to black holes, with no
mass loss.

Joggerst et al. (2010) recently calculated the chemical imprint of low-mass Pop III SNe on later
generations of stars by modeling mixing and fallback onto the central black hole in 15 - 40 M$_
{\odot}$ Pop III core collapse explosions with the CASTRO AMR code.  As shown in Figure
\ref{fig:yields}, a simple power-law IMF average of the elemental yields of these explosions is in
good agreement with the fossil abundances in a sample of 130 extremely metal poor stars with
$Z < 10^{-4}$ $Z_{\odot}$ (Cayrel et al., 2004; Lai et al., 2008).  Although these results suggest
that low-mass Pop III stars shouldered the bulk of the chemical enrichment of the early IGM, 40
- 60 M$_{\odot}$ hypernova explosions, whose energies are intermediate to those of CC and PI
SNe, may also have contributed metals at high redshifts (Iwamoto et al. 2005).

To date, the telltale odd-even nucleosynthetic signature of PI SNe has not been found in the fossil
abundance record, leading some to assert that Pop III stars could not have been very massive.
However, the odd-even effect may have been masked by observational bias in previous surveys
(Karlsson et al., 2008).  Reconciling Pop III SN yields to the elemental patterns in metal-poor
stars is still in its infancy for several reasons.  First, only small numbers of extremely metal-poor
stars have been discovered to date, and larger sample sizes would better constrain early SN yields.
Second, measurements of some elements in low-metallicity stars are challenging and in the past
have been subject to systematic error.  Finally, there are many intervening hydrodynamical
processes between the expulsion of the first metals and their uptake into new stars that are not
yet understood.

\section{Finding the First Cosmic Explosions}

\begin{figure*}
\begin{center}
\begin{tabular}{cc}
\epsfig{file=f4a.eps,width=0.40\linewidth,clip=} &
\epsfig{file=f4b.eps,width=0.40\linewidth,clip=} \\
\epsfig{file=f4c.eps,width=0.40\linewidth,clip=} &
\epsfig{file=f4d.eps,width=0.40\linewidth,clip=}
\end{tabular}
\end{center}
\caption{Pop III PI SN light curves for the JWST NIRCam. Clockwise from the upper left panel,
the redshifts are $z =$ 10, 15, 20 and 30.  The optimum filter for each redshift is noted on the
y-axis labels and the times on the x-axes are for the observer frame.  F200, F277 and F356
are at 2.0, 2.77 and 3.56 $\mu$m, respectively.}
\label{fig:JWST_LC}
\end{figure*}

Detection of Pop III SNe would unambiguously probe the masses of primordial stars for the
first time.  Since these explosions are 100,000 times brighter than either their progenitors or
the primitive galaxies that host them, they could be found by JWST or the Wide-Field Infrared
Survey Telescope (WFIRST). However, unlike the Type Ia SNe used to constrain cosmic
acceleration, light from primeval supernovae must traverse the vast cosmic web of neutral
hydrogen that filled the universe prior to the epoch of reionization.  Lyman absorption by this
hydrogen removes or scatters most of the light from ancient supernovae out of our line of
sight, obscuring them.

Whalen et al. (2012a,b,c) have calculated JWST NIR light curves for Pop III pair-instability SNe
with the Los Alamos RAGE and SPECTRUM codes (Gittings et al., 2008; Frey et al., 2012),
which are shown for $z =$ 10, 15, 20 and 30 in Figure \ref{fig:JWST_LC}.  These simulations
include radiation hydrodynamical calculations of the SN light curve and spectra in the local
frame, cosmological redshifting, and Lyman absorption by intergalactic hydrogen.  JWST
detection limits at 2 - 4 $\mu$m are AB magnitude 31 - 32, so it is clear that JWST will be able
to detect the first cosmic explosions in the universe if they are PI SNe (and even perform
spectrometry on them).  Even given JWST's very narrow fields of view at high redshifts,
recent calculations indicate that at least a few PI SNe should be present in any given JWST
survey (Hummel et al. 2012).  Also, because WFIRST detection limits will be AB magnitude 26.5
at 2.2 $\mu$m, it is clear from Figure \ref{fig:JWST_LC} that Pop III PI SNe will be visible to
WFIRST out to $z \sim$ 15 - 20. Since it is an all-sky survey, and because this redshift range
may favor the formation of very massive Pop III stars because of the rise of Lyman-Werner UV
backgrounds (O'Shea \& Norman, 2008), WFIRST will detect much larger numbers of Pop III
SNe.

Could Pop III SNe be detected at later stages by other means?  Whalen et al., (2008b) found
that most of the energy of Pop III CC SNe is eventually radiated away as H and He lines as the
remnant sweeps up and shocks surrounding gas. At later epochs this energy would instead be
lost to fine structure cooling by metals.  In both cases the emission is too dim, redshifted and
drawn out over time to be detected by any upcoming instruments.  However, PI SNe deposit
up to half of their energy into the cosmic microwave background (CMB) by inverse Compton
scattering at $z \sim$ 20 (Kitayama \& Yoshida, 2005; Whalen et al., 2008b) and could impose
excess power on the CMB at small scales (Oh, Cooray \& Kamionkowski, 2003).  The resolution
of current ground-based CMB telescopes such as the \textit{Atacama Cosmology Telescope}
and \textit{South Pole Telescope} approaches that required to directly image Sunyaev-Zeldovich
(SZ) fluctuations from individual Pop III PI SN remnants, so future observatories may detect
them.  Unlike PI SNe, CC SNe deposit little of their energy into the CMB at $z \sim$ 20, and
even less at lower redshifts because the density of CMB photons falls with cosmological
expansion.

The extreme NIR luminosities of primordial PI SNe could contribute to a NIR background
excess, as has been suggested for Pop III stars themselves, i.e. Kashlinsky (2005).  New
calculations reveal that enough synchrotron emission from CC SN remnants is redshifted
into the 21 cm band above $z \sim$ 10 to be directly detected by the \textit{Square Kilometer
Array} (\textit{SKA}) (Meiksen \& Whalen 2012).  Somewhat more energetic hypernovae could
be detected by existing facilities such as the \textit{Extended Very-Large Array} ({\textit{eVLA}})
and \textit{eMERLIN}. PI SN remnants generally expand into ambient media that are too diffuse
to generate a detectable synchrotron signal.  Pop III SN event rates make it unlikely that they
will be found in absorption at 21 cm at $z >$ 10.

The detection of the first cosmic explosions will be one of the most spectacular discoveries in
extragalactic astronomy in the coming decade, opening our first observational window on the
era of first light and the end of the cosmic Dark Ages at $z \sim$ 30.  They will unveil the nature
of primordial stars and constrain scenarios for early cosmological reionization, the process
whereby the universe was gradually transformed from a cold, dark, featureless void into the vast,
hot, ionized expanse of galaxies we observe today.  At somewhat lower redshifts ($z \sim$ 10 -
15), detections of Pop III supernovae will probe the era of primitive galaxy formation, marking the
positions of nascent galaxies on the sky that might otherwise have eluded detection by JWST.
Finally, finding the first supernovae could also reveal the masses of the seeds of the supermassive
black holes lurking at the centers of massive galaxies today.

\thanks

DJW was supported by the Bruce and Astrid McWilliams Center for Cosmology at Carnegie
Mellon University.  Work at LANL was done under the auspices of the National Nuclear Security
Administration of the U.S. Dept of Energy at Los Alamos National Laboratory under Contract No.
DE-AC52-06NA25396. All numerical simulations were performed on Institutional Computing (IC)
and Yellow network platforms at LANL (Conejo, Lobo and Yellowrail).

\begin{thebibliography}{99}

\bibitem{} Abel, T., Bryan, G. \& Norman, M.~L.:  2000, 540, 39.

\bibitem{} Abel, T., Bryan, G. \& Norman, M.~L.:  2002, Science, 295, 93.

\bibitem{} Bromm, V., Coppi, P.~S. and Larson, R.~B.:  1999, ApJL, 527, L5.

\bibitem{} Bromm, V., Coppi, P.~S. and Larson, R.~B.: 2002, ApJ, 564, 23.

\bibitem{} Cayrel, R. et al.:  2004, A\&A, 416, 1117.

\bibitem{} Clark, P.~C. et al.:  2011, Science, 331, 1040.

\bibitem{} Frey, L.~H. et al.:  2012, ApJS, submitted, arXiv:1203.5832.

\bibitem{} Gittings, M. et al.:  2008, Comp. Sci. \& Disc., 1, 015005.

\bibitem{} Greif, T.~H. et al.:  2011, ApJ, 737, 75.

\bibitem{} Greif, T.~H. et al.:  2012, MNRAS, 424, 399.

\bibitem{} Hasegawa, K. et~al.: 2009, MNRAS, 445

\bibitem{} Heger, A. \& Woosley, S.~E.:  2002, ApJ, 567, 532.

\bibitem{} Hosokawa, T. et al.:  2011, Science, 334, 1250.

\bibitem{} Hummel, J. et al.:  2012, ApJ, submitted, arXiv:1112.5207.

\bibitem{} Iliev, I. et al.:  2005, MNRAS, 361, 405

\bibitem{} Iwamoto, N. et al.:  2005, Science, 309, 451.

\bibitem{} Johnson, J.~L. \& Khochfar, S.:  2011, MNRAS, 413, 1184.

\bibitem{} Joggerst, C.~C. et al.:  2010, ApJ, 709, 11.

\bibitem{} Karlsson, T., Johnson, J.~L. \& Bromm, V.:  2008, ApJ, 679, 6.

\bibitem{} Kashlinsky, A. et~al.:  2005, Nature, 438, 45.

\bibitem{} Kitayama K. \& Yoshida, N.:  2005, ApJ, 630, 675

\bibitem{} Lai, D.~K. et al.:  2008, ApJ, 681, 1524.

\bibitem{} Meiksen, A. \& Whalen, D.~J.: 2012, MNRAS, submitted, arXiv:1209:1915.

\bibitem{} Nakamura, F. \& Umemura, M.:  2001, ApJ, 548, 19.

\bibitem{} Norman, M.~L.:  2010, in \textit{First Stars and Galaxies: Challenges in the Coming
Decade}, AIP Conf. Ser. 1294, 17.

\bibitem{} Oh, S.~P., Cooray, A. \& Kamionkowski, M.:  2003, MNRAS, 342, L20.

\bibitem{} O'Shea, B.~L. \& Norman, M.~L.:  2007, ApJ, 654, 66.

\bibitem{} O'Shea, B.~L. \& Norman, M.~L.:  2008, ApJ, 673, 14.

\bibitem{} Schaerer, D.:  2002, A\&A, 382, 28.

\bibitem{} Shapiro, P.~R. et~al.:  2004, MNRAS, 348, 753

\bibitem{} Smith, R.~J. et al.:  2011, MNRAS, 414, 3633.

\bibitem{} Stacy, A. et al.:  2010, MNRAS, 403, 45.

\bibitem{} Susa, H.  \& Umemura, M.:  2006, ApJL, 645, L93

\bibitem{} Susa, H. et al.:  2009, ApJ, 702, 480

\bibitem{} Turk, M.~J. et al.:  2009, Science, 325, 601.

\bibitem{} Turk, M.~J. et al.:  2012, ApJ, 745, 154.

\bibitem{} Weaver, T.~A., Zimmerman, G.~B. \& Woosley, S.~E.:  1978, ApJ, 225, 1021.

\bibitem{} Whalen, D.~J., Abel, T. \& Norman, M.~L.:  2004, ApJ, 610, 14.

\bibitem{} Whalen, D.~J. et al.:  2008a, ApJ, 682, 49.

\bibitem{} Whalen, D.~J. et al.:  2008b, ApJ, 679, 925.

\bibitem{} Whalen, D.~J. et al.:  2010, ApJ, 712, 101.

\bibitem{} Whalen, D.~J. \& Fryer, C.~L.:  2012, ApJL, ApJL, 756, L19

\bibitem{} Whalen, D.~J. \& Norman, M.~L.: 2006, ApJ, 162, 281.

\bibitem{} Whalen, D.~J. \& Norman, M.~L.: 2008, ApJ, 673, 664.

\bibitem{} Whalen, D.~J. et al.:  2012a, arXiv:1209.3457

\bibitem{} Whalen, D.~J. et al.:  2012b, arXiv:1209.5459

\bibitem{} Whalen, D.~J. et al.:  2012c, ApJ, submitted

\bibitem{} Widrow, L.~M. et al.:  2012, SSR, 166, 37.

\bibitem{} Woosley, S.~E., Heger, A. \& Weaver, T.~A.:  2002, Rev. Mod. Phys., 74, 1015.

\bibitem{} Yorke, H.  \& Welz, A.:  1996, A\&A, 315, 555.

\bibitem{} Yoshida, N., Omukai,  K \& Hernquist, L.:  2008, Science, 321, 669.

\end{thebibliography}

\bigskip
\bigskip
\noindent {\bf DISCUSSION}

\bigskip
\noindent {\bf Wolfgang Kundt:} You spoke of a mass range ($> 40$ M$_{\odot}$) for primordial
stars for which they would collapse to black holes.  Where are the nearest of them today?

\bigskip
\noindent {\bf Daniel Whalen:} This is an excellent question.  We recently published a letter (Whalen
\& Fryer 2012) in which we found that most 20 - 40 M$_{\odot}$ Pop III black holes would be ejected
from the cosmological halos that gave birth to them at velocities of 500 - 1000 km s$^{-1}$ by natal
kicks due to asymmetries in the core-collapse engine.  Such velocities are far above the escape
velocity of any halo they would encounter for over a Hubble time, so there is a good chance that
these black holes would be exiled to the voids between galaxies today.  Black holes above 40 M$_{
\odot}$ are unlikely to be born with kicks and remain in the halo, accreting and growing over cosmic
time.  These black holes are much more likely to reside in the galaxies into which their host halos
were taken, a few of which could become the supermassive black holes found in the SDSS quasars
today.

\bigskip
\noindent {\bf Maurice Van Putten:} What fraction of the gas in a cosmological halo ends up in
primordial stars?

\bigskip
\noindent {\bf Daniel Whalen:} It is currently thought that the minimum halo mass for forming a Pop
III star is $\sim$ 10$^5$ M$_{\odot}$ and that 1 - 10 stars are formed with masses of 30 - 300 M$_
{\odot}$.  Thus, a conservative estimate is that 0.1 - 1\% of the baryons in the halo are converted
into stars, and that the rest are evicted from the halo by strong ionized flows over the life of the
stars.

\end{multicols}
```


abstract: Magnetic-dipole (M1) excitations of $^{18}$O and $^{42}$Ca nuclei are investigated within a relativistic nuclear energy density functional framework. In our last work , these nuclei are found to have unique M1 excitation and its sum rule, because of their characteristic structure: the system consists of the shell-closure core plus two neutrons. For a more systematic investigation of the M1 mode, we have implemented a framework based on the relativistic nuclear energy density functional (RNEDF). For benchmark, we have performed the RNEDF calculations combined with the random-phase approximation (RPA). We evaluate the M1 excitation of $^{18}$O and $^{42}$Ca, whose sum-rule value (SRV) of the M1 transitions can be useful to test the computational implementation . We also apply this RNEDF method to $^{208}$Pb, whose M1 property has been precisely measured . Up to the level of the M1 sum rule, our result is in agreement with the experiments, except the discrepancy related with the quenching factors for $g$ coefficients.
address: $^{1}$Department of Physics, Faculty of Science, University of Zagreb, Bijenička c. 32, 10000 Zagreb, Croatia; $^{2}$Research department, Ericsson - Nikola Tesla, Krapinska 45, 10000, Zagreb, Croatia
author: Tomohiro Oishi$^{1}$, Goran Kruz̆ić$^{2}$, and Nils Paar$^{1}$
title: Relativistic nuclear energy density functional approach to magnetic-dipole excitation

# Introduction

The M1 excitation is one of the fundamental phenomena triggered by the electro-magnetic interactions with atomic nuclei. This is the leading mode to couple the unnatural-parity states, i.e. $J^{P}=1^+$ states. One can expect that, from the form of the M1 operator, its resonance can be useful for investigation of spin-orbit level splitting, tensor-force effect, pairing correlations in medium , etc. Noticeable collective motions, including scissors mode in deformed nuclei , can be also activated by the M1 excitation. Also, the analogy between the M1 and Gamow-Teller (GT) modes has attracted a special interest in recent studies . Indeed, zero component of the GT transition is almost identical to the isovector spin-M1 excitation. The GT resonance is expected as the dominant ingredient in neutrino-nucleus reactions in the energy scale of supernova, which can be a key to explain the origin of several elements. For an accurate evaluation of neutrino-nucleus reactions, certain theoretical framework, which can predict the GT as well as M1 excitations throughout the nuclear chart, has been on a serious demand. See also Refs. for more details on the M1 phenomena.

For an evaluation of the M1 mode without limitations on mass numbers, the mean field calculation based on the relativistic nuclear energy density functional (RNEDF) theory can be a suitable option . As an important feature of the M1 mode, its transition mainly occurs between the spin-orbit partners, e.g. $f_{7/2}\longrightarrow f_{5/2}$ for Ca isotopes. From the RNEDF effective Lagrangian, this spin-orbit splitting can be naturally concluded . On the other hand, for computations including unnatural-parity states, one should be careful for the residual interactions in RNEDF. These interactions do not contribute in the ground state (GS) with $J^{P}=0^{+}$, and thus, GS data cannot provide a reference to determine their model parameters. In order to optimize those parameters, one needs to shift focus to the measurable process, where the residual interaction plays an essential role. The M1 excitation is indeed a suitable reference for this purpose.

In this work, we develop a RNEDF-based framework to compute the collective M1 excitation. We adopt the CGS-Gauss system of units in this article.

# Formalism

In the present study, the RNEDF framework has been employed to describe the nuclear ground state properties within the relativistic mean field model at the Hartree level, and nuclear excitations are described using the relativistic random phase approximation (RPA). The respective formalism is derived from an effective Lagrangian density with four fermion contact interaction terms including the isoscalar-scalar, isoscalar-vector, and isovector-vector channels . The effective Lagrangian contains the free-nucleon and density dependent interaction terms, coupling of protons to the electromagnetic field, and the derivative term accounting for the leading effects of finite-range interactions necessary for a quantitative description of nuclear density distribution and radii. Detailed formalism and overview of the model calculations are given in Refs. . In this work, we employ the point-coupling interaction DD-PC1 . The respective set of parameters for the RNEDF has been utilized in several applications, resulting in successful agreement with the experimental data on nuclear ground state properties and excitations.

In this work, we consider the collective M1 excitation of the $^{A}Z$ nucleus up to the one-body-operator level[^1]: $$\hat{\mathcal{Q}}_{\mu} ({\rm M1} ) \equiv  \sum_{k \in A=N+Z} \hat{\mathcal{P}}_{\mu}(k),$$ where $\hat{\mathcal{P}}_{\mu}(k)$ with $\mu=0,\pm 1$ is the single-particle M1 operator for the $k$th nucleon. That is , $$\hat{\mathcal{P}}_{0}   = \mu_{\rm N} \sqrt{\frac{3}{4\pi}} \left( g_l \hat{l}_0 + g_s \hat{s}_0 \right),~~~
 \hat{\mathcal{P}}_{\pm} = (\mp) \mu_{\rm N} \sqrt{\frac{3}{4\pi}} \left( g_l \hat{l}_{\pm} + g_s \hat{s}_{\pm} \right),$$ where $\mu_{\rm N}$ is the nuclear magneton, $\hat{l}_0=\hat{l}_z$, $\hat{l}_{+}=(\hat{l}_x+i \hat{l}_y)/\sqrt{2}$, $\hat{l}_{-}=\hat{l}_{+}^{\dagger}$, and similarly defined for spin operators. Considering the different $g$ coefficients for protons and neutrons, $\hat{\mathcal{Q}}_{\mu}({\rm M1})$ reads $$\hat{\mathcal{Q}}_{\mu} ({\rm M1} ) = %\oprt{Q}^{(p)}_{\mu} ({\rm M1} )  +\oprt{Q}^{(n)}_{\mu} ({\rm M1} ) \nonumber \\ &=&
 \mu_{\rm N} \sqrt{\frac{3}{4\pi}} \left[
   \sum_{i \in Z} \left(  g^{(p)}_s \hat{s}_{\mu}(i)   +g^{(p)}_l \hat{l}_{\mu}(i) \right)
 + \sum_{j \in N} \left(  g^{(n)}_s \hat{s}_{\mu}(j)   +g^{(n)}_l \hat{l}_{\mu}(j) \right) \right].$$ Here $g_l=1~(0)$ and $g_s=5.586~(-3.826)$ for the proton (neutron) . Note that, utilizing the isospin $\tau_0(k)=2t_0(k)=+1~(-1)$ for the $k$th proton (neutron), one can separate the collective M1 operator into the isoscalar (IS) and isovector (IV) terms. That is, $$\begin{aligned}
 \hat{\mathcal{Q}}_{\mu} ({\rm M1} ) &=& \hat{\mathcal{Q}}^{\rm IS}_{\mu} ({\rm M1} ) + \hat{\mathcal{Q}}^{\rm IV}_{\mu} ({\rm M1} ) \nonumber \\ %&& ~~~ \cdot \sum_{k \in A}  \left[  \hat{l}_{\mu}(k)     \left( \frac{g^{(p)}_l+g^{(n)}_l}{2} + \frac{g^{(p)}_l-g^{(n)}_l}{2} \hat{\tau}_0(k) \right)    \right.  \nonumber \\ && ~~~  \left. + \hat{s}_{\mu}(k)    \left(  \frac{g^{(p)}_s+g^{(n)}_s}{2} + \frac{g^{(p)}_s-g^{(n)}_s}{2} \hat{\tau}_0(k) \right)  \right] \nonumber \\
 &=& \mu_{\rm N} \sqrt{\frac{3}{4\pi}} \sum_k \left[ \left( g^{\rm IS}_l\hat{l}_{\mu}(k) + g^{\rm IS}_s \hat{s}_{\mu}(k) \right) +\hat{\tau}_0(k) \left( g^{\rm IV}_l\hat{l}_{\mu}(k) + g^{\rm IV}_s\hat{s}_{\mu}(k)  \right)  \right],
\end{aligned}$$ where $g^{\rm IS}_l = g^{\rm IV}_l = \frac{1}{2}$, $g^{\rm IS}_s \equiv \frac{g^{(p)}_s+g^{(n)}_s}{2} =0.880$, and $g^{\rm IV}_s \equiv \frac{g^{(p)}_s-g^{(n)}_s}{2} = 4.706$. Notice that the IS spin-M1 response is often minor because of the cancellation of the $g$ coefficients.

For the $g$ coefficients, so-called quenching factors have been utilized in M1 calculations : $g^{\rm IS,IV}_{l,s}\longrightarrow \zeta g^{\rm IS,IV}_{l,s}$, where $\zeta$ often should be less than one for the agreement with experimental M1 data. This quenching effect is mainly from the the second-order configuration mixing, or equivalently, coupling with two-particle-two-hole states . In this article, however, we fix $\zeta=1$, except the case with special mentioning.

The M1 excitation strength is evaluated as $$B_{\rm M1}(E_{\gamma}) = \sum_{\mu =0,\pm 1} \left|  \Braket{f | \hat{\mathcal{Q}}_{\mu}({\rm M1} ) |  i}   \right|^2,$$ where $E_{\gamma}=E_f -E_i$ is the excitation energy. For this evaluation, we utilize the random-phase approximation (RPA). Namely, the same procedure has been employed as in Refs. , but in the present analysis the relativistic point coupling interaction is used, with the parameterization DD-PC1. In the transition matrix elements, the magnetic operator $\hat{\mathcal{Q}}_{\mu}({\rm M1} )$ is used. Then $B_{\rm M1}(E_{\gamma})$ is evaluated for the excitation from the $0^{+}$ ground state (GS), $\ket{i}$, to the $1^{+}$ excited state, $\ket{f}$. We assume the spherical symmetry in this work.

Sum-rule values of M1 ($S_{\rm M1}$) obtained in this work. The unit is $\mu_{\rm N}^2$. The corresponding analytic result in Ref.  with the three-body model (3BM) is also shown for comparison.

This work
Ref.

Method

RNEDF (DD-PC1)
3BM

$S_{\rm M1}$ for 18O
2.73
2.79

$S_{\rm M1}$ for 42Ca
2.91
2.99

$S_{\rm M1}$ for 208Pb
52.86
−

# Result and Discussion

## No-pairing sum rule in $^{18}$O and $^{42}$Ca

In order to check the numerical implementation, sum rules of the excitation strength often provide a useful guidance. For the M1 mode, we can refer to one case-limited but available version of its sum rule in Ref. . There, the non-energy-weighted sum-rule value (SRV) of the M1 excitation was evaluated for some specific systems, which consist of the shell-closure core and two valence neutrons or protons. Those systems include, e.g. $^{18}$O and $^{42}$Ca. An advantage of that sum rule is that, when the pairing correlation between the valence nucleons is neglected, the SRV is determined analytically for the corresponding system of interest.

The numerical SRV is determined as $$S_{\rm M1} \equiv \int dE_{\gamma} B_{\rm M1}(E_{\gamma}),$$ where $E_{\gamma}$ is the excitation energy. Our results for the M1 SRV are shown in Table and in Figure for $^{18}$O and $^{42}$Ca. In our RNEDF calculations, the pairing energy is neglected, in order to keep consistency between the no-pairing result in Ref. . For comparison, the results of the three-body model are also shown. We take strength values up to $60$ MeV into account for this SRV. The actual $B_{\rm M1}(E_{\gamma})$ distribution is plotted in Figure . There is one significant peak each in $^{18}$O and $^{42}$Ca. From the configuration results in RNEDF plus RPA calculations, it is found that these peaks are attributable to the neutron transitions of $d_{5/2}\longrightarrow d_{3/2}$ and $f_{7/2}\longrightarrow f_{5/2}$ in $^{18}$O and $^{42}$Ca, respectively. Namely, only the two valence neutrons are active for M1 transitions in these systems.

As displayed in Table , our SRV is obtained consistently to that in Ref. . Thus, our implementation can be valid in the level of no-pairing sum rule. Note that, for $^{18}$O or $^{42}$Ca, the accurate measurement of M1 strength has not been achieved yet.

The excitation energy (position of the peak) of $B_{\rm M1}(E_{\gamma})$ shows an unnegligible difference between Ref. and this work. This problem can be independent of the no-pairing SRV, but should be related with the spin-orbit splitting energies from the two models. We would like to remind that, in Ref. , the $B_{\rm M1}(E_{\gamma})$ distribution is shown to be sensitive to the choice of the pairing model. Also, in the RNEDF side, the M1-excitation energy is expected to depend on the residual interactions, especially pseudo-vector interaction. For more precise discussions, one needs to optimize these model parameters with respect to the experimental data. More details will be given in forthcoming publication .

## Result in $^{208}$Pb

Following the consistency test of no-pairing SRV, we next apply our RNEDF framework to the $^{208}$Pb nucleus. This nucleus is one of the most precisely measured systems with respect to the M1 excitations. Its mean-excitation energy is measured as $E_{\gamma} = 7.3$ MeV . The total SRV of M1 is also evaluated as $S_{\rm M1}= 15-20~\mu^2_{\rm N}$ .

In Figure , the calculated $B_{\rm M1}$ distribution is plotted. The mean-excitation energy is in a good agreement with experiments. As a remarkable difference from $^{18}$O or $^{42}$Ca, the M1 strength shows two peaks. This result is consistent to the two-peak structure in the experimental data . The origin of this two-peak distribution is simple: in $^{208}$Pb, both the valence protons in the $0h_{11/2}$ orbit and neutrons in the $0i_{13/2}$ orbit can be available for the M1 transition. Note also that the other spin-orbit-partner levels are fully occupied, and thus, cannot be active.

The SRV is obtained as $S_{\rm M1} =52.86~\mu^2_{\rm N}$ from our calculation. This value indeed overshoots the experimental result . Here we should mention that, in our calculations, the quenching factor $\zeta$ on the $g$ coefficient has been fixed as one. In some literature , however, it has been suggested that $\zeta \simeq 0.6-0.7$ is necessary for consistency with experimental data. Notice that $S_{\rm M1}$ as well as $B_{\rm M1}(E_{\gamma})$ should be reduced by $\zeta^2$. This procedure then concludes the result $S_{\rm M1} \simeq 20~\mu^2_{\rm N}$, which is in a reasonable agreement with the experimental data. Note that the quenching factor does not change the excitation energy $E_{\gamma}$, which may be shifted only by changing the RNEDF parameters.

# Summary and Outlook

We have developed the RNEDF framework using the RPA in order to investigate the properties of M1 excitations in nuclei. In the benchmark test for $^{18}$O and $^{42}$Ca, our result consistently reproduces the no-pairing SRV . We have also investigated the $^{208}$Pb, and found that, except for the quenching effect, our results are consistent to the experimental energy and the SRV. We note that in the forthcoming study the role of the pseudo-vector interaction terms in the residual RPA interaction will be studied in more details. In addition, the pairing effects on M1 transitions will be studied using the relativistic quasiparticle random phase approximation.

We sincerely thank A. Tamii for suggestions from the experimental side. This work is supported by the QuantiXLie Centre of Excellence, a project co financed by the Croatian Government and European Union through the European Regional Development Fund, the Competitiveness and Cohesion Operational Programme (KK.01.1.1.01).

# References

[^1]: We neglect the meson-exchange-current effect, for which one needs to consider the relevant multi-body terms.