\section{Introduction}
\label{sec:introduction}
The joint estimation of camera parameters and scene structure from a set of images is a fundamental Computer Vision problem, with applications from online camera pose estimation for augmented reality, to large scale reconstruction of objects, buildings and cities for mapping, game asset generation and historical archiving. The former, visual odometry task has recently been shown~\cite{engel2017direct} to significantly improve in accuracy when using a photometric error, rather than the geometric error of more traditional, feature-based methods. There are good theoretical reasons for this: these ``direct'' methods optimize in the domain of pixel errors, the true source of measurement noise. In addition, the approach requires localizability in only \dims{1}, along epipolar lines, rather than \dims{2} for feature-based methods. This enables the use of intensity \emph{edges}, in addition to corners, allowing for a denser reconstruction, and thus more constraints on camera parameters also.

Despite these advantages, the latter task of large scale reconstruction, in particular from sets of internet images, consisting of a large number of photos, each with their own camera intrinsics and lighting conditions, has not yet benefited from a joint, photometric treatment. The de-facto standard approach to this task is to compute camera parameters and sparse geometry using a feature-based structure from motion (SfM) method~\cite{ozyecsil2017survey}, followed by dense geometry reconstruction using a multi-view stereo (MVS) method. The goal of this work is to bring the benefits of a photometric error to the first stage, joint camera and structure estimation, improving the accuracy of inputs to an MVS second stage. Specifically, we tackle the problem of large scale, photometric bundle adjustment, \ie the joint refinement of camera and structure parameters under a photometric error, addressing two key challenges:
\begin{enumerate*}
	\item Handling the variety of both lighting conditions and intrinsic parameters present in a large and diverse set of source images, such as those downloaded from the internet.
	\item Solving an optimization problem involving thousands of camera variables and millions of geometry variables in an efficient and effective manner.
\end{enumerate*}
We do not tackle the initialization problem, required for a fully photometric SfM pipeline, instead using off-the-shelf software to generate initial parameters.
However, we demonstrate that even a photometric refinement of feature-based estimates yields a significant improvement in reconstruction accuracy, as demonstrated quantitatively using the Tanks and Temples (TT) benchmark~\cite{tanks-temples}, and qualitatively on an internet photo collection.

\section{Related work}
\label{sec:related_work}

The joint optimization of structure and camera parameters is common within feature-based systems~\cite{ozyecsil2017survey}, which minimize a geometric error. Often the feature locations themselves are a result of a photometric optimization (\eg KLT tracking~\cite{lucas1981iterative}).  Alternating optimizations of such geometric and photometric errors yields improvements~\cite{furukawa2009accurate}. However, few methods exist which minimize a photometric error directly over structure and motion.
These fall into two main categories: offline reconstruction~\cite{delaunoy2014photometric,goldlucke2014super}, and visual odometry (VO)~\cite{alismail2016photometric,alismail2016direct,engel2017direct,kahler2011tracking}.

The offline methods~\cite{delaunoy2014photometric,goldlucke2014super} model scene structure densely with triangulated meshes, regularized for smoothness. A texture map is inferred, using a texture-to-image error, allowing appearance to be super-resolved~\cite{goldlucke2014super}.
This significantly increases both the number of variables (due to the texture) and dependence between them (due to the mesh and smoothness regularization). As a result, optimization is either alternated over different sets of variables (texture, structure, cameras)~\cite{goldlucke2014super}, or a simple, first order, gradient descent solver~\cite{delaunoy2014photometric}.

The VO methods minimize an image-to-image error~\cite{alismail2016photometric,alismail2016direct,engel2017direct,kahler2011tracking,park2017illumination}, using the structure to compute correspondences between images, thus avoiding the need to infer texture. Image-to-image errors require handling both lens distortion and inverse, or un-, distortion. Camera intrinsics are assumed known and fixed, thus avoiding optimizing lens parameters through the undistortion process. Most methods~\cite{alismail2016photometric,alismail2016direct,engel2017direct,park2017illumination} model structure with sparse, ray-based landmarks: fronto-parallel patches anchored to a pixel location in a source frame, with variable depth. Some MVS methods~\cite{furukawa2010pmvs,habbecke2006plane} optimize both the depth and normal of landmarks, though not jointly with camera parameters. Similarly, earlier photometric VO work~\cite{kahler2011tracking} tracks a few planes of broad extent, optimizing both plane parameters and camera extrinsics. Without smoothness regularization, the landmarks or planes of these VO methods~\cite{alismail2016photometric,alismail2016direct,engel2017direct,kahler2011tracking} are independent of each other. The methods do joint optimization using second order solvers, improving the speed of convergence, but on relatively small problems.

Most of these methods assume constant brightness of a scene point in all images~\cite{alismail2016photometric,delaunoy2014photometric,engel2017direct,goldlucke2014super,kahler2011tracking}. Non-Lambertian surfaces or lighting changes due to time of day or year, or a shifting light source, or by images taken with different cameras, invalidate this assumption. Alismail \etal~\cite{alismail2016direct} transform images into an 8-channel, lighting invariant, binary feature space prior to minimizing the photometric error; this makes the method invariant to local lighting changes, at a cost to computation time and convergence basin size~\cite{woodford2018ncc}. Park \etal~\cite{park2017illumination} evaluated this and other approaches to illumination robustness in the context of direct SLAM. MVS frameworks often use the Normalized Cross Correlation (NCC) photometric score~\cite{furukawa2010pmvs,schonberger2016colmapdense}, which is invariant to affine intensity variations,  over local patches. Recent work on image alignment~\cite{woodford2018ncc} has incorporated this measure into a standard, least squares optimization framework, employed here.

\subsection{Our contributions}
Despite computing dense geometry, our approach has more in common with the VO approaches mentioned above, using an independent, ray-based, planar landmark representation for structure, and a joint, second order solver for optimization.
We contribute the following:
\begin{enumerate*}
	\vspace{-1pt}
	\item Applying an NCC-based photometric framework~\cite{woodford2018ncc} to bundle adjustment. While this measure has been applied to both tracking and MVS, it has not been optimized jointly over both structure and camera parameters.

	\item Optimizing lens distortion parameters with image-to-image errors, requiring differentiation through the lens undistortion process.
	\item A memory efficient implementation of the Variable Projection optimizer~\cite{golub1973varpro,hong2016varproba}, enabling the joint optimization of thousands of camera parameters and millions of structure parameters on a desktop PC.
\end{enumerate*}

\section{Method}
\label{sec:method}
We now describe the key components of our framework: parameterization of camera and structure variables, the photometric cost function, and the optimization framework, plus additional implementation details.

\subsection{Problem parameterization}
Camera parameters define the projection of a \dims3 point, $\vec{X}\in\real^3$, in world coordinates, onto the image plane, in pixel coordinates.
Camera extrinsics consist of $P$ world to image rotations and translations, $\{\mat{R}_i,\vec{t}_i\}_{i=1}^P, \mat{R}_i\in\SO3$, $\vec{t}_i\in\real^3$, one pair per \emph{image}. Intrinsics consist of $C$ linear and lens calibration parameters, $\{\vec{s}_j,\vec{l}_j\}_{j=1}^C$, $\vec{s}_j\in\real^4$, $\vec{l}_j\in\real^2$, one pair per \emph{camera}, where $C\leP$. When $C<P$, some camera intrinsics are shared across input images; in this case an index mapping from image $i$ to camera $j$ is required as input. To simplify notation, we hide this mapping where necessary, and refer to both extrinsics and intrinsics of a given image using the same index.
The world to pixel coordinate ($\vec{x}'$) transform is then given by
\begin{equation}
\label{eqn:world2image}
\vec{x}' = \kappa_{\vec{s}_j}(\varphi_{\vec{l}_j}(\pi(\mat{R}_j\vec{X} + \vec{t}_j))),
\end{equation}
where $\pi(\cdot) : \real^3 \rightarrow \real^2$ is the projection function $\pi([x,y,z]^\trsp) = [x/z,y/z]^\trsp$, $\varphi_\vec{l}(\cdot) : \real^2 \rightarrow \real^2$ is a lens distortion function, and $\kappa_\vec{s}(\cdot) : \real^2 \rightarrow \real^2$ is the linear calibration function
\begin{align}
\kappa_\vec{s}(\vec{x}) = \begin{bmatrix} s_1 & 0 \\ 0 & s_2 \end{bmatrix} \vec{x} + \begin{bmatrix} s_3 \\ s_4 \end{bmatrix}, ~~~~~\text{\suchthat}~~~~~
\kappa^{-1}_\vec{s}(\vec{x}) &= \begin{bmatrix} 1/s_1 & 0 \\ 0 & 1/s_2 \end{bmatrix} \left(\vec{x} - \begin{bmatrix} s_3 \\ s_4 \end{bmatrix}\right).
\end{align}
For lens distortion, we use a standard polynomial radial distortion model:
\begin{equation}
\varphi_\vec{l}(\vec{x}) = \vec{x}(1 + l_1r + l_2r^2 + .. + l_nr^n), ~~~~~\textrm{where}~~~~~r = \|\vec{x}\|^2.
\end{equation}
For world to camera distortion, $n=2$ and $[l_1,l_2]=\vec{l}_j$. For camera to world undistortion (required for our ray-based formulation, described below), the same model can be used with a different set of polynomial coefficients representing the inverse transformation, \suchthat $\varphi^{-1}_\vec{l}(\vec{x})=\varphi_{\phi(\vec{l})}(\vec{x})$. We compute the undistortion coefficients, $\phi(\vec{l})$, in closed form using the first six coefficient formulae of Drap \& Lef{\`e}vre~\cite[Appendix C]{drap2016lens}.

We use a ray-based parameterization of structure, whereby each landmark is anchored to a pixel in an input image. Since we are comparing image texture around such points, we avoid making assumptions about the normal of the surface, and instead model it explicitly. Each landmark consists of a given (fixed) pixel location $\vec{x}$, source frame index $i$, and the variable surface plane parameterization $\vec{n} \in \real^3$ of Habbecke \& Kobbelt~\cite{habbecke2006plane}, in the source image coordinate frame. Its world coordinates are then computed as follows:
\begin{equation}
\label{eqn:image2world}
\vec{X} = \mat{R}_i^\trsp\left(\frac{\bar{\vec{x}}}{\vec{n}^\trsp\bar{\vec{x}}} - \vec{t}_i\right), ~~~~~\text{where}~~~~~\bar{\vec{x}} = \begin{bmatrix} \varphi^{-1}_{\vec{l}_i}(\kappa^{-1}_{\vec{s}_i}(\vec{x})) \\ 1 \end{bmatrix}.
\end{equation}
A pixel to pixel correspondence $\vec{x}\rightarrow\vec{x}'$ from source frame $i$ to target frame $j$, for landmark $k$, can thus be achieved through the substitution of equation (\ref{eqn:image2world}) into equation (\ref{eqn:world2image}), which we represent, for $N$ image coordinates, constituting a patch around the landmark, with the function $\Pi^\params_{ijk}(\cdot) : \real^{2\timesN} \rightarrow \real^{2\timesN}$. $\mathrm{\Theta}=\{\{\mat{R}_i,\vec{t}_i\}_{i=1}^P,\{\vec{s}_j,\vec{l}_j\}_{j=1}^C, \{\vec{n}_k\}_{k=1}^L\}$ denotes the set of all problem parameters, the variables to be optimized. $L$ is the number of landmarks.

\subsubsection{Update parameterization}
Each iteration of optimization computes a parameter update, $\delta\mathrm{\Theta}$. Most parameters, with the exception of rotations, minimally parameterize a Euclidean space, therefore are updated additively, \eg $\vec{n}_k \leftarrow \vec{n}_k + \update{\vec{n}}_k$. For rotations, the update is parameterized (minimally) as $\mat{R}_i \leftarrow \mat{R}_i \Omega(\update{r}_i)$, where $\Omega(\cdot)$ is Rodrigues' formula~\cite{rodriguesformula} for converting a 3-vector into a rotation matrix. In an abuse of notation, by a derivative of rotation, $\frac{\partial}{\partial \mat{R}}$, we mean the derivative of the update, $\frac{\partial}{\partial \update{r}}\big|_{\update{r}=0}$. The update of parameters in general is denoted $\mathrm{\Theta}\leftarrow\mathrm{\Theta}\oplus\delta\mathrm{\Theta}$.

\subsection{Cost formulation}
\label{sec:ls_cost}

Our parameterization gives us a mapping from pixels in one image to pixels in another, via the scene geometry and camera positions and intrinsics; our cost should measure the difference between those two sets of pixels. To ensure that our cost is invariant to local lighting changes as well as unexpected occlusions, we use a robust, locally normalized, least squares NCC cost~\cite{woodford2018ncc}. Specifically, for each landmark (indexed by $k$), anchored in image $\im{I}_i \in \real^{\textrm{H}\times\textrm{W}}$ (we use grayscale images), where $i = I_k$ is the source image index of the \kth landmark, we define a $4 \times 4$ patch of pixels centered on it, with the set of image coordinates $\mat{P}_k \in \real^{2\timesN}$ ($N$ = 16). Each landmark is visible in a subset of input frames, the (given) set of indices of which is denoted $\mathcal{V}_k$. The cost over all landmarks and images is thus given by
\begin{align}
\label{eqn:cost}
\textit{Total cost:} &&
\E{}(\mathrm{\Theta}) &= \|\mathcal{E}_\mathrm{reg}\|^2 + \sum_k\sum_{j\in\mathcal{V}_k}\rho\left(\|\mathcal{E}_{jk}\|^2\right), ~~~~~~~ \rho(s) = \frac{s}{s+\tau^2},\\
\textit{Patch residual:} &&
\mathcal{E}_{jk} &= \Psi\left(\im{I}_j\left(\Pi^\params_{ijk}\left(\mat{P}_k\right)\right)\right) - \Psi\left(\im{I}_i(\mat{P}_k)\right), ~~~~~~~ i = I_k, \\
\textit{NCC normalization:} &&
\Psi(\bar{\im{I}}) &= \frac{\bar{\im{I}} - \mean{\patch}}{\sd{\patch}},~~~~~~~~~~~
\mean{\patch} = \frac{\onestrsp\,\bar{\im{I}}}{N},~~~~~~~~~~~
\sd{\patch} = \|\bar{\im{I}} - \mean{\patch}\|,
\end{align}
with $\im{I}(\mat{P}) = \bar{\im{I}}$ representing sampling, $\ones$ denoting a vector of ones, and $\mathcal{E}_\mathrm{reg}$ being a regularization term (see eq.~(\ref{eqn:intrinsic_regularization})) that ensures  camera intrinsics, known to suffer from degeneracies~\cite{brooks1996recovering}, are well constrained. The Geman-McClure kernel~\cite{black1996robust,geman1985bayesian}, $\rho$, robustifies costs with $\tau = 0.5$. The source frame, $I_k$, can be ignored in $\mathcal{V}_k$, since it contributes no error, by construction.

\subsection{Cost optimization}
\label{sec:ls_optimization}
Equation (\ref{eqn:cost}) defines a robustified non-linear least squares cost, for which many solvers exist~\cite{tingleff2004methods}. These generally involve computing the partial derivatives of residual errors, $\mathcal{E}$, \wrt to the optimization variables, known as the Jacobian,\!\!\footnote{Formulae for specific Jacobians of our cost function are not presented. They can be derived straightforwardly, but modern auto-differentiation tools, such as the C++ Jet type~\cite{ceres-solver} (employed here), make implementing these formulae unnecessary.} $\jac=\frac{\partial\mathcal{E}}{\partial\mathrm{\Theta}}$. Standard implementations of such solvers, \eg Ceres Solver~\cite{ceres-solver}, cache the whole Jacobian, which would be close to 3TB for one dataset used here.
It is not surprising that some approaches resort to alternative strategies to optimize this problem~\cite{goldlucke2014super,delaunoy2014photometric}.

However, our problem has a special structure, common to BA: without surface regularization, the landmarks are independent of each other. Enter the Variable Projection (VarPro) method \cite{golub1973varpro,hong2016varproba}, that, using the Schur complement, allows us to construct and solve a small Reduced Camera System (RCS) problem, then solve for the structure using Embedded Point Iterations (EPIs). The RCS involves the set of all problem variables excluding structure variables, which we denote $\bar{\params}$. The RCS is constructed and solved, using Levenberg-style damping~\cite{levenberg1944method}, as follows~\cite{hong2016varproba}:
\begin{align}
\label{eqn:update_cams}
\delta\bar{\params} &= -(\mat{H}_\mathrm{rcs} + \jac_\mathrm{reg}^\trsp \jac_\mathrm{reg} + \lambda\identity)^{-1}(\vec{g}_\mathrm{rcs} + \jac_\mathrm{reg}^\trsp \mathcal{E}_\mathrm{reg}),\\
\label{eqn:accum_rcs}
\mat{H}_\mathrm{rcs} &= \sum_{k=1}^L \bar{\jac}_k^\trsp\left(\identity - \hat{\jac}_k\jacs^+\right)\bar{\jac}_k, & \vec{g}_\mathrm{rcs} =& \sum_{k=1}^L \bar{\jac}_k^\trsp\left(\identity - \hat{\jac}_k\jacs^+\right)\mathcal{E}_k,\\
\mathcal{E}_k &= \left[\rho'(\mathcal{E}_{jk})\mathcal{E}_{jk}\right]_{\forall j\in\mathcal{V}_k}, & \rho'(s) =& \frac{\partial}{\partial s}\rho(s) = \frac{\tau^2}{(s+\tau^2)^2},\\
\bar{\jac}_k &= \left[\rho'(\mathcal{E}_{jk})\frac{\partial\mathcal{E}_{jk}}{\partial\bar{\params}}\right]_{\forall j\in\mathcal{V}_k}, & \hat{\jac}_k =& \left[\rho'(\mathcal{E}_{jk})\frac{\partial\mathcal{E}_{jk}}{\partial\vec{n}_k}\right]_{\forall j\in\mathcal{V}_k},\\
\label{eqn:intrinsic_regularization}
\mathcal{E}_\mathrm{reg} &= 10^5\cdot\begin{bmatrix}\frac{s_{1i}-s_{2i}}{s_{1i}+s_{2i}}~\frac{s_{3i}-W_i/2}{\max(W_i, H_i)}~\frac{s_{4i}-H_i/2}{\max(W_i, H_i)} \end{bmatrix}_{\forall i\in\{1,..,C\}}^\trsp, & \jac_\mathrm{reg} =& \frac{\partial\mathcal{E}_\mathrm{reg}}{\partial\bar{\params}},
\end{align}
where $\jac^+ = (\jac^\trsp\jac)^{-1}\jac^\trsp$, denoting the matrix pseudo-inverse, $\identity$ is the identity matrix, $\lambda$ is the damping parameter, and $W_i$ \& $H_i$ are the width \& height of the \ith image respectively.
Following the camera parameter update, EPIs are run using a Gauss-Newton update:
\begin{align}
\label{eqn:update_plane}
\update{n}_k &= -\hat{\jac}_k^+\mathcal{E}_k,
\end{align}
until convergence.  Our implementation groups Jacobians per landmark, and sums the reduced system over landmarks (eq. (\ref{eqn:accum_rcs})). While mathematically equivalent to standard VarPro \cite{hong2016varproba,strelow2016varpro}, this explicitly orders the Jacobian computations.  Both the EPIs and construction of the RCS can thus be run over each landmark independently, in parallel. Jacobians for each landmark are not referenced outside these computations, therefore we do not store Jacobians beyond each iteration of the loops over landmarks, slashing the memory requirements of this method.\!\!\footnote{Previous VarPro bundle adjustment methods~\cite{hong2016varproba,strelow2016varpro} do not provide an explicit Jacobian ordering, therefore cannot exploit this memory reduction. Note that this low memory VarPro can be applied to all bundle adjustments, not just our photometric one.} To limit computation time, VarPro is stopped after just ten iterations. The full optimization is described in Algorithm 1.

\subsection{Initialization, and other implementation details}
The framework presented here jointly refines structure and camera parameters of an existing reconstruction, to improve accuracy. Off-the-shelf SfM + MVS systems can provide an initial $\mathrm{\Theta}$. In particular, we use COLMAP~\cite{colmap} with out-of-the-box parameters\footnote{\texttt{colmap automatic\_reconstructor}, with TT datasets using \texttt{--single\_camera}.} to produce initial camera parameters (via SfM~\cite{schonberger2016colmapsfm}) and landmark parameters (via MVS~\cite{schonberger2016colmapdense}).\!\!\footnote{COLMAP outputs the position and normal direction for each landmark, from which our landmark parameterization, $\vec{n}_k$, can be initialized.} Refining \emph{dense} structure jointly with camera parameters in this way might not be needed in some applications, but serves to demonstrate what is feasible with our low memory formulation. Unnecessary background points slow computation, so we manually select landmarks roughly on the object of interest.

In addition to the camera and landmark parameters, our framework needs a source frame index, $I_k$, and visibilities, $\mathcal{V}_k$, per landmark; these remain fixed throughout the optimization. We perform a Poisson surface reconstruction~\cite{kazhdan2013poisson,colmap} on the selected landmarks, and use the resulting mesh to compute visibilities: the mesh is rendered into each view as a depth map, landmarks are projected into the view, and their depths compared to the depth map; those that differ by $<1\%$ are deemed visible. In order to avoid selecting a source frame that is a photometric outlier (\eg due to a specularity), $I_k$ is chosen as the frame whose patch is closest to a robust mean of the visible, normalized patches:
\begin{align}
\label{eqn:select_source}
I_k &= \argmin_{j\in\mathcal{V}_k} \| \bar{\im{I}}_j - \boldsymbol{\mu} \|^2, ~~~~ \boldsymbol{\mu} = \argmin_{\hat\boldsymbol{\mu} \in \real^N} \sum_{j\in\mathcal{V}_k}\rho\left(\|\bar{\im{I}}_j -\hat\boldsymbol{\mu}\|^2\right),\\
\bar{\im{I}}_j &=\Psi\left(\im{I}_j\left(\kappa_{\vec{s}_j}(\varphi_{\vec{l}_j}(\pi(\mat{R}_j\mat{X}_k + \vec{t}_j)))\right)\right),
\end{align}
where $\mat{X}_k$ is a $3\timesN$ matrix of world coordinates, a $4\times 4$ grid of points, spaced such that the mean spacing in visible views is 1 pixel, on the plane around the \kth landmark. $\boldsymbol{\mu}$ is computed using iteratively reweighted least squares~\cite{holland1977irls}, starting from the unrobustified mean. To ensure landmarks are only initialized in textured image regions, we remove ones for which $\|\bar{\im{I}}_{I_k}\| < 0.5N$ (assuming 256 gray levels).

Image pyramids are used to improve convergence. We run the optimization on half size source frames first, followed by full size. Furthermore, to reduce aliasing, target frames are sampled, using bilinear interpolation, at the image pyramid level which produces image samples that are closest to one pixel apart, for each residual $\mathcal{E}_{jk}$. Finally, we optimize structure alone prior to commencing joint optimization at the first resolution.

\begin{figure}[p]
	\resizebox{0.43\linewidth}{!}{
		\begin{minipage}{0.64\linewidth}
			\begin{algorithm}[H]
				\SetAlgoLined
				$\lambda \leftarrow L; ~~ \omega \leftarrow 10;$~~ \# Set damping parameters \\
				Compute initial cost, $S_0 \leftarrow \E{}(\mathrm{\Theta}_0)$, (eq. (\ref{eqn:cost}))\;
				\For{t~=~1:10}{
					$\mathrm{\Theta}_t \leftarrow \mathrm{\Theta}_{t-1};$\\
					\For{k~=~1:L}{
						Add landmark $k$ to RCS (eq. (\ref{eqn:accum_rcs}))
					}
					\nlset{retry} Compute cameras update (eq. (\ref{eqn:update_cams})); \\
					$\bar{\params}_t \leftarrow \bar{\params}_t \oplus\delta\bar{\params}$\;

					\For{k~=~1:L}{
						\While{cost decreases}{
							Compute landmark $k$ update (eq. (\ref{eqn:update_plane}));\\
							$\vec{n}_{kt} \leftarrow \vec{n}_{kt} + \delta\vec{n}_k$;\\
						}
					}
					Compute new cost, $S_t \leftarrow \E{}(\mathrm{\Theta}_t)$, (eq. (\ref{eqn:cost}))\;
					\eIf{$S_t < S_{t-1}$}{
						$\lambda \leftarrow \lambda/10; ~~ \omega \leftarrow 10;$ \# Reduce damping \\
					}{
						$\mathrm{\Theta}_t \leftarrow \mathrm{\Theta}_{t-1};$\\
						$\lambda \leftarrow \max(\lambda \omega, 10^{-6})$; \# Increase damping \\
						$\omega \leftarrow 2\omega$;\\
						\textrm{go to} $\mathbf{retry}$\;
					}
				}
				\caption{Low memory VarPro optimization}
			\end{algorithm}
	\end{minipage}}
	\begin{minipage}{0.55\textwidth}
		\setlength{\imlen}{143pt}\
		{\footnotesize
			\begin{tabularx}{\textwidth}{@{}*{3}{c@{\extracolsep{\fill}}}c@{}}
				\includegraphics[height=\imlen]{precision_training_mean2.pdf} &
				\includegraphics[height=\imlen]{Ignatius_stacked.jpg} &
				\includegraphics[height=\imlen]{Truck_stacked.jpg} &
				\rotatebox{90}{\hspace{0.19\imlen}LSPBA\hspace{0.3\imlen}COLMAP} \\
				(a) Mean precision & (b) Ignatius & (c) Truck
		\end{tabularx}}
		\captionof{figure}{\label{fig:ablation} Results on the TT training sets.}
	\end{minipage}
\end{figure}

\begin{figure}[p]
	\setlength{\imlen}{0.074\linewidth}
	{\tiny
		\begin{tabularx}{\linewidth}{@{}*{9}{c@{\extracolsep{\fill}}}c@{}}
			\rotatebox{90}{~~COLMAP} &
			\includegraphics[height=\imlen]{Family_precision_2.jpg} &
			\includegraphics[height=\imlen]{Francis_precision_2.jpg} &
			\includegraphics[height=\imlen]{Horse_precision_2.jpg} &
			\includegraphics[height=\imlen]{Lighthouse_precision_2.jpg} &
			\includegraphics[height=\imlen]{M60_precision_2.jpg} &
			\includegraphics[height=\imlen]{Panther_precision_2.jpg} &
			\includegraphics[height=\imlen]{Playground_precision_2.jpg} &
			\includegraphics[height=\imlen]{Train_precision_870.jpg} \\
			\rotatebox{90}{~~~~LSPBA} &
			\includegraphics[height=\imlen]{Family_precision_870.jpg} &
			\includegraphics[height=\imlen]{Francis_precision_870.jpg} &
			\includegraphics[height=\imlen]{Horse_precision_870.jpg} &
			\includegraphics[height=\imlen]{Lighthouse_precision_870.jpg} &
			\includegraphics[height=\imlen]{M60_precision_870.jpg} &
			\includegraphics[height=\imlen]{Panther_precision_870.jpg} &
			\includegraphics[height=\imlen]{Playground_precision_870.jpg} &
			\includegraphics[height=\imlen]{Train_precision_886.jpg} \\
			\scalebox{.7}{\rotatebox{90}{\parbox{1.3\imlen}{\centering LSPBA + COLMAP-MVS}}} &
			\includegraphics[height=\imlen]{Family_precision_886.jpg} &
			\includegraphics[height=\imlen]{Francis_precision_886.jpg} &
			\includegraphics[height=\imlen]{Horse_precision_886.jpg} &
			\includegraphics[height=\imlen]{Lighthouse_precision_886.jpg} &
			\includegraphics[height=\imlen]{M60_precision_886.jpg} &
			\includegraphics[height=\imlen]{Panther_precision_886.jpg} &
			\includegraphics[height=\imlen]{Playground_precision_886.jpg} &
			\includegraphics[height=\imlen]{Train_precision_886.jpg}
	\end{tabularx}}
	\caption{\label{fig:intermediate} Precision error visualization for methods on the TT intermediate sets.}
\end{figure}

\begin{table*}[p]
	\centering
	\resizebox{\linewidth}{!}{
		\begin{tabular}{r||c|c|c|c|c|c|c|c|c}
			& \resizebox{2.3em}{!}{Family} & \resizebox{2.3em}{!}{Francis} & \resizebox{2.3em}{!}{Horse} & \resizebox{2.3em}{!}{\parbox{2.7em}{Light- house}} & M60 & \resizebox{2.3em}{!}{Panther} & \resizebox{2.3em}{!}{\parbox{2.7em}{Play- ground}} & Train & Mean \\
			\hline COLMAP~\cite{schonberger2016colmapsfm,schonberger2016colmapdense} & 56.02 & 34.35 & 40.34 & 53.51 & 41.07 & 39.94 & 38.17 & 41.93 & 43.16 \\
			(Baseline) & \it 45.82 & \it 16.46 & \it 18.79 & \it 59.69 & \it 49.34 & \textbf{\emph{57.01}} & \textbf{\emph{66.61}} & \textbf{\emph{42.15}} & \it 44.48 \\
			\hline LSPBA & \bf 68.76 &  \bf 55.79 & 44.95 & \bf 61.91 & \bf 50.48 & 51.02 & 45.75 & 41.99 & \bf 52.58 \\
			(Our method) & \it 41.48 & \textbf{\emph{26.82}} & \it 15.70 & \textbf{\emph{64.43}} & \it 45.30 & \it 46.45 & \it 53.33 & \it 34.63 & \it 41.48 \\
			\hline LSPBA + & 66.15 & 44.60 &  \bf 45.28 & 57.16 & 50.36 & \bf 51.43 & \bf 48.32 & \bf 43.38 & 50.84 \\
			COLMAP-MVS & \textbf{\emph{55.86}} & \it 23.48 & \textbf{\emph{19.08}} & \it 57.71 & \textbf{\emph{52.50}} & \it 56.15 & \it 64.44 & \it 30.35 & \textbf{\emph{44.95}}
		\end{tabular}
	}
	\caption{\label{tbl:intermediate} Published precision \& \emph{recall} scores for methods on the TT intermediate sets.}
\end{table*}

\begin{table}[p]
	\centering
	\resizebox{\linewidth}{!}{
		\begin{tabular}{r||c|c|c|c|c|c|c|c}
			& Barn & \resizebox{2.4em}{!}{\parbox{2.7em}{Cater- pillar}} & \resizebox{2.4em}{!}{Church} & \resizebox{2.4em}{!}{Ignatius} & \resizebox{2.4em}{!}{\parbox{3.2em}{Meeting- room}} & Truck & Mean & \resizebox{2.4em}{!}{\parbox{2.7em}{Change (\%)}} \\
			\hline COLMAP output & 38.00 & 34.79 & 50.04 & 60.39 & 39.50 & 50.96 & 45.61 & 0 \\
			\hline Structure only & 38.13 & 35.66 & 48.78 & 66.16 & 40.29 & 50.48 & 46.58 & 17.21 \\
			\hline Structure + poses & 41.18 & 35.47 & 47.41 & 69.90 & 40.41 & 50.71 & 47.51 & 33.79 \\
			\hline Full method & \bf 48.00 & \bf 39.39 & 49.42 & 72.61 & \bf 43.00 & \bf 55.00 & \bf 51.24 & 100 \\
			\hline Fixed scale & 45.87 & 39.32 & 48.30 & \bf 72.63 & 42.54 & 54.88 & 50.59 & 88.47 \\
			\hline Alternate & 40.50 & 38.41 & \bf 50.35 & 72.54 & 41.61 & 52.15 & 49.26 & 64.84 \\
			\hline One resolution & 40.50 & 37.76 & 47.51 & 71.98 & 41.08 & 53.66 & 48.75 & 55.74 \\
			\hline SSD cost & 35.77 & 22.28 & 30.43 & 43.86 & 27.42 & 41.75 & 33.58 & -214.0 \\ \hline
			\hline Low qual. initial & 38.35 & 29.76 & 51.91 & 50.63 & 37.26 & 47.55 & 42.58 & - \\
			\hline Low qual. refined & 52.48 & 41.80 & 60.42 & 71.88 & 44.16 & 58.18 & 54.82 & -
		\end{tabular}
	}
	\caption{\label{tbl:ablation}Precision AUC scores for various optimizations (discussed in \S\ref{sec:ablation}) on the TT training sets. The final column shows the percentage increase in score of each method, relative to the increase of LSPBA (full method) over the baseline.}
\end{table}

\section{Evaluation}
\label{sec:evaluation}
While we compute both scene geometry and camera poses, our metric of choice is reconstruction accuracy, rather than the camera position accuracy used by VO methods, since reconstruction is more often the end goal of batch methods. Furthermore, ground truth geometry is more readily available than camera poses on large scale datasets, such as Temples and Tanks (TT)~\cite{tanks-temples}.

We perform a quantitative evaluation of metric reconstruction accuracy (up to scale) using the TT benchmark~\cite{tanks-temples}, whose ground truth geometry was captured by LIDAR. We additionally use their training datasets to run an ablation study highlighting the impact of several elements of our framework. The TT sequences, captured as video from a single camera,\!\!\footnote{For TT sequences we optimize a single, global set of camera intrinsics; for internet photo collections we optimize separate intrinsics for each image.} do not have the variety of lighting conditions and camera intrinsics of an internet-sourced dataset, therefore we also provide qualitative results on an internet photo collection.

Ours is the first \emph{photometric} bundle adjustment method suitable for large, diverse image sets with unknown camera poses and intrinsics; previous approaches have all been feature-based. We therefore pick a baseline from that category: COLMAP (SfM~\cite{schonberger2016colmapsfm} + MVS~\cite{schonberger2016colmapdense})). This method leads publicly available, complete SfM + MVS pipelines on TT in terms of precision (our metric of interest), and it is the initializer for our method, so any difference in performance can be entirely attributed to our framework. Photometric bundle adjustment methods exist for more controlled scenarios~\cite{alismail2016photometric,alismail2016direct,delaunoy2014photometric,engel2017direct,goldlucke2014super,kahler2011tracking}, but the VO methods~\cite{alismail2016photometric,alismail2016direct,engel2017direct,kahler2011tracking} cannot be applied to batches of images, while code is not available for existing batch methods~\cite{delaunoy2014photometric,goldlucke2014super}. Nevertheless, our ablation study contrasts features of our framework with those of other photometric methods, so that our contributions can be fairly evaluated against those. In addition, direct comparisons can be done via the TT online leaderboard. We do not compare to state-of-the-art MVS methods, since they don't optimize camera parameters and also incorporate surface regularization and other prior knowledge.

\subsection{Quantitative precision scores on TT}
We ran our algorithm (LSPBA) on the TT intermediate image sets, and also ran COLMAP-MVS~\cite{schonberger2016colmapdense} using the camera parameters produced by our method (LSPBA + COLMAP-MVS), and submitted both sets of results to the online leaderboard~\cite{tanks-temples}. The resulting scores\footnote{Please refer to the TT paper~\cite{tanks-temples} for details on how the scores are computed.}  are presented in Table \ref{tbl:intermediate}, along with those published for COLMAP.\!\footnote{TT COLMAP results may differ from the initial solutions used here, due to different settings, software versions, stochastic effects, and our culling of landmarks. COLMAP results given in \S\ref{sec:ablation} \emph{are} our initialization (\ie after landmark culling).} Figure  \ref{fig:intermediate} visualizes the reconstructions, with colour encoding the distance from ground truth (lighter is closer).

The LSPBA method significantly improves the metric accuracy of reconstruction over COLMAP, improving the mean precision score by 21.8\%. The recall score is 7\% lower, which is unsurprising given its lack of surface smoothness regularization; this allows some poorly constrained landmarks to leave the surface, particularly visible on the playground sequence. Nevertheless, recall is improved on two sequences. Running an MVS method using the refined camera parameters might be expected to give a similar improvement in accuracy, while maintaining the previous level of recall. This is exactly what LSPBA + COLMAP-MVS achieves; it improves accuracy over COLMAP on every sequence, by 17.8\% on average, whilst slightly improving the average recall also.

\vspace{-3pt}
\subsection{Quantitative ablation study}
\label{sec:ablation}
In order to understand which elements of this framework provide benefit, we ran an ablation study on the TT training image sets,
resulting in an error-recall curve for precision per sequence, the mean of which is shown in Figure \ref{fig:ablation}(a), where $\tau$ is the sequence dependent error threshold used in the TT benchmark. Also shown in Figure \ref{fig:ablation}(b,c) are error visualizations for the COLMAP (top) and LSPBA (bottom) methods on two sets. The results are summarized in Table \ref{tbl:ablation}, by computing the area under each curve (per sequence), as a percentage of the total plot area. This AUC score captures more information than reporting recall at $\tau$, the value used in the TT benchmark. We describe and discuss each of the results below.

\textbf{Initialization} is the result of COLMAP, with textureless landmarks culled. It is the baseline, and starting point for all the other optimizations. Marginally better than LSPBA (full method) on the Church sequence, it is otherwise significantly worse.

\textbf{Structure only} is a two pyramid level optimization of structure parameters only, keeping camera poses and intrinsics fixed at their initial values. It delivers 17\% of the improvement of the full method, on average, validating the need for a joint optimization.

\textbf{Structure + poses} is a two pyramid level, joint optimization of structure and pose parameters, keeping camera intrinsics fixed at their initial values. It provides 34\% of the total improvement, validating the need to optimize camera intrinsics as well as poses.

\textbf{Full method} is the complete LSPBA method proposed here; a two pyramid level, joint optimization of structure and camera parameters. It achieves the best score on four of the six sequences, with a significant 12.3\% improvement in AUC over COLMAP.

\textbf{Fixed scale} samples the target image pyramid at the same level as the source image pyramid, rather than using dynamic level selection. Very marginally best on Ignatius, it achieves 89\% of the full method's improvement, demonstrating the modest gains delivered by dynamic level selection.

\textbf{Alternate} replaces the RCS of VarPro with a camera system computed assuming structure is fixed. This then alternates camera and structure updates (10 times), similar to previous work~\cite{furukawa2009accurate,goldlucke2014super}. Marginally best on Church, this approach delivers 65\% of the improvement of the full method overall, validating the benefit of VarPro over alternation.

\textbf{One resolution} applies LSPBA at only the largest image pyramid level, reducing the improvement to 56\% of that using two pyramid levels, demonstrating the benefit of a coarse to fine approach.

\textbf{SSD cost} exchanges the NCC cost of the full method with the sum of squared differences (SSD) cost, which enforces the common constant brightness assumption~\cite{alismail2016photometric,delaunoy2014photometric,engel2017direct,goldlucke2014super}. We used the Huber kernel as robustifier, with a transition threshold of $40^2$\!. This method significantly reduces the precision of the initial solution on all but one sequence, validating the need for a lighting invariant photometric cost in practical applications.

\textbf{Low quality} initial and refined rows refer to using COLMAP on the lowest quality setting\footnote{\texttt{colmap automatic\_reconstructor --single\_camera --quality low}. The density of landmarks is lower (though average precision can be higher), so results are not directly comparable to other rows.} for initialization, and refining this with LSPBA, respectively. Our method improves the accuracy of all sequences, with an average gain in AUC of 29\% (much larger than for the standard initialization), suggesting that it extends well to other initializations.

\subsection{Qualitative results on internet photo collections}
Internet photo collections have a more diverse set of cameras and lighting conditions than the TT datasets, but lack ground truth data. We therefore present only qualitative results, on a publicly available dataset, ``Notre Dame''~\cite{snavely2008modeling}, in Figure \ref{fig:notre_dame}. Panel (b) shows the landmarks, coloured by relief depth, and camera positions before (red) and after (black) refinement. The lowest 10\% of landmarks, ranked by mean photometric cost, are removed to filter out outliers. Comparing the filtered landmarks meshed using Poisson meshing~\cite{kazhdan2013poisson} (d) with the COLMAP landmarks meshed similarly (c), our reconstruction captures significantly finer details, \eg of arches on the towers. It does fail to fix existing, larger scale errors, such as missing balustrade, and introduces more noise on flat regions of the building, due to a lack of texture and smoothness regularization.

To give an idea of the computational resources required for our method, this photo collection, with 701 images and 755k landmarks, took about a week to optimize (not including COLMAP running time), using parallelized\footnote{The two for loops in Algorithm 1 are easily parallelized, \eg using OpenMP.} C++ code on an 8 core 3.7GHz Xeon desktop PC, using 44GB of memory at peak; the full Jacobian for this problem would be 900GB. To accelerate experiments we used a 96 core 3GHz Xeon server; optimization of this dataset took under 5 hours on this machine. This would further improve with GPU acceleration.

\section{Conclusion}
\label{sec:conclusion}
In solving some key challenges, this work enables a new tool for the \dims{3} reconstruction task: refining structure and camera parameters jointly, using a photometric error that is robust to local lighting variations. The framework was evaluated on 15 sets of 150-700 images, with a variety of subject matter. The result is a significant, broad increase in the metric accuracy of reconstruction (up to scale), over a baseline that is representative of the current approach used on this problem: feature-based SfM followed by photometric MVS. Our ablation study provides valuable insight into exactly which aspects of this new approach deliver the most benefit, highlighting the gain in accuracy due specifically to such a refinement.

We have not presented a full system, nor optimized peripheral aspects of the framework, such as landmark selection or visibilities, source frame indices, the robust kernel, landmark weights, or patch sample spacing. We rely on other methods for initialization, which may fail. Improvements are possible in all these areas. Also, we do not propose a replacement to traditional MVS; such systems are complementary, and can be applied after a photometric refinement (which could then use far fewer landmarks), as we show, taking advantage of improved camera pose and intrinsic estimates. We note, however, that our framework could also be incorporated into an MVS method (or surface priors could be added to our method), where all camera parameters are fixed, as well as VO methods, where intrinsic parameters are fixed. Indeed, two widely used MVS frameworks, PMVS~\cite{furukawa2010pmvs} and COLMAP-MVS~\cite{schonberger2016colmapdense}, both use NCC, but neither currently use a second order optimizer or analytic gradients.

\input{paper.bbl}
\end{document}


\section{Introduction}
\label{sec:1}

The massive globular cluster (GC) $\omega$~Cen\ has been studied since the `60s
because of its peculiar stellar populations (see van Leeuwen, Hughes
\& Piotto 2002 for a review). The discovery of its multiple stellar
generations (Bedin et al.\ 2004, Bellini et al.\ 2010) has further
revitalized detailed photometric and spectroscopic investigations of
this object.  The white-dwarf (WD) cooling sequence (CS) is the
least-studied part of the color-magnitude diagram (CMD) of this
cluster, for the obvious reason that WDs are faint and are usually
concentrated in the most-crowded central regions.

Ortolani \& Rosino (1987) were the first to detect a dozen WDs in $\omega$~Cen,
but it is only with the advent of \textit{HST} that it became possible
to analyze the WDCS in more detail (Monelli et al.\ 2005; Calamida et
al.\ 2008).  In particular, Calamida et al.\ showed that the WDCS
spread in color is larger than expected from photometric errors alone.
We will show here that the WDCS of $\omega$~Cen\ actually splits into
\textit{two distinct sequences}, which can be explained as separate WD
populations of different mass and chemical stratification,
corresponding to the evolved components of the separate subpopulations
with differing helium abundances, as revealed by the photometric
analysis of main-sequence stars (e.g.  King et al.~2012 and references
therein).

\section{Data sets and reduction}
\label{sec:2}

This analysis is based on \textit{HST}'s WFC3/UVIS observations taken
between 2009 and 2011 under WFC3 calibration programs.
Specifically:\ we have $27 \times 900\,$s exposures in F225W, $31
\times 800\,$s in F275W, $37 \times 350\,$s in F336W, and $34 \times
350\,$s in F438W.  The field of view (FoV) is centered on the
cluster's center, and covers the inner $\sim 150^{\prime\prime}$.

All WFC3/UVIS \texttt{\_flt} images for a given filter were analyzed
simultaneously to generate an astrometric and photometric catalog of
stars in the field, using an evolution of the software tools described
in Anderson et al.\ (2008).  Briefly, we used information from all the
available exposures for a given filter to find significant detections
across the FoV.  These detections were then fit independently in each
exposure by employing a fixed array of $7\times8$ spatially-varying
static library PSF models, plus a $5\times5$ array of perturbation
PSFs tailored to fit the star profiles in each exposure in order to
account for focus/breathing variations.  Positions were corrected for
geometric distortion according to the routines in Bellini, Anderson \&
Bedin (2011).  The photometry was calibrated into the Vega-mag flight
system following Bedin et al.\ (2005), using the WFC3 zero points
provided by the
STScI\footnote{\url{http://www.stsci.edu/hst/wfc3/phot\_zp\_lbn}.}.

\section{The Double WD CS}
\label{sec:3}

Figure~1 shows the WDCS of $\omega$~Cen\ in three different CMDs, keeping the
$m_{\rm F438W}$ magnitude on the vertical axis, and using a variety of
color combinations.  WDs are shown with heavier dots, for clarity.  We
identified over 2,000 WDs across the FoV.  The WDCS can be followed
for over 6 magnitudes in $m_{\rm F438W}$, starting from just below the
termination point of the extreme horizontal branch (HB).  Two distinct
and well defined WD sequences are visible in the bright part of the
CS, before becoming indistinguishable below $m_{\rm F438W}\sim 25$.  A
few stars, about 1 magnitude redder than the WDCS, define an
additional, scarcely populated sequence.

Stars on the main sequence (MS) of $\omega$~Cen\ are well known to be separated
into three main components: (1) a He-rich bMS, (2) a He-normal,
[Fe/H]-poor rMS, and (3) a [Fe/H]-rich MS-a (Anderson 1997; Bedin et
al.\ 2004; Piotto et al.\ 2005; Bellini et al.\ 2010). The bMS has
been found to be more centrally concentrated than the rMS (Sollima et
al.\ 2007; Bellini et al.\ 2009), a fossil signature of the formation
process of the two populations. Indeed, the present-day relaxation
time ($t_{\rm relax}=12.3\,$Gyr, Harris 1996, 2010 edition) suggests
the two populations have not yet had enough time to mix.

In order to look for a connection between the split WDCS and the split
MS, we measured the radial distribution of the two WD components as
follows.  We started by selecting stars in the $m_{\rm F275W}-m_{\rm
  F438W}$ CMD (Figure 2a).  To optimally separate the populations, we
used a procedure developed by Milone et al.\ (2009).  We first
selected by hand two points (${\rm P}_{\rm 1b}$,${\rm P}_{\rm 2b}$) on
the blue CS, and two points (${\rm P}_{\rm 1r}$,${\rm P}_{\rm 2r}$) on
the red one.  We then linearly transformed the CMD into a reference
frame in which the coordinates of ${\rm P}_{\rm 1b}$, ${\rm P}_{\rm
  2b}$, ${\rm P}_{\rm 1r}$, and ${\rm P}_{\rm 2r}$, map to ($-$1,1),
($-$1,0), (1,1) and (1,0), respectively (Figure~2b). Here the blue
component (the bWD) maps to negative ``abscissa'' values, and the red
component (the rWD) to positive ``abscissa'' values (color coded in
blue and red, respectively, in panel~(b)).  To examine the radial
distribution we followed the methods by Bellini et al.\ (2013).  We
divided the FoV into three equally-populated radial intervals, and fit
in each of them a Gaussian to each of the color-like distribution of
bWD and rWD stars (panels~(c)).  (A dual-Gaussian fit is more robust
than simply adding up stars in histogram bins.)  The radial coverage
of interest and the areas of each of the two Gaussians are given in
the top-left corner of each of these panels.  Panel~(d) shows the
quantity $\hat{p}_{\rm rWD}=N_{\rm rWD}/(N_{\rm rWD}+N_{\rm bWD})$ as
a function of distance from the cluster center. Binomial errors are
$\sim 3.7$~\%.  There is marginal evidence for a gradient, with rWD
stars more concentrated that bWD ones.  Moreover, rWD stars (326) are
$\sim 64$~\% of the total bWD+rWD stars (507).  We note that Bellini et
al.\ (2010) also did not find any strong evidence of a radial gradient
in the ratio of the two main MSs in the inner 2 arcmin of $\omega$~Cen\ (see
their Fig.~18).

The top and middle panels of Fig.~3 show the WDCS CMD in a wide
variety of color systems, with the same stars color-coded the same
ways in all the panels.  The split is present in all of them.  It is
interesting to note, however, that even though all the other
populations (MS, sub-giant branch, etc) show dramatic splits in
two-color diagrams (2CDs), there is {\it no\,} split in the 2CDs
plotted in the bottom panels of Fig.~3.  This implies that the split
is likely due to a discrete mass/radius difference between the
populations, rather than an atmospheric effect.

\begin{figure}[!ht]
\centering
\includegraphics[width=\columnwidth]{figura3n.ps}\\
\caption{\small{\textit{Top and middle panels}: multicolor analysis
                        for the WDCS.
                \textit{Bottom panels}: two-color diagrams. Red and
                        blue WDs are those defined in Figure~2b.}}
\label{f2}
\end{figure}

\section{The nature of the two WDCSs}
\label{sec:4}

We have adopted the distance modulus derived from the cluster
eclipsing binary OGLEGC~17 (Thompson et al.~2001), i.e., $ (m-M)_{\rm
  o}=13.66\pm0.12$, and a mean value of the reddening $E(B-V)=0.12$ as
in King et al.~(2012).  To calculate the extinction in our filters we
have applied the extinction law of Cardelli et al.~(1989, with $R_V$ =
3.1) to our WD spectra used for the calculation of the bolometric
corrections to the UVIS filters
\footnote{\url{http://www.astro.umontreal.ca/\~{}bergeron/CoolingModels}.}.
The resulting extinctions in the F275W, F438W and F336W filters are:
0.75, 0.50 and 0.62~mag, respectively.  These values were applied to
our WD tracks -- transformed to the UVIS filters -- from BaSTI CO-core
WD models by Salaris et al.~(2010), He-core tracks by Bedin et
al.~(2008a) and Serenelli et al.~(2002), and additional CO-core models
calculated specifically for this project.  We focus here on the
$m_{\rm F275W}$ versus $m_{\rm F275W}-m_{\rm F438W}$ CMD, which shows
a very clear separation of the two WDCSs.

The brightest few magnitudes of a WDCS are populated by objects that
just entered the cooling sequence and are therefore cooling extremely
rapidly.  We can thus approximate all the stars in this transient
stage as having the same WD mass, since they presumably all started
with the same turnoff (TO) mass.  It is therefore also safe to match
them with single-mass cooling tracks rather than formal WD isochrones.
Theory predicts a typical WD mass of $\sim 0.54 {\mathcal{M}_{\odot}}$
in old populations (see, e.g., Weiss \& Ferguson~2009), which is
consistent with what Kalirai et al.~(2004) found in a spectroscopic
study of six bright DA WDs in M~4.

\begin{figure}[!ht]
\centering
\includegraphics[width=\columnwidth]{f4n3.ps}
\caption{\small{
   \textit{Left panels}: one of the observed CMDs of Fig.~\ref{f2}
   (top) and a theoretical counterpart (bottom). WD tracks for
   CO-core 0.55${\mathcal{M}_{\odot}}$ DA (blue solid) and DB models
   (blue dash-dotted), 0.46${\mathcal{M}_{\odot}}$ DA CO-core (red
   dotted) and He-core (red dashed) models are fit to the observed
   CMD. In the theoretical panels, stars are color coded in azure
   (0.55${\mathcal{M}_{\odot}}$) and red
   (0.46${\mathcal{M}_{\odot}}$).
    \textit{Right panels}: one of the observed 2CDs of Fig.~\ref{f2}
    (top) and a theoretical counterpart (bottom).}}
\label{ftheory}
\end{figure}

There are a few possible theoretical explanations for the split of the
bright WDCS of $\omega$~Cen: (1) unresolved WD+WD binaries, (2) a combination
of DA+DB objects, or (3) bimodal WD mass distribution, presumably
related to the He-enhanced subpopulation.

Regarding the binary scenario, we note that the split WDCS at these
bright magnitudes cannot be explained by unresolved $\sim 0.54
{\mathcal{M}_{\odot}}$ WD+WD binaries.  Although the split we observe
is very nearly equal to that expected for equal-mass binaries, the
masses of the two components would have to be nearly identical at
birth, since only an extremely narrow range of initial mass occupies
the transient cooling phase at the same time.  If the mass ratio of
the progenitors is distributed approximately homogeneously between 0
and 1 (as found by Milone et al.\ 2012), then the two components would
be continuously distributed over a large magnitude range, and the net
effect would be a moderate widening of the bright WDCS , not a split
(as shown for example by the simulations for NGC~6791, Bedin et
al.~2008b).

We now explore the possibilities that the split is caused by either a
combination of DA+DB objects or a WD mass dicothomy. The top-left
panel of Fig.~\ref{ftheory} superposes on the observed WDCS four
theoretical cooling sequences: a $0.55 {\mathcal{M}_{\odot}}$ CO-core
DA track (blue solid line), the DB counterpart (blue dash-dotted
line), a $0.46 {\mathcal{M}_{\odot}}$ CO-core DA track (red dotted
line), and a He-core track (and DA atmosphere) with the same $0.46
{\mathcal{M}_{\odot}}$ mass (red dashed line).

The $0.55 {\mathcal{M}_{\odot}}$ DA model that fits the bWD sequence
for the eclipsing binary distance represents the ``canonical'' WD
population.  The DB track for the same mass is very close to the DA
track in the region of interest, so the split cannot be explained by
DA and DB populations with the same mass.  We note that a bimodal
H-layer mass distribution is also ruled out because any sequence with
lower thickness (the ``standard'' value ${\rm
  10^{-4}\mathcal{M}_{WD}}$ employed in our models is an upper
evolutionary limit) is shifted to bluer colors and cannot explain the
red sequence.

The only explanation for the rWD objects is therefore a sequence of
lower-mass WDs, which would conseguently have larger radii.  We find
that the rWD sequence can be reproduced by $\sim 0.46
\mathcal{M}_{\odot}$ models, with either CO- or He-cores.

The bottom left-hand panel of Fig.~\ref{ftheory} shows a CMD obtained
from a composite synthetic sample made of the $0.55
{\mathcal{M}_{\odot}}$ DA and $\sim 0.46 \mathcal{M}_{\odot}$ He-core
models.  The photometric error has been implemented following the
results of the data reduction.  The synthetic CMD reproduces the
bright split sequence, which in the simulation merges around $m_{\rm
  F275W} \sim 23.5$ --consistent with the observations-- due to the
increased photometric error.  The right-hand panels of
Fig.~\ref{ftheory} show one of the 2CDs of Fig.~\ref{f2}, both
observed (top) and simulated (bottom).  Notice that, like the
observations, the simulated sample also does not show the bWD+rWD
split in the color-color plane.

\section{The Origin of the rWD Sequence}
\label{sec:5}

It is tempting to map the two WD populations to the two main MS
components, and associate the rWD sequence with the helium-rich MS
population.  To test this hypothesis, we can also use the ratio of bWD
to rWD stars, and compare it with theoretical predictions.  First of
all, we need to know what kind of WDs each is made up of, since CO and
He WDs cool at different rates. An answer to these questions requires
a brief examination of the cluster HB morphology.

Cassisi et al.\ (2009, C09) showed that the progeny of the He-enhanced
MS (${\rm Y}=0.40$) must be confined to the blue tail of the observed
HB, and found that the blue clump of stars at the end of the extended
HB tail (see Fig.~2 of C09) can be reproduced by either hot flashers
or canonical He-rich HB stars, or a mixture of both types.

The minimum mass for the He-rich component along the HB is $\sim 0.46
{\mathcal{M}_{\odot}}$ (C09). These objects will form CO-core WDs with
essentially the same mass.  Even a small amount of additional mass
loss along the red giant branch (RGB) would prevent core He-ignition
and produce $\sim 0.46 \mathcal{M}_{\odot}$ He-core WDs.  Note that
these stars with high initial He abundances need to lose only $\sim$
0.10--0.15 ${\mathcal{M}_{\odot}}$ along the RGB to fall below the
limit for He-ignition, since their TO mass is about $\sim 0.2
{\mathcal{M}_{\odot}}$ less than that of the normal-He stars.

The presence of the ``mass gap'' between the two WD sequences (0.55
vs.\ 0.46 ${\mathcal{M}_\odot}$) supports the idea that the blue clump
in the HB is populated mainly by the evolved members of the He-rich
subpopulation, which eventually become CO and --possibly-- He-core WDs
with a mass of $\sim 0.46 \mathcal{M}_{\odot}$, creating the rWD
sequence.  On the other hand, the bWD sequence is instead produced by
the end-products of He-normal stars (${\rm Y}=0.25$) that populate the
rest of the observed HB. Although a fraction of the He-normal
population {\it could\,} explain the red end of the blue clump, such
stars would have $\sim 0.49 {\mathcal{M}_{\odot}}$ cores and the
resulting WDs, with essentially the same mass (because of the
vanishingly small envelopes of their HB progenitors), would occupy the
gap between the observed sequences.

With this understanding of the evolutionary origin of the rWD
sequence, we investigate whether our interpretation can also explain
the observed $N_{\rm rWD}/N_{\rm bWD}=1.8\pm0.2$ number ratio in the
magnitude range $20.4<m_{\rm F275W}<23.4$.  Our CMDs show that the
ratio between blue-clump stars and the rest of the HB population is
$\sim 0.32$, and the ratio between the He-normal and He-rich
populations along the MS is $\sim 1$ in this central field.

We consider two 13.5 Gyr isochrones from the BaSTI database
(Pietrinferni et al.~2006):\ one with [Fe/H]=$-1.6$, Y=0.246 to
represent the He-normal population, and the other with [Fe/H]=$-1.3$,
Y=0.40 to represent the He-rich component. The exact age is not
critical:\ if the age of both components is decreased by 1-2 Gyr
and/or the He-rich population is made younger by $\sim 1$~Gyr, the
quantitative result of our analysis is largely unchanged.  The
isochrones allow us to calculate the so-called {\sl evolutionary
  flux}, which tells us the number of stars leaving the MS per year:
$$
b(t)=\Phi(\mathcal{M}_{\rm TO}) \left|\frac{\rm d}{{\rm d}t} \mathcal{M}_{\rm TO}\right| =
{\rm A}\times\mathcal{M}_{\rm TO}^{-2.3}\left|\frac{\rm d}{{\rm d}t}\mathcal{M}_{\rm TO}\right|
$$
(Renzini \& Buzzoni~1986). The adopted mass function ${\rm
  \Phi(\mathcal{M}_{\rm TO})}$ has the Salpeter slope.  The TO mass
${\mathcal{M}_{\rm TO}}$ and its time-derivative are determined from
the isochrones, and the constant A is fixed by constraining that the
number of MS stars (below the TO) of the two components to be the
same.  (Note that the MS is the only region that allows us to
determine the mass-function normalization and initial fraction of
He-rich stars).  The value of $b(t)$ of the He-enhanced population
turns out to be 1.1 times the value for the He-normal component. Given
that beyond the TO (apart from the faint end of the WDCS) the initial
value of the evolving mass along an isochrone is
$\sim{\mathcal{M}_{\rm TO}}$, the number of stars in a given post-MS
phase can be approximated by $N_{\rm PMS}= b(t) \times t_{\rm PMS}$
(where ${t_{\rm PMS}}$ is the lifetime of the post-MS evolutionary
stage).

The typical HB mass for stars in the He-normal component (i.e., not in
the blue clump) is $\sim$ 0.61--0.62 ${\mathcal{M}_{\odot}}$. Using
$b(t)$, the model HB timescales provide an expected number ratio of
$\sim 1.7$ between blue-clump stars (He-rich) and the rest of the HB
population.  The observed ratio is $\sim 0.32$, thus only $\sim 20$~\%
of the He-rich population must have produced blue-clump stars. This
means that $\sim 80$~\% of the He-burning stars that we expect from the
He-rich component seen along the MS, must have missed the He-ignition
and evolved onto the WDCS, as He-core WDs.  If these stars left the
RGB close to He-ignition (where mass loss is expected to be more
efficient), they now populate the rWD sequence as $\sim 0.46
{\mathcal{M}_\odot}$ He-core WDs.  Likewise, the progeny on the
(He-rich) HB blue clump produced $\sim 0.46 {\mathcal{M}_\odot}$
CO-core WDs, and the He-normal population produced ``canonical'' $0.55
\mathcal{M}_\odot$ CO-core WDs.

We can now split the {\sl evolutionary flux} for the He-rich
population into two components, according to the fraction of stars
that achieve He-ignition.  Over the magnitude range of interest, the
$0.46 {\mathcal{M}_\odot}$ CO-core model has approximately the same
cooling time of the $0.55 {\mathcal{M}_\odot}$ model (on the order of
$10^8$ yrs), while the $0.46 {\mathcal{M}_\odot}$ He-core model has
cooling timescales a factor of $\sim 2.4$ longer.  By considering
these $b(t)$ and timescale ratios, we derive an expected number ratio:

\begin{eqnarray}
\frac{N_{\rm rWD}}{N_{\rm bWD}}&=&
\frac{b(\rm prog.\ rWD)^{\rm He\ core} \times  {\it t}_{\rm rWD}^{\rm He\ core}}
{b({\rm prog.\ bWD}) \times  t_{\rm bWD}} \nonumber \\
&+&
\frac{b({\rm prog.\ rWD})^{\rm CO\ core} \times  t_{\rm rWD}^{\rm CO\ core}}
{b({\rm prog.\ bWD}) \times  t_{\rm bWD}} \nonumber
\end{eqnarray}
and, since $t_{\rm rWD}^{\rm He\ core} / t_{\rm bWD} = 2.4$ and
$t_{\rm rWD}^{\rm CO\ core} / t_{\rm bWD} = 1.0$, we have
$$
\frac{N_{\rm rWD}}{N_{\rm bWD}}\simeq
 0.8\times1.1\times2.4 + 0.2\times 1.1 \times 1.0  \simeq 2.3,
$$
not far from the observed value of $1.8 \pm 0.2$.  This also implies
that the number of He-core WDs along the rWD sequence is $\simeq
(0.8\times 1.1 \times 2.4) / (0.2 \times 1.1 \times 1.0) \simeq 10$
times the number of CO-core WDs.

To summarize, when taking into account the various evolutionary paths
during the post-MS phases of the He-rich and He-normal MS components
and the appropriate timescales, we can reasonably reproduce the
observed $N_{\rm rWD}/N_{\rm bWD}$ ratio, and derive a very high
fraction of He-core WDs along the red CS.

\section{Summary and Conclusions}
\label{sec:6}

Our high-precision, empirical-PSF-based photometric techniques have
revealed a never-before-seen split in the bright part of the WDCS in a
Galactic GC.  We have interpreted the observed WDCS as follows:
\begin{itemize}
\item{The blue sequence corresponds to $\sim 0.55
  {\mathcal{M}_{\odot}}$ CO-core WDs from the He-normal MS population,
  and the red sequence corresponds to $\sim 0.46
  {\mathcal{M}_{\odot}}$ both CO-core and He-core WDs from the
  He-enriched population.  The radial gradients of the various
  populations are consistent with this scenario.}
\item{The ``mass gap'' between the two WD sequences means that the
  blue clump along the cluster HB has to be populated mainly by stars
  belonging to the He-rich component.}
\item{The observed number ratio between blue-clump HB stars and the
  rest of the HB population is consistent with the number ratio of
  He-rich to He-normal MS stars, only if $\sim 80$~\% of the HB stars
  expected to be produced by the He-rich component failed to achieve
  He-ignition and evolved onto the red WDCS as $\sim 0.46
  {\mathcal{M}_{\odot}}$ He-core WDs.}
\item{It is {\it not\,} extreme mass loss that produces these He-core
  WDs, but rather the fact that the TO mass of the He-rich
  subpopulation is $\sim 0.2 {\mathcal{M}_{\odot}}$ lower than that of
  the He-normal component. A total mass loss of just $\sim 0.15
  {\mathcal{M}_{\odot}}$ along the RGB suffices to prevent central
  He-ignition for this subpopulation.}
\item{The number ratio between CO-core and He-core WDs (with longer
  cooling times) along the red WDCS is $\sim 0.1$.  This mixture of
  WDs along the red WDCS approximately explains the observed number
  ratio (rWD/bWD).  This large contribution of massive He-core WDs
  confirms the more indirect inference by Calamida et al.~(2008), that
  the bright WD sequence must be populated by a substantial fraction
  of He-core WDs.}
\end{itemize}

In conclusion, our observations and theoretical analysis have for the
first time disclosed a connection between globular clusters with
extreme blue HB stars and enhanced He, and the morphology -- hence
mass distribution -- of their WDCS. It would be extremely important to
study the bright WDCS in other clusters with extreme HBs and He-rich
subpopulations (i.e. NGC~6752, NGC~2808) to see whether a split WDCS
analogous to the one detected in $\omega$~Cen\ can be found.

 \acknowledgments \noindent \textbf{Acknowledgments.} AB and JA
 acknowledge support from STScI grant AR-12656.  GP acknowledges
 partial support by the Universit\`a degli Studi di Padova CPDA101477
 grant.

\small{