 stacked spherical harmonic basis functions m depending on the surface normals n parametrized with z, i.e. we actually consider a vector-field m[n[z]] : Ω HR → R 4. In order to perform photometric stereo, n ≥ 3 images under different illumination need to be acquired [56], and since geometry and reflectance are properties of the scene and stay fixed, these changes become only apparent in lighting and intensity, this can be mathematically realized with Practically, changes in illumination can be achieved by using a light stage, illuminating the scene with a light-bulb or LED light source, or capturing pictures during different times of the day or even during different seasons of the year [3,4,28]. Based on (2) and (9), it was shown in [19,44] that under the assumption of {I i } i∈I being corrupted with homoskedastically Gaussian-distributed noise a purely data-driven, photometric regularization term can be established, R [44] 2 2(ΩHR). Apart from avoiding a man-made regularization term, both, the depth prior term (left summand in (2)) and photometric term R [44] (z) can be each others helping hand. Treated separately, depth superresolution and uncalibrated photometric stereo are ill-posed problems by itself [6,8,14], but the combination of both can help disambiguating the other: The depth prior term resulting from an RGB-D sensors initial depth uniquely defines the Lorentz transformation in order to disambiguate uncalibrated photometric stereo [9]. Similarly, using the information comprised in the intensity image and thus in the photometric term can help to result in fine detailed depth estimates and to disambiguate depth super-resolution. To this end it seems intuitive to combine both approaches to get the best out of both worlds. Yet, [44] assumes a static camera-to-object setup during the capturing process which makes it impractical to use in a scenario where camera motion is present. A natural way to capture data would be to attach the light source to the RGB-D sensor and freely move the camera around. This results in an easy way to capture data from different viewpoints while simultaneously changing illumination. Furthermore, resorting to RGB-D cameras has the advantage of robustly capturing depth data, independently of illumination changes in the visible spectrum, which gives coarse, but robust knowledge of the geometry of the scene which can be used during inference. The next section is concerned with how to incorporate the practical setup of attaching an LED light source to an RGB-D camera in the variational formulation. To this end, rigid body motions and how we will use them to model multi-view photometric stereo are discussed.

Using Rigid Body Motion for Multi-View Photometric Stereo
Rigid body motions can be used to describe how a camera moves in space, and additionally gives a relation between pixel positions of different images captured from different viewpoints. Under perspective projection a pixel p ∈ Ω HR ⊂ R 2 and its depth value z(p) is used to express its conjugate 3D point P ∈ R 3 in the scene, where π −1 : Ω HR × R → R 3 can be referred to as the reprojection function. The relation between P seen from one camera and the same corresponding pointP seen from a second camera can be expressed as a rigid body motion. This comprises a rotation R ∈ SO(3), where SO (3) is the space of 3 × 3 dimensional rotation matrices, and a transla- with T[R, t] : R 3 → R 3. After applying the coordinate transformation with T[R, t], the resulting 3D pointP = (X,Ỹ,Z) has to be projected on the image plane again, where π : R 3 → Ω HR. We can describe the process of reprojection (10), camera coordinate transformation (11) and projection (12) using a single warping function w[R, t, z] : Ω HR → Ω HR, w [R, t, z] (p) = π T [R, t] π −1 (p, z (p)). (13) This warping function describes how a pixel from one image frame is transformed to locate the same 3D point in the second image. As we have multiple (namely n ≥ 3) observations due to the multi-view setup, we have to keep track of each rotation and translation. To this end, we transform the camera coordinate system from the reference frame (0th frame, cf. Section 3.1) to the i-th frame using the rotation R i and translation t i. Note that this transformation needs only the depth map from the reference frame, as the re-projection is applied to the reference frame, thus only the rigid body motion is depending on the camera pose of the i-th frame. Photometrically, under the Lambertian assumption the intensity in the i-th frame at pixelp i can be described as illuminating the scene from the i-th frame, but seen from the reference frame at pixel p, mathematically this can be written by combining (9) and (13) as For simplicity, we defineĨ i [R i, t i, z] : and use this representation throughout the rest of this work, before discussing our proposed variational setup.

Proposed Variational Formulation
Putting together the forward model on depth superresolution (1) and multi-view uncalibrated photometric stereo (14), we formulate the joint recovery of camera motion, geometry, reflectance and lighting as the following variational problem We denote with Φ C (·) the sum over all pixels and all channels over data-fitting discrepancy, where φ C (r) = λ 2 2 log 1 + r 2 λ 2 is Cauchy's M-estimator (evaluated at residual r) to make the approach robust against outliers. In the context of photometric stereo outliers can be self-or cast-shadows, specularities, and image noise or quantization, see [47] for more detail. The parameter λ is a tunable hyper-parameter and we heuristically found λ = 0.04 to be a good choice. We will now discuss our numerical approach to solve the variational problem on depth super-resolution using multi-view uncalibrated PS.

Numerical Resolution
The problem depicted in (15) is nonconvex wrt. z (due to the normalization factor, cf. (7)), R i, as well as Cauchy's M-estimator. This makes our variational problem difficult to solve and we will now discuss a strategy to make optimization tractable. To this end, we follow [22,44] and rewrite the normals in terms of its unnormalized normal n [z] and its norm, dA [z], this allows us to split the nonconvexity induced by the normal into its convex (linear) and its nonconvex part. Camera rotation and translation represent elements of the Lie group SE(3), which for the numerical solution we parameterize by the corresponding Lie algebra element, i.e. the 6-dimensional twist coordinates ξ i ∈ R 6, R i = R i (ξ i ) and t i = t i (ξ i ). This allows to optimize over the Euclidean parameters ξ i, cf. [29,39]. We can now rewrite (15) as The nonconvexity induced by φ C in (16) gives rise to perform a reweighted least squares optimization scheme, cf. [47]. This boils down to approximating φ C at the k-th iteration and residual r (k) with its weighted least squares es- and φ 2 r (k) = r (k) 2. Analogously, we denote this approximation at the k-th iteration with Φ (k) C. That is, at the (k)-th iteration with the given estimates ({ξ i }) we update each quantity according to the following sweep: We follow a coarse-to-fine strategy with five pyramid levels, i.e. on the coarsest level the resolution is 1 16 of the original size. For each pyramid level we solve (17) using (18) - (21). The estimated quantities are then (bilinearly upsampled for z and ρ) used as input for the next pyramid level. We initialize ξ (0) 0 ≡ 0 (identity), and ξ i, i ∈ I (small baseline assumption). z (0) and ρ (0) are bilinearly downsampled versions of z 0 and I 0, respectively and l (0) i = (0.2, 0, 0, −1) (little ambient amount and frontal lighting), ∀i ∈ I. Our algorithm has converged if the relative energy falls below a value of 10 −5, which on average took 20 iterations for each pyramid level. Note that (19) - (21) are linear least square problems, while we resort to Gauss-Newton iterations for (18), cf. [29]. Next, we quantitatively and qualitatively evaluate our approach on synthetic and real-world data.

Empirical Validation
We validate our method on synthetic and real-world data. We draft insightful experiments of the involved parameters and compare against other state-of-the-art depth SR approaches.

Synthetic Data
We use two publicly available datasets for synthetic evaluation, "Joyful Yell" [1] and " Stanford Bunny" [53]. Both objects are used and colored within the rendering software Blender. A virtual camera with collocated light source is used to render images from different viewpoints. The resulting data, with its corresponding ground truth is then used for evaluation. Low-resolution depth maps ( 1 2, 1 4, 1 8 of original size) are generated by adding