Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference
Yao Yao Zixin Luo Shiwei Li Tianwei Shen Tian Fang Long Quan







Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference
    
Yao Yao^1Yao Yao is an intern at Shenzhen Zhuke Innovation Technology.
Zixin Luo^1
Shiwei Li^1
Tianwei Shen^1
Tian Fang^2
Long Quan^1


^1The Hong Kong University of Science and Technology 

{yyaoag, zluoag, slibc, tshenaa, quan}@cse.ust.hk 

^2Shenzhen Zhuke Innovation Technology (Altizure) 

fangtian@altizure.com

    December 30, 2023
==================================================================================================================================================================================================================================================================================================================================






Deep learning has recently demonstrated its excellent performance for multi-view stereo (MVS). However, one major limitation of current learned MVS approaches is the scalability: the memory-consuming cost volume regularization makes the learned MVS hard to be applied to high-resolution scenes. 
In this paper, we introduce a scalable multi-view stereo framework based on the recurrent neural network. Instead of regularizing the entire 3D cost volume in one go, the proposed Recurrent Multi-view Stereo Network (R-MVSNet) sequentially regularizes the 2D cost maps along the depth direction via the gated recurrent unit (GRU).
This reduces dramatically the memory consumption and makes high-resolution reconstruction feasible. 
We first show the state-of-the-art performance achieved by the proposed R-MVSNet on the recent MVS benchmarks. Then, we further demonstrate the scalability of the proposed method on several large-scale scenarios, where previous learned approaches often fail due to the memory constraint. Code is available at <https://github.com/YoYo000/MVSNet>.








§ INTRODUCTION

Multi-view stereo (MVS) aims to recover the dense representation of the scene given multi-view images and calibrated cameras. While traditional methods have achieved excellent performance on MVS benchmarks, recent works <cit.> show that learned approaches are able to produce results comparable to the traditional state-of-the-arts. In particular, MVSNet <cit.> proposed a deep architecture for depth map estimation, which significantly boosts the reconstruction completeness and the overall quality. 

One of the key advantages of learning-based MVS is the cost volume regularization, where most networks apply multi-scale 3D CNNs <cit.> to regularize the 3D cost volume. However, this step is extremely memory expensive: it operates on 3D volumes and the memory requirement grows cubically with the model resolution (Fig. <ref> (d)). Consequently, current learned MVS algorithms could hardly be scaled up to high-resolution scenarios. 

Recent works on 3D with deep learning also acknowledge this problem. OctNet <cit.> and O-CNN <cit.> exploit the sparsity in 3D data and introduce the octree structure to 3D CNNs. SurfaceNet <cit.> and DeepMVS <cit.> apply the engineered divide-and-conquer strategy to the MVS reconstruction. MVSNet <cit.> builds the cost volume upon the reference camera frustum to decouple the reconstruction into smaller problems of per-view depth map estimation. However, when it comes to a high-resolution 3D reconstruction (e.g., volume size > 512^3 voxels), these methods will either fail or take a long time for processing.  

To this end, we present a novel scalable multi-view stereo framework, dubbed as R-MVSNet, based on the recurrent neural network. The proposed network is built upon the MVSNet architecture <cit.>, but regularizes the cost volume in a sequential manner using the convolutional gated recurrent unit (GRU) rather than 3D CNNs. With the sequential processing, the online memory requirement of the algorithm is reduced from cubic to quadratic to the model resolution (Fig. <ref> (c)). As a result, the R-MVSNet is applicable to high resolution 3D reconstruction with unlimited depth-wise resolution. 

We first evaluate the R-MVSNet on DTU <cit.>, Tanks and Temples <cit.> and ETH3D <cit.> datasets, where our method produces results comparable or even outperforms the state-of-the-art MVSNet <cit.>. 
Next, we demonstrate the scalability of the proposed method on several large-scale scenarios with detailed analysis on the memory consumption. R-MVSNet is much more efficient than other methods in GPU memory and is the first learning-based approach applicable to such wide depth range scenes, e.g., the advance set of Tanks and Temples dataset <cit.>.







§ RELATED WORK




  
Learning-based MVS Reconstruction
Recent learning-based approaches have shown great potentials for MVS reconstruction. Multi-patch similarity <cit.> is proposed to replace the traditional cost metric with the learned one. SurfaceNet <cit.> and DeepMVS <cit.> pre-warp the multi-view images to 3D space, and regularize the cost volume using CNNs. LSM <cit.> proposes differentiable projection operations to enable the end-to-end MVS training. Our approach is mostly related to MVSNet <cit.>, which encodes camera geometries in the network as differentiable homography and infers the depth map for the reference image. While some methods have achieved excellent performance in MVS benchmarks, aforementioned learning-based pipelines are restricted to small-scale MVS reconstructions due to the memory constraint. 



  
Scalable MVS Reconstruction 
The memory requirement of learned cost volume regularizations <cit.> grows cubically with the model resolution, which will be intractable when large image sizes or wide depth ranges occur. Similar problem also exists in traditional MVS reconstructions (e.g., semi-global matching <cit.>) if the whole volume is taken as the input to the regularization. To mitigate the scalability issue, learning-based OctNet <cit.> and O-CNN <cit.> exploit the sparsity in 3D data and introduce the octree structure to 3D CNNs, but are still restricted to reconstructions with resolution < 512^3 voxels. Heuristic divide-and-conquer strategies are applied in both classical <cit.> and learned MVS approaches <cit.>, however, usually lead to the loss of global context information and the slow processing speed. 

On the other hand, scalable traditional MVS algorithms all regularize the cost volume implicitly. They either apply local depth propagation <cit.> to iteratively refine depth maps/point clouds, or sequentially regularize the cost volume using simple plane sweeping <cit.> and 2D spatial cost aggregation with depth-wise winner-take-all <cit.>. In this work, we follow the idea of sequential processing, and propose to regularize the cost volume using the convolutional GRU <cit.>. GRU is a RNN architecture <cit.> initially proposed for learning sequential speech and text data, and is recently applied to 3D volume processing, e.g., video sequence analysis <cit.>. For our task, the convolutional GRU gathers spatial as well as temporal context information in the depth direction, which is able to achieve comparable regularization results to 3D CNNs.








§ NETWORK ARCHITECTURE

This section describes the detailed network architecture of R-MVSNet. Our method can be viewed as an extension to the recent MVSNet <cit.> with cost volume regularization using convolutional GRU. We first review the MVSNet architecture in Sec. <ref>, and then introduce the recurrent regularization in Sec. <ref> and the corresponding loss formulation in Sec. <ref>.



 §.§ Review of MVSNet
 
Given a reference image 𝐈_1 and a set of its neighboring source images {𝐈_i}_i=2^N, MVSNet <cit.> proposes an end-to-end deep neural network to infer the reference depth map 𝐃. In its network, deep image features {𝐅_i}_i=1^N are first extracted from input images through a 2D network. These 2D image features will then be warped into the reference camera frustum by differentiable homographies to build the feature volumes {𝐕_i}_i=1^N in 3D space. To handle arbitrary N-view image input, a variance based cost metric is proposed to map N feature volumes to one cost volume 𝐂. Similar to other stereo and MVS algorithms, MVSNet regularizes the cost volume using the multi-scale 3D CNNs, and regresses the reference depth map 𝐃 through the soft argmin <cit.> operation. A refinement network is applied at the end of MVSNet to further enhance the depth map quality. As deep image features {𝐅_i}_i=1^N are downsized during the feature extraction, the output depth map size is 1 / 4 to the original image size in each dimension.

MVSNet has shown state-of-the-art performance on DTU dataset <cit.> and the intermediate set of Tanks and Temples dataset <cit.>, which contain scenes with outside-looking-in camera trajectories and small depth ranges. However, MVSNet can only handle a maximum reconstruction scale at H × W × D = 1600 × 1184 × 256 with the 16 GB large memory Tesla P100 GPU, and will fail at larger scenes e.g., the advanced set of Tanks and Temples. To resolve the scalability issue especially for the wide depth range reconstructions, we will introduce the novel recurrent cost volume regularization in the next section.




 §.§ Recurrent Regularization
 



  
Sequential Processing 
An alternative to globally regularize the cost volume 𝐂 in one go is to sequentially process the volume through the depth direction. 
The simplest sequential approach is the winner-take-all plane sweeping stereo <cit.>, which crudely replaces the pixel-wise depth value with the better one and thus suffers from noise (Fig. <ref> (a)). To improve, cost aggregation methods <cit.> filter the matching cost 𝐂(d) at different depths (Fig. <ref> (b)) so as to gather spatial context information for each cost estimation.  
In this work, we follow the idea of sequential processing, and propose a more powerful recurrent regularization scheme based on convolutional GRU. The proposed method is able to gather spatial as well as the uni-directional context information in the depth direction (Fig. <ref> (c)), which achieves regularization results comparable to the full-space 3D CNNs but is much more efficient in runtime memory. 










  
Convolutional GRU
Cost volume 𝐂 could be viewed as D cost maps {𝐂(i)}_i=1^D concatenated in the depth direction. If we denote the output of regularized cost maps as {𝐂_r(i)}_i=1^D, for the ideal sequential processing at the t^th step, 𝐂_r(t) should be dependent on cost maps of the current step 𝐂(t) as well as all previous steps {𝐂(i)}_i=1^t-1. Specifically, in our network we apply a convolutional variant of GRU to aggregate such temporal context information in depth direction, which corresponds to the time direction in language processing. In the following, we denote `⊙' as the element-wise multiplication, `[]' the concatenation and `*' the convolution operation. Cost dependencies are formulated as:

    𝐂_r(t) = (1 - 𝐔(t)) ⊙𝐂_r(t - 1) + 𝐔(t) ⊙𝐂_u(t)

where 𝐔(t) is the update gate map to decide whether to update the output for current step, 𝐂_r(t - 1) is the regularized cost map of late step, and 𝐂_u(t) could be viewed as the updated cost map in current step, which is defined as:

    𝐂_u(t) = σ_c(𝐖_c * [𝐂(t), 𝐑(t) ⊙𝐂_r(t - 1)] + 𝐛_c)

𝐑(t) here is the reset gate map to decide how much the previous 𝐂_r(t - 1) should affect the current update. σ_c(·) is the nonlinear mapping, which is the element-wise sigmoid function. The update gate and reset gate maps are also related to the current input and previous output:

    𝐑(t) = σ_g(𝐖_r * [𝐂(t), 𝐂_r(t - 1)] + 𝐛_r)


    𝐔(t) = σ_g(𝐖_u * [𝐂(t), 𝐂_r(t - 1)] + 𝐛_u)

𝐖 and 𝐛 are learned parameters. The nonlinear σ_g(·) is the hyperbolic tangent to make soft decisions for the updates. 

The convolutional GRU architecture not only spatially regularizes the cost maps through 2D convolutions, but also aggregates the temporal context information in depth direction. We will show in the experiment section that our GRU regularization can significantly outperform the simple winner-take-all or only the spatial cost aggregation.



  
Stacked GRU 
The basic GRU model is comprised of a single layer. To further enhance the regularization ability, more GRU units could be stacked to make a deeper network. In our experiments, we adopt a 3-layer stacked GRU structure (Fig. <ref>). Specifically, we first apply a 2D convolutional layer to map the 32-channel cost map 𝐂(t) to 16-channel as the input to the first GRU layer. The output of each GRU layer will be used as the input to the next GRU layer, and the output channel numbers of the 3 layers are set to 16, 4, 1 respectively. The regularized cost maps {𝐂_r(i)}_i=1^D will finally go through a softmax layer to generate the probability volume 𝐏 for calculating the training loss.




 §.§ Training Loss
 
Most deep stereo/MVS networks regress the disparity/depth outputs using the soft argmin operation <cit.>, which can be interpreted as the expectation value along the depth direction <cit.>. The expectation formulation is valid if depth values are uniformly sampled within the depth range. However, in recurrent MVSNet, we apply the inverse depth to sample the depth values in order to efficiently handle reconstructions with wide depth ranges. Rather than treat the problem as a regression task, we train the network as a multi-class classification problem with cross entropy loss:

    Loss = ∑_𝐩( ∑_i = 1^D -𝐏(i, 𝐩) ·log𝐐(i, 𝐩) )

where 𝐩 is the spatial image coordinate and 𝐏(i, 𝐩) is a voxel in the probability volume 𝐏. 𝐐 is the ground truth binary occupancy volume, which is generated by the one-hot encoding of the ground truth depth map. 𝐐(i, 𝐩) is the corresponding voxel to 𝐏(i, 𝐩). 

One concern about the classification formulation is the discretized depth map output <cit.>. To achieve sub-pixel accuracy, a variational depth map refinement algorithm is proposed in Sec. <ref> to further refine the depth map output. In addition, while we need to compute the whole probability volume during training, for testing, the depth map can be sequentially retrieved from the regularized cost maps using the winner-take-all selection.





§ RECONSTRUCTION PIPELINE

The proposed network in the previous section generates the depth map per-view. This section describes the non-learning parts of our 3D reconstruction pipeline.



 §.§ Preprocessing

To estimate the reference depth map using R-MVSNet, we need to prepare: 1) the source images {𝐈_i}_i=2^N of the given reference image 𝐈_1, 2) the depth range [d_min, d_max] of the reference view and 3) the depth sample number D for sampling depth values using the inverse depth setting. 

For selecting the source images, we follow MVSNet <cit.> to score each image pair using a piece-wise Gaussian function w.r.t. the baseline angle of the sparse point cloud <cit.>. The neighboring source images are selected according to the pair scores in descending order. The depth range is also determined by the sparse point cloud with the implementation of COLMAP <cit.>. Depth samples are chosen within [d_min, d_max] using the inverse depth setting and we determine the total depth sample number D by adjusting the temporal depth resolution to the spatial image resolution (details are described in the supplementary material).



 §.§ Variational Depth Map Refinement
 
As mentioned in Sec. <ref>, a depth map will be retrieved from the regularized cost maps through the winner-take-all selection. Compare to the soft argmin <cit.> operation, the argmax operation of winner-take-all cannot produce depth estimations with sub-pixel accuracy. To alleviate the stair effect (see Fig. <ref> (g) and (h)), we propose to refine the depth map in a small depth range by enforcing the multi-view photo-consistency.

Given the reference image 𝐈_1, the reference depth map 𝐃_1 and one source image 𝐈_i, we project 𝐈_i to 𝐈_1 through 𝐃_1 to form the reprojected image 𝐈_i → 1. The image reprojection error between 𝐈_1 and 𝐈_i → 1 at pixel 𝐩 is defined as:

    E^i(𝐩)    = E^i_photo(𝐩) + E^i_smooth(𝐩)  
       = 𝒞(𝐈_1(𝐩), 𝐈_i → 1(𝐩)) +
    				∑_𝐩' ∈𝒩(𝐩)𝒮(𝐩,𝐩')

where E_photo^i is the photo-metric error between two pixels, E_smooth^i is the regularization term to ensure the depth map smoothness. We choose the zero-mean normalized cross-correlation (ZNCC) to measure the photo-consistency 𝒞(·), and use the bilateral squared depth difference 𝒮(·) between 𝐩 and its neighbors 𝐩' ∈𝒩(𝐩) for smoothness.

During the refinement, we iteratively minimize the total image reprojection error between the reference image and all source images E = ∑_i ∑_𝐩 E_i → 1(𝐩) w.r.t. depth map 𝐃_1. It is noteworthy that the initial depth map from R-MVSNet has already achieved satisfying result. The proposed variational refinement only fine-tunes the depth values within a small range to achieve sub-pixel depth accuracy, which is similar to the quadratic interpolation in stereo methods <cit.> and the DenseCRF in DeepMVS <cit.>.



 §.§ Filtering and Fusion
 
Similar to other depth map based MVS approaches<cit.>, we filter and fuse depth maps in R-MVSNet into a single 3D point cloud. The photo-metric and the geometric consistencies are considered in depth map filtering. As described in previous sections, the regularized cost maps will go through a softmax layer to generate the probability volume. In our experiments, we take the corresponding probability of the selected depth value as its confidence measurement (Fig. <ref> (f)), and we will filter out pixels with probability lower than a threshold of 0.3. The geometric constraint measures the depth consistency among multiple views, and we follow the geometric criteria in MVSNet <cit.> that pixels should be at least three view visible. For depth map fusion, we apply the visibility-based depth map fusion <cit.> as well as the mean average fusion <cit.> to further enhance the depth map quality and produce the 3D point cloud. Illustrations of our reconstruction pipeline are shown in Fig. <ref>.







§ EXPERIMENTS





 §.§ Implementation




  
Training
We train R-MVSNet on the DTU dataset <cit.>, which contains over 100 scans taken under 7 different lighting conditions and fixed camera trajectories. While the dataset only provides the ground truth point clouds, we follow MVSNet <cit.> to generate the rendered depth maps for training. The training image size is set to W × H = 640 × 512 and the input view number is N=3. The depth hypotheses are sampled from 425mm to 905mm with D = 192. In addition, to prevent depth maps from being biased on the GRU regularization order, each training sample is passed to the network with forward GRU regularization from d_min to d_max as well as the backward regularization from d_max to d_min. The dataset is splitted into the same training, validation and evaluation sets as previous works <cit.>. We choose TensorFlow <cit.> for the network implementation, and the model is trained for 100k iterations with batch size of 1 on a GTX 1080Ti graphics card. RMSProp is chosen as the optimizer and the learning rate is set to 0.001 with an exponential decay of 0.9 for every 10k iterations.





  
Testing
For testing, we use N=5 images as input, and the inverse depth samples are adaptively selected as described in Sec. <ref>. For Tanks and Temples dataset, the camera parameters are computed from OpenMVG <cit.> as suggested by MVSNet <cit.>. Depth map refinement, filtering and fusion are implemented using OpenGL on the same GTX 1080Ti GPU. 




 §.§ Benchmarks

We first demonstrate the state-of-the-art performance of the proposed R-MVSNet, which produces results comparable to or outperforms the previous MVSNet <cit.>.





  
DTU Dataset <cit.> 
We evaluate the proposed method on the DTU evaluation set. To compare R-MVSNet with MVSNet <cit.>, we set [d_min, d_max] = [425, 905] and D = 256 for all scans. Quantitative results are shown in Table <ref>. The accuracy and the completeness are calculated using the matlab script provided by the DTU dataset. To summarize the overall reconstruction quality, we calculate the average of the mean accuracy and the mean completeness as the overall score. Our R-MVSNet produces the best reconstruction completeness and overall score among all methods. Qualitative results can be found in Fig. <ref>.





  
Tanks and Temples Benchmark <cit.>
Unlike the indoor DTU dataset, Tanks and Temples is a large dataset captured in more complex environments. Specifically, the dataset is divided into the intermediate and the advanced sets. The intermediate set contains scenes with outside-look-in camera trajectories, while the advanced set contains large scenes with complex geometric layouts, where almost all previous learned algorithms fail due to the memory constraint. 

The proposed method ranks 3^rd on the intermediate set, which outperforms the original MVSNet <cit.>. Moreover, R-MVSNet successfully reconstructs all scenes and also ranks 3^rd on the advanced set. The reconstructed point clouds are shown in Fig. <ref>. It is noteworthy that the benchmarking result of Tanks and Temples is highly dependent on the point cloud density. Our depth map is of size H/4×W/4, which is relatively low-resolution and will result in low reconstruction completeness. So for the evaluation, we linearly upsample the depth map from the network by two (H/2×W/2) before the depth map refinement. The f_scores of intermediate and advanced sets increase from 43.48 to 48.40 and from 24.91 to 29.55 respectively. 





  
ETH3D Benchmark <cit.>
We also evaluate our method on the recent ETH3D benchmark. The dataset is divided into the low-res and the high-res scenes, and provides the ground truth depth maps for MVS training. We first fine-tune the model on the ETH3D low-res training set, however, observe no performance gain compared to the model only pre-trained on DTU. We suspect the problem may be some images in low-res training set are blurred and overexposed as they are captured using hand-held devices. Also, the scenes of ETH3D dataset are complicated in object occlusions, which are not explicitly handled in the proposed network. We evaluate on this benchmark without fine-tuning the network. Our method achieves similar performance to MVSNet <cit.> and ranks 6^th on the low-res benchmark. 















 §.§ Scalability
 
Next, we demonstrate the scalability of R-MVSNet from: 1) wide-range and 2) high-resolution depth reconstructions. 





  
Wide-range Depth Reconstructions 
The memory requirement of R-MVSNet is independent to the depth sample number D, which enables the network to infer depth maps with large depth range that is unable to be recovered by previous learning-based MVS methods. Some large scale reconstructions of Tanks and Temples dataset are shown in Fig. <ref>. Table <ref> compares MVSNet <cit.> and R-MVSNet in terms of benchmarking rankings, reconstruction scales and memory requirements. We define the algorithm's memory utility (Mem-Util) as the size of volume processed per memory unit (H/4×W/4× D / runtime memory size). R-MVSNet is ×8 more efficient than MVSNet in Mem-Util. 





  
High-resolution Depth Reconstructions 
R-MVSNet can also produce high-resolution depth reconstructions by sampling denser in depth direction. For the DTU evaluation in Sec. <ref>, if we fix the depth range and change the depth sample number from D=256 to D=512, the overall distance score will be reduced from 0.422mm to 0.419mm (see last row of Table <ref>).










 §.§ Ablation studies




  §.§.§ Networks

This section studies how different components in the network affect the depth map reconstruction. We perform the study on DTU validation set with W × H × D = 640 × 512 × 256, and use the average absolute difference between the inferred and the ground truth depth maps for the quantitative comparison. We denote the learned 2D image features as 2D CNNs. The comparison results of following settings are shown Fig. <ref> and Fig. <ref>:





  
2D CNNs + 3D CNNs Replace the GRU regularization with the same 3D CNNs regularization in MVSNet <cit.>. As shown in Fig. <ref> and Fig. <ref>, 3D CNNs produces the best depth map reconstructions. 





  
2D CNNs + GRU The setting of the proposed R-MVSNet, which produces the 2^nd best depth map results among all settings. The qualitative comparison between 3D CNNs and GRU is shown in Fig. <ref> (d) and (e).





  
2D CNNs + Spatial Replace the GRU regularization with the simple spatial regularization. We approach the spatial regularization by a simple 3-layer, 32-channel 2D network on the cost map. The depth map error of spatial regularization is larger than the GRU regularization.





  
2D CNNs + Winner-Take-All Replace the GRU regularization with simple the winner-take-all selection. We apply a single layer, 1-channel 2D CNN to directly map the cost map to the regularized cost map. The depth map error is further larger than the spatial regularization.





  
ZNCC + Winner-Take-All Replace the learned image feature and cost metric with the engineered ZNCC (window size of 7 × 7). This setting is also referred to the classical plane sweeping <cit.>. As expected, plane sweeping produces the highest depth map error among all methods.




  §.§.§ Post-processing

Next, we study the influences of post processing steps on the final point cloud reconstruction. We reconstruct the DTU evaluation without the variational refinement, photo-metric filtering, geometric filtering or depth map fusion. Quantitative results are shown in Table <ref>.





  
Without Variational Refinement This setting is similar to the post-processing of MVSNet <cit.>. The f_score is changed to a larger number of 0.465, demonstrating the effectiveness of the proposed depth map refinement.





  
Without Photo-metric Filtering Table <ref> shows that the f_score without photo-metric filtering is increased to a larger number of 0.467, which demonstrates the importance of the probability map for photo-metric filtering (Fig. <ref> (f)).





  
Without Geo-metric Filtering The f_score is increased to 0.432, showing the effectiveness of depth consistency.





  
Without Depth Map Fusion The f_score is also increased to 0.431, showing the effectiveness of depth fusion.




 §.§ Discussion




  
Running Time
For DTU evaluation with D=256, R-MVSNet generates the depth map at a speed of 9.1 s / view. Specifically, it takes 2.9 s to infer the initial depth map and 6.2 s to perform the depth map refinement. It is noteworthy that the runtime of depth map refinement only relates to refinement iterations and the input image size. Filtering and fusion takes neglectable runtime. 





  
Generalization
R-MVSNet is trained with fixed input size of N × W × H × D = 3 × 640 × 512 × 256, but it is applicable to arbitrary input size during testing. It is noteworthy that we use the model trained on the DTU dataset <cit.> for all our experiments without fine-tuning. While R-MVSNet has shown satisfying generalizability to the other two datasets <cit.>, we hope to train R-MVSNet on a more diverse MVS dataset, and expect better performances on Tanks and Temples <cit.> and ETH3D <cit.> benchmarks in the future. 





  
Limitation on Image Resolution
While R-MVSNet is applicable to reconstructions with unlimited depth-wise resolution, the reconstruction scale is still restricted to the input image size. Currently R-MVSNet can handle a maximum input image size of 3072 × 2048 on a 11GB GPU, which covers all modern MVS benchmarks except for the ETH3D high-res benchmark (6000 × 4000). 




§ CONCLUSIONS

We presented a scalable deep architecture for high-resolution multi-view stereo reconstruction. Instead of using 3D CNNs, the proposed R-MVSNet sequentially regularizes the cost volume through the depth direction with the convolutional GRU, which dramatically reduces the memory requirement for learning-based MVS reconstructions. 
Experiments show that with the proposed post-processing, R-MVSNet is able to produce high quality benchmarking results as the original MVSNet <cit.>. Also, R-MVSNet is applicable to large-scale reconstructions which cannot be handled by the previous learning-based MVS approaches.



ieee







[


Supplemental Materials




]










§ NETWORK ARCHITECTURE

This section describes the network architecture of R-MVSNet (Table <ref>). R-MVSNet constructs cost maps at different depths, and recurrently regularizes cost maps through the depth direction. The probability volume need to be explicitly computed during the network training, but for testing, we can sequentially retrieve the regularized cost maps and all layers only require the GPU memory with size linear to the input image resolution. 







§ DEPTH SAMPLE NUMBER

Given the depth range [d_min, d_max], we sample depth values using the inverse depth setting:

    d(i) = ( (1/d_min - 1/d_max) i/D - 1 + 1/d_max)^-1, i ∈ [1, D]

where i is the index of the depth sampling and D is the depth sample number. To determine the sample number D, we assume that the spatial image resolution should be the same as the temporal depth resolution. Supposing 𝐗_1 and 𝐗_2 are two 3D points by projecting the reference image center (W/2, H/2) and its neighboring pixel (W/2 + 1, H/2) to the space at depth d_min, the spatial image resolution at depth d_min is defined as ρ = ||𝐗_2 - 𝐗_1||_2. Meanwhile, we define the temporal depth resolution at depth d_min as d(2) - d(1). Considering Equation <ref>, the depth sample number is calculated as:

    D = (1/d_min - 1/d_max) / (1/d_min - 1/d_min + ρ).






§ VARIATIONAL DEPTH MAP REFINEMENT

We derive the iterative minimization procedure for Equation 8 in the main paper. Focusing on one pixel 𝐩_1 in the reference image, we denote its corresponding 3D point in the space as 𝐗 = Π_1^-1(𝐩_1) · d_1 + 𝐜_1, where Π_1, 𝐜_1 and d_1 are the projection matrix, camera center of the reference camera and the depth of pixel 𝐩_1. The projection of 𝐗 in the source image is 𝐩_i = Π_i (𝐗). For the photo-consistency term, we assume 𝒞(𝐈_1(𝐩_1), 𝐈_i → 1(𝐩_1)) = 𝒞(𝐈_1 → i(𝐩_i), 𝐈_i(𝐩_i)) and abbreviate it as 𝒞_1 → i(𝐩_i). The image reprojection error will be changed as 𝐃_1 deforms, and we take the derivative of the photo-consistency term w.r.t. to depth d_1:

    ∂ E^i_photo(𝐩_1)/∂ d_1   = ∂𝒞_1 → i(𝐩_i)/∂ d_1
       = ∂𝒞_1 → i(Π_i (Π_1^-1(𝐩_1) · d_1 + 𝐜_1)/∂ d_1
       = ∂𝒞_1 → i(𝐩_i)/∂𝐩_i·∂𝐩_i/∂𝐗·∂𝐗/∂ d_1
       = ∂𝒞_1 → i(𝐩_i)/∂𝐩_i·𝐉_i ·Π_1^-1(𝐩_1)

where 𝐉_i is the Jacobian of the projection matrix Π_i. ∂𝒞_1 → i(𝐩_i)/∂𝐩_i is the derivative of the photo-metric measurement w.r.t. the pixel coordinate. For computing the derivatives of NCC and ZNCC, we refer readers to <cit.> for detailed implementations. Also, considering d_1 = 𝐃_1(𝐩_1), the derivative of the smoothness term 𝒮(𝐩,𝐩') = w(𝐩_1, 𝐩_1') (𝐃_1(𝐩) - 𝐃_1(𝐩'))^2 can be derived as:

    ∂ E^i_smooth(𝐩_1)/∂ d_1
         = ∑_𝐩_1' ∈𝒩(𝐩_1)w(𝐩_1, 𝐩_1') ∂ (𝐃_1(𝐩_1) - 𝐃_1(𝐩_1'))^2/∂ d_1
         = ∑_𝐩_1' ∈𝒩(𝐩_1)2w(𝐩_1, 𝐩_1')(𝐃_1(𝐩_1) - 𝐃_1(𝐩_1'))

where w(𝐩_1, 𝐩_1') = exp(-(𝐈_1(𝐩_1) - 𝐈_1(𝐩_1'))^2/10) is the bilateral smoothness weighting.

We iteratively minimize the total image reprojection error E by gradient descent with a descending step size of λ(t) = 0.9 ·λ(t-1) and λ(0) = 10. The reference depth map 𝐃_1 and all reprojected images {𝐈_1 → i}_i=2^N will be updated at each step. The refinement iteration is fixed to 20 for all our experiments.




§ SLIDING WINDOW 3D CNNS

One concern about R-MVSNet is that whether the proposed GRU regularization could be simply replaced by streaming the 3D CNNs regularization in the depth direction. To address this concern, we conduct two more ablation studies. For DTU dataset, we divide the cost volume 𝐂 (D=256) into sub-volumes (D_sub=64) along the depth direction. To better regularize the boundary voxels, we set the overlap between two adjacent sub-volumes to D_overlap = 32, so in this way 𝐂 is divided into 7 subsequent sub-volumes {𝐂_i}_i=0^6. We then sequentially apply 3D CNNs (except for the softmax layer) on {𝐂_i}_i=0^6 to obtain the regularized sub-volumes. Then, we generate the final depth map by two different fusion strategies:




  * Volume Fusion First concatenate the regularized sub-volumes (truncated with D_trunc=16 to fit the overlap region) in depth direction. Then apply softmax and soft argmin to regress the final depth map.



  * Depth Map Fusion First regress 7 depth maps and probability maps from the regularized sub-volumes. Then fuse the 7 depth maps into the final depth map by winner-take-all selection on probability maps.



Qualitative and quantitative results are shown in Fig. <ref>. Both sliding strategies produce errors higher than GRU and 3D CNNs. Also, sliding strategies take ∼10s to infer depth map (H × W × D = 1600 × 1184 × 256), which is ∼2× slower than MVSNet and R-MVSNet.

The sliding window 3D CNNs regularization is a depth-wise divide-and-conquer algorithm and there are two major limitations: 1) One is the discrepancies among sub-volumes, as sub-volumes are not regularized as a whole. 2) The second is the limited size of the sub-volume, which is far less than the actual receptive field size of the multi-scale 3D CNNs (∼256^3). As a result, such strategies cannot be fully benefit from the powerful 3D CNNs regularization. 






§ POST-PROCESSING

We show in Fig. <ref> the qualitative point cloud results of DTU evaluation set <cit.> using different post-processing settings. The photo-metric filtering and the geometric filtering are able to remove different kinds of outliers and produce visually clean point clouds. Depth map refinement and depth map fusion have little influence on the qualitative results, however, they are able to reduce the overall score for the quantitative evaluation (Table 3 in the main paper).





§ POINT CLOUD RESULTS

This section presents the point cloud reconstructions of DTU dataset <cit.>, Tanks and Temples benchmark <cit.> and ETH3D benchmark <cit.> that have not been shown in the main paper. The point cloud results of the three datasets can be found in Fig. <ref>, Fig. <ref> and Fig. <ref> respectively. R-MVSNet is able to produce visually clean and complete point cloud for all reconstructions.










