PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views
3D Gaussian splatting;3D reconstruction;Novel view synthesis
Yes
No ethics review needed.
Kindly refer to [Weaknesses].
* The motivation is reasonable, emphasising the significance of updating the predicted pixel-aligned 3D Gaussians under more input views.
* Experiments on two existing benchmarks verify the efficacy of the introduced PixelGaussian
This work introduces PixelGaussian, a feed-forward 3DGS model that can adaptively update the number of Gaussian via pruning and splitting. In particular, PixelGaussian starts by predicting the pixel-aligned 3D Gaussians. These 3D Gaussians are used as input to the introduced Cascade Gaussian Adapter (CGA) and Iterative Gaussian Refiner (IGR) for further processing, yielding a set of updated 3D Gaussians aligned with the geometry complexity. Experiments on RealEstate10K and ACID demonstrate the effectiveness of PixelGaussian, showing state-of-the-art performance.
* Accessing the quality of depth map and extracted point cloud. As mentioned in L91, this work aims to “mitigate Gaussian overlap and redundancy”. However, this contribution cannot be justified by the RGB image-based results. It is crucial to access the 3D Gaussian quality by visualising the depth map and point cloud, similar to Fig. 4 in MVSplat. In particular, it would be better to compare with pixelSplat and MVSplat in terms of point cloud and depth map, especially using more input views, e.g., 4 input views. It is also suggested that similar comparisons be made among different ablated models, which would make it easier to understand the effectiveness of CGA and IGR.

* Accessing more input views. The significance of refining 3D Gaussians would be more obvious under the settings of more input views. Hence, it would be interesting to see how well the introduced PixelGaussian can perform using more input views, e.g., 12 views, 16 views, or even 32 views, similar to the concurrent work Long-LRM [Ziwen et. al, arXiv:2410.12781].

* Accessing more complex datasets. As shown in Fig. 4, the visual differences between PixelGaussian and other state-of-the-art methods are minor, possibly because RealEstate10K is too simple to demonstrate the effectiveness of Gaussian refinement. Hence, it is highly recommended to report additional comparisons on more complex datasets, e.g., MipNeRF360, Tanks and Temples.

* Accessing cross-dataset generalization. It would be interesting to see how well the introduced CGA and IGR can generalize to other datasets. For example, trained on RealEstate10K but tested on DTU, similar to Fig. 5 in MVSplat.

* The motivation of using Deformable Attention is unclear. It might be better to provide experiments with further analysis to justify why it is better to use Deformable Attention instead of typical attention blocks.

* The paper title and model name are confusing. It might be better to highlight in the title that this work predicts an adaptive quantity of 3D Gaussians. Besides, the model name ‘PixelGaussian’ might unintentionally imply ‘pixel-align Gaussian’, which is the opposite of the objective. It might be better to change it to ‘AdaptiveGaussian’ or some other name that better reflects the main contribution of this work.
Yes
No ethics review needed.
1.Is it possible to provide details on the view selection process, including the angular separation between views and whether test views are interpolated between or extrapolated beyond the training views as well as the camera pose distribution for both training and testing sets.

2.Since the author argues their method can be trained on two views and tested on more views. Is it possible to show that such training strategies have better generalization ability for large scale/wide range scenes and compare results with those diffusion-based sparse view reconstruction methods including ReconFusion, ViewCrafter, ReconX following their experiments settings. Also give more discussions on such settings.
1.The paper proposes a generalizable gaussian prediction method with Cascade Gaussian Adapter (CGA) to dynamically adapt the distribution and quantity of Gaussians, which is effectively shown in the experiments part.

2.It further refines Gaussian representations via proposed Iterative Gaussian Refiner with deformable attention.

3.It shows with better distributed gaussians, the final rendering results are better than those methods with per-pixel predicted gaussians and better generalization when trained with 2 views and tested with 2-4 views.
They propose a feed-forward framework 3D Gaussian reconstruction from arbitrary views. With propose CGA and IGR, they can produce well distributed gaussians base on geomtry complexity and generalize better to arbitrary views than other methods. They have validation on benchmarks to show the effectiveness of their method.
1. While the paper demonstrates improved generalization with increased view numbers during inference through adjusted Gaussian distributions. However, it would make more sense by including results for baseline methods trained specifically on 2, 4, and 6 views.

2. The current experiments primarily showcase scenarios with substantial FOV overlap between input views. To better demonstrate the method's generalization capabilities, consider including more challenging test cases - for instance, evaluating a model trained on 2 views when tested with 2-6 widely-spaced views with minimal overlap. Such examples would validate the method's ability to handle challenging novel view synthesis scenarios.
Yes
No ethics review needed.
1. As shown in Table 1, the 3D-GS-based PixelGaussian demonstrates marginally improved performance over the NeRF-based MuRF, especially on the ACID benchmark. A more detailed analysis is required to explain the underlying reasons for this performance difference, training and inference efficiency, and memory usage.
2. RealEstate10K and ACID benchmarks were first officially introduced in pixelSplat, but I think the efficiency should be demonstrated in more benchmarks. I would like to see more comparisons in more diverse datasets, such as MuRF's DTU, LLFF, and Mip-NeRF 360 dataset, or even self-captured real-world via cameras or phones (see the teaser in Metric3D [1] and F2-nerf [2]).
3. I would like to see the full efficiency comparison across 2-6 views, in addition to only 4 views in Table 3.
4. The training efficiency comparison with MVSplat should also be analyzed with the same amount of training data and computing resources.
5. I think the 3D-GS adaptions in Fig. 6 and Fig. 1 are not visually-pleasing in the ICLR-level paper and can be drawn and demonstrated better.


[1] Metric3d: Towards zero-shot metric 3d prediction from a single image. ICCV 2023

[2] F2-nerf: Fast neural radiance field training with free camera trajectories. CVPR 2023
1. The proposed CGA and IGR modules are novel in the feed-forward 3D-GS scenarios.
2. The experiments effectively support the proposed method: Table 1 demonstrates visual performance compared to SOTAs, Table 2 illustrates scalability concerning input views, Table 3 highlights inference efficiency achieved through reduced 3D Gaussians, and Table 4 validates the effectiveness of the designed modules.
3. The paper is well-motivated and easy to understand and follow.
PixelGaussian employs a dynamic mechanism to identify the region of interest within the underlying 3D spatial locations, and accordingly adjusts the 3D-GS distribution and quantity to enhance visual quality while reducing memory usage. Experiments on ACID and RealEstate10K validate the effectiveness of the proposed method.
Please see the "Questions" section.
Yes
No ethics review needed.
Please see the above weaknesses. If they are adequately addressed, I am willing to increase the rating.
1. Motivations are clear. PixelSplat and MVSplat indeed have problems when the number of views increase. This paper tackles this issue, which I think is very reasonable.

2. Ablation study is thorough and the performance of the proposed method is impressive.

3. Writing is good and easy to understand.
This work first points out the unique issue posed by the use of pixel-aligned 3D Gaussians, which is the redundant 3D Gaussians depicting the same obejcts' surfaces. Due to this issue, existing works, such as MVSplat and PixelSplat, perform bad when the number of views increases, unlike the NeRF-based generalized NVS works, PixelNeRF, IBRNet or MVSNeRF, that benefit from more input views. To handle this issue, the authors propose so called CGA, to adjust Gaussian distribution according to local geometry complexity identified by a keypoint scoring function. Subsequently, a transformer-based IGR is proposed to model interactions between image and 3D Gaussians to realize further refinements. The proposed method is evaluated on large-scale indoor/outdoor datasets and is shown to achieve new state-of-the-art.
1. It seems that the performance of PixelGaussian is better in 2 view setting as well. Why is this? I understand that for more views, it will the proposed method will perform better than them, but i can not think of a convincing reason for the superior performance in 2 view setting. I am guessing that the model is trained on 8 A6000 GPUs, which incorporates larger effective batch size than MVSplat. So I would say it's not fair to compare directly with MVSplat, thus we can not be so sure that the proposed method would work also well with 2 views. It might even perform worse if trained with similar iterations and batch size. Would it be possible if the authors  provide a more detailed comparison of training conditions between PixelGaussian and MVSplat, including batch sizes, number of iterations, and computational resources used. Additionally, if PixelGaussian trained under similar conditions as MVSplat to ensure a fair comparison in the 2-view setting, what would be the results?

2. In Table 3, the authors highlight that the rendering FPS is higher than MVSPlat. This makes sense, since the # of Gaussians are less compared to MVSplat. However, Since the architecture of the proposed method builds on top of MVSplat, it will likely to infer slower overall, if both 3D reconstruction and nvs were accounted for, which is reported and supported by the latency. So I think the efficiency is not really a big contribution, **unless** the authors showed the differences in higher resolutions. Currently 256x 256 is too small to see the dramatic difference. But the current submission unfortunately does not have efficiency comparisons in higher resolutions, which could've been a strength of this work.  Would it be possible if the authors provide detailed comparisons at higher resolutions (e.g., 512x512, 1024x1024 or higher).  This could be this paper's strength if properly compared.

3. It would've been even more interesting if the paper explored the impacts of increasing the number of input views more and more. I would expect at some point, the performance will saturate, but it will be very interesting to see those. Would it be possible if the authors systematically increases the number of input views (e.g., from 2 to 8 or 10) and report how the performance changes, including where it potentially saturates?
@inproceedings{
anonymous2024pixelgaussian,
title={PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=sR0xz6ZaH7},
note={under review}
}
We propose $\textbf{PixelGaussian}$, an efficient feed-forward framework for learning generalizable 3D Gaussian reconstruction from arbitrary views. Most existing methods rely on uniform pixel-wise Gaussian representations, which learn a fixed number of 3D Gaussians for each view and cannot generalize well to more input views. Differently, our PixelGaussian dynamically adapts both the Gaussian distribution and quantity based on geometric complexity, leading to more efficient representations and significant improvements in reconstruction quality. Specifically, we introduce a Cascade Gaussian Adapter (CGA) to adjust Gaussian distribution according to local geometry complexity identified by a keypoint scorer. CGA leverages deformable attention in context-aware hypernetworks to guide Gaussian pruning and splitting, ensuring accurate representation in complex regions while reducing redundancy. Furthermore, we design a transformer-based Iterative Gaussian Refiner (IGR) module that refines Gaussian representations through direct image-Gaussian interactions. Our PixelGaussian can effectively reduce Gaussian redundancy as input views increase. We conduct extensive experiments on the large-scale ACID and RealEstate10K datasets, where our method achieves state-of-the-art performance with good generalization to various numbers of views.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
3D Gaussian splatting 3D reconstruction Novel view synthesis
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4ad952254da7dc019d304caf0360435de4dceb22.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission