MutualNeRF: Improve the Performance of NeRF under Limited Samples with Mutual Information Theory
nerf;mutual information;sparse view sampling;few-shot view synthesis
Yes
No ethics review needed.
See the weaknesses above.
- This paper is well written and easy to follow.
- The framework’s design is comprehensive, considering both macro (semantic) and micro (pixel) perspective in the task of sparse view sampling and few-shot NVS.
The paper proposes MutualNeRF, a method to improve the performance of Neural Radiance Fields (NeRF) when training samples are limited. The authors integrate mutual information theory to develop a unified approach, enhancing NeRF's effectiveness in sparse view sampling and few-shot view synthesis, by modeling uncertainty in semantic space distance and pixel space distance.
- **Complex methodology with marginal gains:** This methodology introduces significant complexity, especially in sparse view sampling, involving greedy algorithms and complex mutual information metrics. However, the observed improvements over simpler baselines are relatively minor, which may not justify the added complexity.
- **Lack of novelty.** The attempt to address the task of few-shot novel view synthesis through minimization of mutual information between viewpoints have already been explored, especially in CVRP 2022 paper **InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering**. The methodology of this paper seems to be re-interpretation of ideas used in DietNeRF (semantic space) and InfoNeRF (pixel space) within the perspective of mutual information theory. I ask the authors to give a more thorough theoretical comparison between this work and the methods that I have mentioned.
Yes
No ethics review needed.
please refer to the weaknesses part.
The authors demonstrate the effectiveness of both sparse view sampling and few-shot view synthesis. Using mutual information to select views for NeRF is reasonable.
This paper proposes a framework to improve the performance of sparse view sampling and few-shot view synthesis by mutual information. For sparse view sampling, the authors first calculate the semantic space distance and pixel space distance, then use a greedy algorithm to select sparse views for training. For few-shot view synthesis, the authors add two regularization terms based on semantic space distance and pixel space distance, to improve the performance of view synthesis. The authors conduct detailed experiments to demonstrate the effectiveness.
1. The motivation for pixel space distance is not clarified. According to Definition 3, the pixel space distance is the expectation of distance between any two points of rays. The authors should clarify the motivation behind Definition 3 since it is important for the following parts. The semantic space distance is fairly reasonable.

2. If my understanding is correct, for the few-shot view synthesis, the authors just added two regularization terms to the NeRF training. However, the relationship between mutual information and few-shot view synthesis is not clear. For sparse view sampling, minimizing mutual information is reasonable.

3. For the few-shot view synthesis, the proposed method needs to evaluate the semantic distance between randomly rendered images and ground truth images. Therefore, this will bring additional training costs. The analysis should be provided.

4. Compared with NeRF, 3D Gaussian splatting is much better in both training and rendering. The authors should conduct experiments based on 3DGS.

5.  There are many typos and grammatical errors. For example, formulas should come with indices.
Yes
No ethics review needed.
Please refer to the weakness part.
Utilizes mutual information theory to improve NeRF under limited data.

Proposes a greedy algorithm for strategic viewpoint selection in sparse view sampling.

Introduces efficient regularization terms to enhance few-shot view synthesis.
The paper introduces MutualNeRF, a framework that enhances Neural Radiance Field (NeRF) performance under limited sample conditions using Mutual Information Theory. It addresses sparse view sampling and few-shot view synthesis by minimizing and maximizing mutual information, respectively. The framework employs a greedy algorithm for viewpoint selection and plug-and-play regularization terms for training.
1.	How about using mutual information for 3DGS?
2.	What is the rendering speed of the proposed method, especially when compared with 3DGS?
3.	I suggest that the authors compare or discuss with more methods, like PixelNeRF[A], CR-NeRF[B], and MVSGaussian[C], to fully verify the effectiveness of the proposed mutual information.
4.	Why choose to pick images instead of training as a whole? Are there images that do not belong to the target scene? If we use a fast reconstruction method like 3DGS, training all images won't take much time.

**Reference**

[A] Yu, Alex, et al. "pixelnerf: Neural radiance fields from one or few images." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[B] Yang, Yifan, et al. "Cross-ray neural radiance fields for novel-view synthesis from unconstrained image collections." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[C] Liu, Tianqi, et al. "MVSGaussian: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo." European Conference on Computer Vision. Springer, Cham, 2025.
Yes
No ethics review needed.
1) Figure 4 shows that some regions, such as the stairs, are not improved. Why might this be the case? Could this indicate limitations in the Mutual Information design? Specifically, which types of regions can be enhanced by the macro and micro losses? Currently, it is unclear which specific issues the proposed framework addresses. Could the authors provide a more detailed explanation?

2) What are the comparisons in terms of training time and peak memory consumption?

3) Table 3 and Table 4 show that the performance gain for "FreeNeRF + Ours" is marginal. This raises concerns about whether MutualNeRF will improve performance for more advanced backbones, such as Sparsenerf [A] or Sparsenerf combined with FreeNeRF. Could the authors comment on this?

4) Does the proposed macro loss also improve the performance of 3DGS-based methods? If so, did the authors experiment with sparse-view 3DGS splatting methods, such as [B]?

5) What would happen if we applied CLIP as a perceptual loss to support FreeNeRF training?
1. Introducing mutual information to enhance Neural Radiance Field (NeRF) performance is an innovative idea.
2. The experimental results in Table 3 show that the proposed method improves performance across various NeRF frameworks.
3. This paper is well-organized.
This paper proposes utilizing Mutual Information Theory to enhance the performance of NeRF with limited samples. Mutual information serves as a metric to measure the correlation between images at both macro and micro levels. The macro perspective focuses on correlations in semantic features, while the micro perspective addresses correlations in the pixel space. By incorporating mutual information, the proposed framework effectively addresses challenges posed by sparse view sampling and few-shot view synthesis, resulting in improved performance. Extensive experiments demonstrate the effectiveness of assessing both semantic and pixel space distances.
1. Results in Figure 4 show limited improvement between FreeNeRF and the proposed MutualNeRF; only the zoomed-in sections reveal marginal enhancements, which are still not substantial.
2. The proposed method introduces more complex training requirements and utilizes a large model, CLIP, for assessing semantic space distance, which may impact training time and peak memory consumption.
3. Performance improvements become marginal with more powerful baseline frameworks. For instance, "RegNeRF + Ours" surpasses RegNeRF by 1.28 PSNR, while "FreeNeRF + Ours" outperforms FreeNeRF by only 0.50 PSNR.
@inproceedings{
anonymous2024mutualnerf,
title={MutualNe{RF}: Improve the Performance of Ne{RF} under Limited Samples with Mutual Information Theory},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=5RPpwW82vs},
note={under review}
}
This paper introduces MutualNeRF, a framework enhancing Neural Radiance Field (NeRF) performance under limited samples using Mutual Information Theory. While NeRF excels in 3D scene synthesis, challenges arise with limited data and existing methods that aim to introduce prior knowledge lack theoretical support in a unified framework. We introduce a simple but theoretically robust concept, Mutual Information, as a metric to uniformly measure the correlation between images, considering both macro (semantic) and micro (pixel) levels.
  For sparse view sampling, we strategically select additional viewpoints containing more non-overlapping scene information by minimizing mutual information without knowing the ground truth images beforehand. Our framework employs a greedy algorithm, offering a near-optimal solution for this task.
  For few-shot view synthesis, we maximize the mutual information between inferred images and ground truth, expecting inferred images to gain more relevant information from known images. This is achieved by incorporating efficient, plug-and-play regularization terms.
  Experiments under limited samples show consistent improvement over state-of-the-art baselines in different settings, affirming the efficacy of our framework.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
nerf mutual information sparse view sampling few-shot view synthesis
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/087c232a7bc68a793c1ba9ceef29cec91cd432e5.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
MutualNeRF: Improve the Performance of NeRF under Limited Samples with Mutual Information Theory
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission