Single-View 3D Representations for Reinforcement Learning by Cross-View Neural Radiance Fields
3D scene representation;Single-view inference;NeRF;Reinforcement Learning
Yes
No ethics review needed.
Please see my questions in the weakness section above. I look forward to author's responses.
In my opinion, below are the strengths of the approach:

1. Designing relevant experiments and showcasing improvement numbers that highlight the method is invariant to the viewpoint and camera matrices. Slight perturbation in the cameras from the reference views shows the learned policies are invariant to disturbances. 

2. The writing and flow of the paper is nice, and the presentation is clear. 

3. Strong qualitative improvement results against competing baselines.
The paper proposes an interesting approach to utilize NeRF based pretraining to bake in viewpoint awareness into an RL system. The authors first pretrain a representation using cross-view completion objective visa NeRF rendering using time contrastive learning objective for scene regularization. The authors then use the pretrained scene encoder for downsteam reinforcement learning task. Relevant experiments are designed which demonstrate viewpoint awareness of the system in a synthetic setup.
In my opinion, the weakness of the paper is as follows:

1. The paper misses various key recent results both for 3D representation learning using NeRFs [1] and for baking in viewpoint awareness for policy learning [2,3]. In my opinion, the paper is incomplete without discussion or comparison to these approaches.  

2. The paper doesn't show any real-world evaluation results while both [2,3] show real-world results. Is it an inherent limitation of the method that it only works in simulation?

3. Follow-up to point 1. While the paper shows qualitative comparison to recent NeRF-based methods, how does the result compare to zero-shot generalizable NeRF-based method i.e. ZeroNVS (zero-shot vs. finetuned on their data) and NeRF representation learning method i.e. NeRF-MAE trained on their data?

4. What is the pretraining data mix and how does it impact OOD policy learning? Can the model generalize to OOD in sim i.e. sim2sim generalization or OOD real i.e. sim2real generalization?

[1] Irshad et al., ECCV 2024 NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fields
[2] Chen et al. CORL 2024, RoVi-Aug: Robot and Viewpoint Augmentation for Cross-Embodiment Robot Learning
[3] Tian et al. CORL 2024, View-Invariant Policy Learning via Zero-Shot Novel View Synthesis
[4] Sargent et al. ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image
Yes
No ethics review needed.
See the weakness above.
* The proposed method can reconstruct 3D scene representation from single-view images, eliminating the need for multi-sensor setup and calibration for learning downstream RL algorithms.
* By using an autoencoder architecture to learn the NeRF representation, it bypasses the time-consuming optimization required in classical NeRF reconstruction methods and potentially predicts occluded regions, unlike traditional NeRF approaches.
* The authors conduct extensive experiments to demonstrate that the proposed methods achieve superior performance for both volume rendering and downstream RL algorithms such as DrM.
This paper introduces a framework that generates 3D-aware representations from single-view camera inputs, which can be rendered into observations for training RL models. The 3D reconstruction model uses an autoencoder architecture, with a masked ViT as the encoder and a latent-conditioned NeRF as the decoder, trained with cross-view completion objectives. Experimental results demonstrate that the proposed method greatly improves the RL agent's performance for complex tasks.
* The time contrastive loss (Eqn. 3) repulses state features at different timesteps. However, this does not hold for static scenes where the actor remains stationary between timesteps $t$ and timestep $t^\prime$.
* The 3D encoder-decoder model $\Omega_\theta$ is trained on multi-view images, with scene representation $z_t = \Omega_\theta(O_{t-2:t}^i, O_{t-2:t}^{r_1}, \cdots, O_{t-2:t}^{r_K})$. How can it generalize when the inputs are from the same viewpoint, as in $z_t = \Omega_\theta(O_{t-2:t}^i, [O_{t-2:t}^i,] * K)$ (line 291)? 
* Table 1 claims that the proposed method does not require camera calibration. However, camera poses are needed to render multi-review reconstruction from $z_t$, making this claim inaccurate.
* In the volume rendering experiments, the authors should also include comparisons with NeRF baselines for sparse views, such as RegNeRF, pixelNeRF, etc.
* The RL experiments are conducted on toy environments. It would be valuable to see the method's performance in real-world robotic settings.
Yes
NA
No ethics review needed.
1. The author should explain more details about the camera calibration.
2. The author should add some baselines with explicit 3d reapresentation in RL.
3. The visualization result of snerl looks much worse than it original paper, it will be great to see the visualization on the same env and setting.
Strengths

The results of the proposed method demonstrate superior performance compared to previous relative methods.
Summary
This paper introduces a 3D representation reinforcement learning (RL) framework that utilizes a single view for inference. The downstream RL process leverages the latent code derived from the single image as input for the RL tasks.
Weaknesses

The writing quality requires improvement, for example in line 300, where the meaning is unclear. The preceding sentence discusses a reinforcement learning (RL) algorithm, but the subsequent sentence shifts focus to data shuffling, creating a disjointed narrative. Additionally, this sentence is ambiguous  and difficult to comprehend (eg, why `randomize viewpoint` but not `random pick a viewpoint` ).

In line 93, the authors do not clarify the concept of a calibrated camera, both in this section and in subsequent ones. Additionally, the process for computing the (x,d) values during rendering is not explained. Therefore, the claim that the proposed method operates 'without requiring camera calibration' is misleading; instead, it could be interpreted that 'camera calibration is addressed through overfitting.' It appears that the authors are utilizing an absolute camera pose along with a fixed intrinsic matrix. Consequently, the image encoder and neural radiance fields (NeRF) are effectively learning a fixed RGB->pose mapping. This principle is referenced in [1] and may lead to poor generalization. Furthermore, recent multi-view stereo (MVS) reconstruction models demonstrate that a calibration matrix is not essential for creating MVS 3D models. The authors should explore relevant literature in the domains of lightweight regression models (LRM), single-view LRMs, LRM with Gaussian distributions, and indoor LRM-like methodologies.

The image encoder and NeRF appear to be overfitting to the given dataset, similar to previous dynamic NeRF approaches that attempt to learn a mapping of f(x,d,t)=c,\rho. The latent variable z in the proposed method effectively serves as a latent code encompassing (t, action, state, pose, intrinsic parameters, and object). For instance, in Figure 3, if the proposed method utilizes only view V3 as input, it can accurately recover the clearly marked red annotation on the box, which is not visible in view V3. To the best of my knowledge, no existing methods—whether single view to 3D, MVS to 3D, learning-based, diffusion-based, for objects, indoors, or outdoors— can achieve it whtiout overfitting.

Several baseline comparisons are missing. Since the proposed method aims to illustrate the effectiveness of a single-view latent 3D representation for RL processes, it is essential for the authors to include baselines that utilize explicit 3D representations, such as depth maps or 3D volumes, as presented in the recent conference proceedings.

[1] PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization
Yes
No ethics review needed.
See weaknesses.
- The technical approach is well-motivated and addresses a practical limitation of existing 3D representation learning methods for RL - the requirement for multi-view or calibrated cameras during deployment
- The empirical results demonstrate the method works as intended, achieving comparable performance to multi-view baselines while requiring only single-view input
This paper presents SinCro, a framework for learning 3D-aware representations for reinforcement learning that can operate with single-view inputs during deployment. The key innovation is combining a masked ViT encoder with a latent-conditioned NeRF decoder, trained through cross-view completion and time contrastive learning. The method enables single-view 3D representation inference without requiring camera calibration during deployment, while previous approaches typically needed multi-view inputs or calibrated cameras.
My primary concerns are:

- The evaluation is limited to MetaWorld environments, which are relatively simple by 2024 standards. Testing on more complex manipulation scenarios would strengthen the paper. There are a lot of other simulated environments like RLBench. Can you explain why MetaWorld is used?
- The quantitative results in Figure 3 show an apparent contradiction - NeRF-RL achieves higher PSNR despite producing visibly blurrier reconstructions. This needs better explanation. Can you explain why NeRF-RL images are blurry but the PSNR is higher than SinCro?

Some additional comments

- The figures could be improved - Figure 2 is a PNG instead of vector graphics which reduces quality
Yes
No ethics review needed.
Of course, it’s not necessary to compare, but it may be good to discuss some concurrent related works, such as Dreamitate, VISTA, RoVi-AUG, which leverage generative models to learn view-invariant RL policies.
Learning viewpoint invariant embeddings is an important problem in robotics as prior work has shown the sensitivity of robot policies to out-of-domain camera viewpoints. 

The experimental evaluation and ablation study as well as qualitative analysis are quite thorough and nice to see.
A 3D-aware representation learning approach is presented in which posed multiview data is leveraged to learn view-invariant representations from images. These representations can be used as auxiliary input to an RL policy, where it is shown they achieve superior performance relative to other such baselines.
* The proposed representation requires synchronized multiview video data to be trained, so it can only be trained on limited data. It would be good to compare against embeddings such as DinoV2 which do not have explicit geometry-aware nature but can be trained on a lot more data and probably have a notion of “view-invariance” to some degree due to their training strategy.

* Table 1 is a bit misleading. While during deployment the proposed algorithm can indeed be run on single view input, if I’m not mistaken during training of the actual embedding the requirement is still for posed multiview data. Perhaps it would be better to disentangle the deployment and training stages in this Table for the proposed method and for baselines as applicable. 

* I think the paper focuses slightly too much on the few- or single-view reconstruction results visually and wr.t. view synthesis metrics, which I don’t think is particularly informative. Single- and few-view reconstruction is a huge field by itself and there are much stronger baselines to compare against if this is the goal such as PixelNeRF, NeRDi, GS-LRM, ZeroNVS, Cat3D, Reconfusion, etc. etc. The goal of the paper is not to solve single or few view 3D reconstruction but to learn 3D-aware representations for downstream RL.
@inproceedings{
anonymous2024singleview,
title={Single-View 3D Representations for Reinforcement Learning by Cross-View Neural Radiance Fields},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Crsl3zbfvW},
note={under review}
}
Reinforcement learning (RL) has enabled robots to develop complex skills, but its success in image-based tasks often depends on effective representation learning. Prior works have primarily focused on 2D representations, often overlooking the inherent 3D geometric structure of the world, or have attempted to learn 3D representations that require extensive resources such as synchronized multi-view images even during deployment. To address these issues, we propose a novel RL framework that extracts 3D-aware representations from single-view RGB input, without requiring camera calibration information or synchronized multi-view images during the downstream RL. Our method employs an autoencoder architecture, using a masked ViT as the encoder and a latent-conditioned NeRF as the decoder, trained with cross-view completion to capture fine-grained, 3D geometry-aware representations. Additionally, we utilize a time contrastive loss that further regularizes the learned representation for consistency across different viewpoints. Our method significantly enhances the RL agent’s performance in complex tasks, demonstrating superior effectiveness compared to prior 3D representation-based methods, even when using only a single, uncalibrated camera during deployment.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
3D scene representation Single-view inference NeRF Reinforcement Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7a88bdaa7eb2ce69372691f6c106f9b825cacc33.pdf
applications to robotics, autonomy, planning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/6ff56c2f4ecaa9e049069f5a92abd8c30457d5de.zip
Single-View 3D Representations for Reinforcement Learning by Cross-View Neural Radiance Fields
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission