Point Cloud Self-supervised Learning via 3D to Multi-view Masked Leaner
Self-supervised learning;Multi-modality learning;3D representation;Masked autoencoder
Yes
No ethics review needed.
Please kindly see the weaknesses above.
1. The paper is well-written and easy to follow, presenting good-quality figures.


2. The experimental results look promising.
This paper proposes Multiview-ML, a novel 3D representation learning model that solely uses 3D point cloud data as input to reconstruct both the original point cloud and multiple depth images from different viewpoints. 

It leverages a two-stage training strategy with a teacher and student model, and outperforms existing approaches across various downstream tasks.
1. The authors mention a limitation in prior work, stating that these methods *"inefficiently require both 2D and 3D modalities as inputs, even though 3D point clouds inherently contain 2D modality through their multi-view properties."* However, the authors provide insufficient evidence or ablation studies to substantiate this claim. Notably, previous works have often utilized only 3D inputs, projecting them into 2D during encoding without requiring both 2D and 3D modalities as explicit inputs.


2. The authors mention that the epoch number is 300, while do not specify how these are distributed across each stage. If both stages indeed run for 300 epochs, it raises the question of whether the observed improvement primarily results from an extended training period, which is computationally intensive.

3. It is better to demonstrate the individual effectiveness of each component in Table 5.

4. The ScanObjectNN and ModelNet40 datasets have reached saturation in point cloud understanding. Additional results on more complex and larger datasets, such as Objaverse, would be valuable.

5. In Supplementary Table 1, should "Ours (Point-M2AE)" actually be labeled as "Ours (Recon)"?

5. Typos: 'pertaining' on line 396.
Yes
No ethics review needed.
My major concerns are:
(1)The inputs of this model consists of both point clouds and 2D depth images. WWhat is the role of depth images within the model? How do they differ from the rendered images provided by the dataset?
(2)“incorporating both 2D and 3D modalities as input for training is redundant and inefficient.”However, the projection for 2D depth images is time-consuming. Additionally, two-stage training always need much time. 
(3)As stated in the abstract, "the input 2D modality causes the reconstruction learning to unnecessarily rely on visible 2D information, hindering 3D geometric representation learning." However, the proposed model also depends on 2D depth images: "These depth images then guide the reconstruction from 3D to 2D." The proposed method does not address or optimize the identified drawbacks.
(4) In the part segmentation experiment, there are no IoU (%) results of each category and visualization results, such as Point-MAE and Point-M2AE. 
(5) In the ablation experiments, there is no experiment conducted with other types of images (e.g., silhouettes, contours) as inputs.
(6)How about the training efficiency and parameters number? Introducing 3D to 2D projection, MSMH, and two-stage training strategy leads to additional training costs. This should also be discussed.
They propose a Multi-Scale Multi-Head (MSMH) attention mechanism that integrates local and global contextual information.
They employ a two-stage training strategy for multi-modality masked feature prediction.
They first project 3D point clouds to multi-view 2D images at the feature level based on 3D-based pose. Then, they introduce two components: (1) a 3D to multi-view autoencoder that reconstructs point clouds and multi-view images from 3D and projected 2D features; (2) a multi-scale multi-head (MSMH) attention mechanism that facilitates local-global information interactions in each decoder transformer block through attention heads at various scales. Additionally, a two-stage self-training strategy is proposed to align 2D and 3D representations.
The contributions are summarized as follows:
(1) They propose a 3D to multi-view autoencoder that reconstructs point clouds and multi-view images solely from 3D point clouds
(2) They propose a Multi-Scale Multi-Head (MSMH) attention mechanism that integrates local and global contextual information by organizing distinct, non-overlapping local groups at multiple scales within the reconstructed features.
(3) They develop a two-stage training strategy for multi-modality masked feature prediction
We think that the paper does not present its designs and motivations clearly
Yes
No ethics review needed.
1. Can authors better justify the motivation of the work? Especially given the fact that leveraging multimodal data for 3D representation learning is indeed achieving impressive results in many applications.
2. Can authors carefully compare their design differences with [1] and the MSMH design with the Grouped Vector Attention operation in [2]?
3. Experiments-wise, the authors need to provide a more comprehensive comparison with [1] and add comparisons with more recent methods such as [3]. Additional ablation studies would also be helpful.

[1] Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining.
[2] Point Transformer V2: Grouped Vector Attention and Partition-based Pooling.
[3] ShapeLLM: Universal 3D Object Understanding for Embodied Interaction.
I would like to summarize the strengths of the submission from the following aspects.
1. The writing is clear and easy to follow. Though some claims are not quite intuitive to me which I will detail later but the overall flow is good.
2. The idea of unsymmetric encoder-decoder design is interesting. Enforcing the autoencoder to decode multiview representations from 3D point cloud only inputs sounds a reasonable way to encourage the multi-view geometric understanding in the learned representations.
3. The figures are quite helpful for presenting the architecture and training flow. 
4. The experiments cover a good range of tasks and the ablation studies also shows the effectiveness of the proposed MSMH attention, the choice of recovering token representations rather than the raw data, and the design of instance-level intra and inter-modality prediction.
The paper focuses on the problem of self-supervised point cloud representation learning and presents a method named Multiview Masked Learner. The method learns 3D representation by first training a 3D to multi-view autoencoder to create informative latent features. Then a student network is trained to predict the latent features from masked point cloud input. The autoencoder is carefully designed so that  it encodes 3D point cloud and decodes both 3D point clouds and the corresponding multi-view projections. Multi-Scale Multi-Head attention mechanism is integrated to increase the expressivity of the features. The resulting 3D representation shows promising results in various point cloud analysis benchmarks including object classification, part segmentation, and object detection.
There are several weaknesses with the submission.
1. I am concerned with the technical novelty. Combining MAE with cross-modal distillation has been explored previously, e.g., as in [1]. Though in [1], images are used rather than projection of point clouds but I think the general framework is quite similar. It seems not quite challenging to replace the images used there with the projection of point clouds. The MSMH attention scheme also looks very similar to the Grouped Vector Attention operation in Point Transformer V2 [2]. The current pipeline seems more like an ensemble of existing techniques.
2. The motivation of the method is not strong enough. I do not see a particularly strong reason to avoid using images during the 3D representation learning stage if the multimodal-based pretraining gives better performance in downstream applications. Notice this line of work does not require images while using the pre-trained 3D backbone. The authors claim that incorporating both 2D and 3D during training is redundant and inefficient but do not provide concrete evidence to back up this claim. The claim from Line 49 to Line 53 also confuses me. I do not understand in what context this discussion happens and what is the key idea the authors want to deliver. In line 90, the authors say that their key insight is “the limited effectiveness of using 2D images as input for 3D geometric learning through MAE”. But doesn’t this suggest that we should develop better ways of using both 2D and 3D modalities instead of aborting the 2D modalities? In my humble opinion, previous works already discovered this and that’s why many works use MAE for the 3D modality and link 2D and 3D modalities through contrastive learning as done in [1].
3. The experiments conducted in the main paper are not comprehensive enough. The compared baselines are not very up-to-date and some more recent baselines are missing. For example, there is no comparison with [3]. Also, it feels very strange to leave important comparisons with ReCon [1] and I2P-MAE to the supplementary. The comparisons with [1] are incomplete in the supplementary with the few-shot classification experiments missing. In [1], experiments are also conducted in the 3D-only pretraining setup, and the results there seem comparable or even better than what is presented in the submission. When it comes to few-shot classification, the submission is not always winning either. In [3], a large collection of images helps further boost the representation quality to another level. This is to say, the presented results in the submission do not seem to be the state of the art as claimed.
4. Some ablation studies can be further improved. For example, what the performance would be if the input modality is 3D and the output modality is 2D? Also, a more detailed analysis of the MSMH design would be helpful for understanding its effectiveness and difference from previous works.

[1] Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining.
[2] Point Transformer V2: Grouped Vector Attention and Partition-based Pooling.
[3] ShapeLLM: Universal 3D Object Understanding for Embodied Interaction.
Yes
No ethics review needed.
N/A
1. The manuscript is well-organized and easy to follow. 
2. The experiments are extensive. The proposed method is tested on four tasks and compared with multiple baselines.
This work proposed a 3D to multi-view autoencoder that reconstructs both point clouds and multi-view images. The proposed mutli-scale multi-head attention module provides broader local and global information. Besides, the two-stage training strategy ensures the student model learns well-aligned representations. The extensive experiments show the effectiveness of the proposed method.
1. For the task part segmentation and few-shot learning, the proposed method achieved little increase. It would be better if more explanation and analysis can be given.
Yes
No ethics review needed.
1. How are the feature-level images in the abstract represented in the main text, please?
2. How does the number or size of multi-view images affect the proposed method, please? This may require further experimentation and analysis by the authors, and reference to CrossNet (TMM’2023) and Inter-MAE (TMM’2023), which use multi-view images for comparative learning between multiple modalities, is highly recommended.
3. Could the authors please provide appropriate qualitative visualisations such as complementary maps in their experiments? It is suggested to refer to Point-MAE (ECCV’2022), Point-M2AE (NeurIPS’2023) and TPM (arxiv’2024), which compare missing inputs and complementary outputs before and after self-supervision.

Overall, this is a relatively good work. If the author can take my comments into consideration and make explanations and revisions, I may further improve my score. AND vice versa.
1. The methodology section of the paper is quite well written, easy to understand and clearly guided. The four consecutive subsections show the implementation of the proposed method in a somewhat innovative way. It is recommended to add more details, e.g. the questions that follow.
2. In particular, the two-stage training strategy mentioned in the paper, i.e., aligning 2D and 3D representations using a network of teachers and students, is feasible and, moreover, the technique is challenging. The ability to appropriately apply it to multimodal self-supervised learning is a significant contribution, and I do hope that the authors will soon open-source this project for community advancement.
3. The proposed method is experimentally quite adequate and clearly outperforms state-of-the-art methods in a variety of downstream tasks including 3D classification, partial segmentation and object detection.
This paper presents a 3D-to-multi-view learner (Multi-View ML) that uses only 3D modalities as input and efficiently captures the rich spatial information in 3D point clouds. Specifically, we first project the 3D point cloud to a feature-level multi-view 2D image based on 3D pose. Then, we introduce two components: (1) a 3D-to-multiview autoencoder that reconstructs point cloud and multiview images from 3D and projected 2D features; and (2) a multi-scale multi-head (MSMH) attention mechanism that facilitates local-global information interaction in each decoder-converter block through different scales of attention heads. Furthermore, a novel two-stage self-training strategy is proposed to align 2D and 3D representations. The proposed method significantly outperforms state-of-the-art methods in a variety of downstream tasks, including 3D classification, part segmentation, and object detection.
1. An obvious grammatical error, "leaner" in the title should be "learner". In addition, it is recommended to enlarge the fonts of the images in the paper to enhance the presentation quality, as they are even smaller than the font size of the main text.
2. The novelty of this work needs to be additionally and strongly illustrated. On the one hand, this paper only uses point clouds and as input, however, I2P-MAE (CVPR’2023) and TAP (ICCV’2023) also follow this paradigm. On the other hand, the multi-scale multi-head (MSMH) attention mechanism mentioned in this paper aims to mine more features of the network, however, this is also explored in Point-M2AE (NeurIPS’2023) and more unsupervised methods.
3. The implementation part of the paper has some advantages over contemporaneous methods. However, as point cloud self-supervised learning evolves, more excellent work such as PointGPT (NeurIPS’2023), ReCon (ICML’2023), and ACT (ICLR’2023) should be considered for inclusion in the comparative experiments.
@inproceedings{
anonymous2024point,
title={Point Cloud Self-supervised Learning via 3D to Multi-view Masked Leaner},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=bw9bvwVwMH},
note={under review}
}
Recently, multi-modal masked autoencoders (MAE) has been introduced in 3D self-supervised learning, offering enhanced feature learning by leveraging both 2D and 3D data to capture richer cross-modal representations. However, these approaches have two limitations: (1) they inefficiently require both 2D and 3D modalities as inputs, even though the inherent multi-view properties of 3D point clouds already contain 2D modality.
(2) input 2D modality causes the reconstruction learning to unnecessarily rely on visible 2D information, hindering 3D geometric representation learning.
To address these challenges, we propose a 3D to Multi-View Learner (Multi-View ML) that only utilizes 3D modalities as inputs and effectively capture rich spatial information in 3D point clouds. 
Specifically, we first project 3D point clouds to multi-view 2D images at the feature level based on 3D-based pose.
Then, we introduce two components: (1) a 3D to multi-view autoencoder that reconstructs point clouds and multi-view images from 3D and projected 2D features; 
(2) a multi-scale multi-head (MSMH) attention mechanism that facilitates local-global information interactions in each decoder transformer block through attention heads at various scales. 
Additionally, a novel two-stage self-training strategy is proposed to align 2D and 3D representations.
Empirically, our method significantly outperforms state-of-the-art counterparts across various downstream tasks, including 3D classification, part segmentation, and object detection.
Such performance superiority showcases that Multi-View ML enriches the model's comprehension of geometric structures and inherent multi-modal properties of point clouds.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Self-supervised learning Multi-modality learning 3D representation Masked autoencoder
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/84ba8ee825d75dd266139619bf8e04135249d366.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/6242e489700d3bdcd4501ff1862c181c654c6298.pdf
Point Cloud Self-supervised Learning via 3D to Multi-view Masked Leaner
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission